How detailed should a pipeline diagram be: four zoom levels and a node budget

Platform level. Systems only: the source systems, the ingestion layer, the warehouse or lakehouse, the transformation layer, the serving surfaces. Ten to fifteen nodes. This is the diagram for a review with people who do not work on the pipeline, and it should fit on a slide without shrinking the text.

Pipeline level. One pipeline or domain: its stages, its stores, its orchestration, its cadences, its failure path. Twenty to thirty nodes. This is the working diagram, the one that lives in the README, and the one most people actually mean when they say pipeline diagram.

Model level. Tables and models within a layer, grouped by folder or domain with counts. This is where the dbt project structure or the medallion table families belong, and it is usually better served by a generated catalog than by a drawn diagram.

Column level. Which field feeds which field. Nobody should draw this by hand at all; it is a query, not a picture, and it belongs in a lineage view where you can trace one column at a time rather than look at all of them.

Set a budget before you start and treat it as a constraint rather than a target. Around thirty nodes is the practical ceiling for a diagram that will be read on a laptop screen or embedded in a wiki page. Past that, labels shrink, the layout engine starts making decisions for you, and readers stop scanning and start hunting.

The test that keeps you honest is the label test: if you cannot fit every node label at a readable size without abbreviating, you are at the wrong level or over budget. Abbreviations are the first symptom of a diagram trying to be two diagrams.

When you go over, collapse rather than delete. Forty connectors become one node reading 40 connectors, sixty staging models become one zone reading staging, 61 models, twelve dashboards become one node reading 12 dashboards. The information moves into the written document beside the diagram, where it can be searched, and the shape survives on the canvas. Deleting loses the fact; collapsing keeps it and moves it.

Over budget            ->  Collapsed
40 connector nodes     ->  1 node: "40 managed connectors (list in doc)"
61 staging models      ->  1 zone: "staging - 61 models, all views"
12 dashboard nodes     ->  1 node: "12 Looker dashboards"
8 identical Spark jobs ->  1 node: "partition_enrich x8 (one per region)"

Cadence. At every zoom level, a reader needs to know how often data moves. At platform level that may be one annotation per zone; at pipeline level it is one per edge. A diagram without cadence cannot distinguish broken from early, which is the most common question asked of any pipeline.

Ownership. Who is responsible for each zone. This is the fact that determines who gets contacted, and it is cheap to carry at any level because it goes on zone labels rather than on nodes.

The trust boundary. Which parts you control and which parts can change without warning you. At platform level it is one enclosing zone; at pipeline level it is the same boundary with the individual sources visible outside it. Everything else is negotiable by level; these three are not.

One more habit worth adopting: state the level on the diagram itself, in a text label, for example platform level, systems only, no tables. It stops the well meaning addition of just one more box that quietly moves the diagram to a level it was not designed for.

A ceiling is easier to hold when something enforces it. Datadef enforces one when it draws infrastructure from a repository, and the numbers are worth borrowing because they were tuned against real repositories rather than picked to sound tidy.

A module zone holds at most eight drawn nodes, and its remaining lone resources fold into a single node labelled "Supporting resources ×N". Repeated types inside a module never get separate boxes at all: two storage credentials become one node reading "Storage credentials ×2" with the member names in its description. The canvas as a whole targets forty nodes, and when a first pass comes in above that, a stricter pass runs which folds lone minor resources too.

Two exceptions exist and both are about not gutting the subject. When one module accounts for more than half of everything visible in the repository, its cap rises from eight to sixteen, since compressing the very thing the reader came for into eight boxes hides it. And a catalog-shaped repository of more than eight independent stacks scales its total target to five nodes per stack, capped at seventy two, because nineteen small architectures read well as a grid of zones and badly as a forced forty.

The measured result on a real Azure platform: 123 declared resources and 30 data lookups, spread over five stacks and eleven local modules, became 36 nodes in 12 zones with the largest zone holding nine. That is the same order of magnitude as the pipeline level budget above, arrived at from the opposite direction, which is the best evidence available that thirty is roughly where readability actually sits.

• How to draw a data pipeline diagram
• What a good pipeline diagram shows
• Column level lineage
• Agents and MCP