dbt and Spark jobs on a diagram: group the models, draw the job not the plan

The dbt DAG is a real dependency graph and it is genuinely useful for tracing one model. As a picture of the project it fails past roughly eighty nodes, because every model is drawn at equal weight and the folder structure that gives the project its meaning is invisible. The fix is to draw the folders, not the models.

Make each layer a zone: sources, staging, intermediate, marts. Put a count on each zone, for example staging with 61 models. Inside a zone, name only the models people say out loud, which is usually a handful of marts and the two or three intermediate models everything depends on. Everything else is a count.

Put the materialization on the nodes you do name. dbt ships five built-ins, view, table, incremental, ephemeral, and materialized_view, and materialization is the single property that determines cost and freshness behaviour, so it carries more information per character than anything else you can write. An incremental model with a stated unique key and strategy tells a reader almost everything about how it behaves. Ephemeral deserves a note of its own: it never exists in the warehouse, dbt inlines it as a CTE into whatever references it, so a box drawn for one is a box for an object nobody can query.

Use the declarations dbt already makes for the two ends. Sources are your sources and exposures are your sinks: each exposure carries a type of dashboard, notebook, analysis, ml, or application, an owner with a name or an email, a depends_on list of ref, source, and metric references, and optional url and maturity of high, medium, or low. A project that declares exposures has already written down which sink depends on which mart, so the right hand edge of the diagram is read out of the project rather than invented.

The scope here is the architecture picture, where a layer is a zone with a count on it. Tracing one model back to the raw table it came from, or one field to the column it was derived from, is a different artifact with a different reader: see data lineage for dbt projects.

ZONE sources        14 declared sources across 4 systems
ZONE staging        61 models, all views, 1:1 with source objects
ZONE intermediate   12 models
  int_orders_enriched      (table)      <- depended on by 9 marts
  int_customer_identity    (incremental, unique_key: customer_id)
ZONE marts          23 models
  fct_revenue_daily        (incremental, merge, unique_key: date_day)
  dim_customer             (table)
EXPOSURES  Finance board (Looker) | CRM sync (Hightouch) | churn_model

The Spark UI DAG shows stages, shuffles, and tasks for one run. It is the right tool for finding a skewed join and the wrong tool for explaining a pipeline, because its nodes are physical operations chosen by the optimizer and they can change between runs of the same code.

Draw the job as one node instead, annotated with the four things an operator needs: the input table or path, the output table or path, what triggers it, and the compute it runs on. A node reading enrich_orders, reads lake.bronze_orders, writes lake.silver_orders, triggered by the core DAG at 02:15, runs on the etl-medium pool answers more questions than a hundred stage boxes.

For Structured Streaming jobs add two more annotations: the trigger and the checkpoint location. The trigger is one of processingTime with an interval, availableNow, or continuous, and Trigger.Once has been deprecated since Spark 3.4 in favour of Trigger.AvailableNow, so a node still annotated "once" usually marks code nobody has revisited. The checkpointLocation path is the other, because it is the first thing anyone asks for when a streaming job is stuck or has to be restarted, and it is almost never written down anywhere findable at speed.

When several jobs form a stage of the pipeline, group them in a zone by what they produce rather than by the cluster they run on. Readers are trying to trace data, and a cluster is a billing detail unless the diagram is specifically about cost.

Transformation layers change more often than infrastructure does. A dbt project can gain a dozen models in a sprint, which means a hand drawn transformation diagram has a shelf life measured in weeks. Generation has to be part of the workflow rather than an occasional project.

Two mechanisms work. Connect the repository read only, pick the branch, and let the daily sync regenerate the diagram and an architecture.md; a commit that changes nothing structural skips generation entirely, and a node counts as hand-positioned once it sits more than 12 pixels from where the last sync left it, so the canvas stays familiar. Or drive it from the editor: an agent with the MCP server connected reads the project it already has open and calls create_diagram plus the canvas_* tools, 36 in total across the two tiers, to update the picture in the same session as the model change.

Then publish it once and stop exporting. The live embed line renders the current diagram in the project README, in Notion, or in Confluence, and readers need no account. PNG and JPEG export stays available for slides, which is the one place a frozen image is the right answer.

• dbt transformation diagram generator
• Living diagram from dbt
• Keep dbt docs in sync
• Airflow DAG diagram conventions
• Data lineage for dbt projects