Data lineage for dbt projects: what the DAG gives you and what it does not

The dbt DAG is model-to-model lineage derived from the compiled project, with upstream models to the left and downstream models to the right. It is regenerated on every docs build, which means it never drifts from the code, and it is the reason dbt teams often have better internal lineage than teams with a purchased platform.

Sources declared in YAML pin the upstream boundary, so the graph knows that stg_orders comes from a specific raw table in a specific schema. That is the first thing to get complete, because a project where half the staging models select from hardcoded table names has a DAG that silently starts one layer too late.

Tests attached to models make the graph actionable rather than decorative. A not-null test on the join key is what turns a downstream edge into something CI can enforce, and it is the difference between knowing about a dependency and being warned when it breaks.

Column-level lineage is not in dbt Core. Model-to-model edges are, field-to-field derivations are not. dbt Labs ships column-level lineage in dbt Catalog, the product previously called Explorer, and its documentation (docs.getdbt.com/docs/explore/column-level-lineage) restricts it to the Enterprise plans. Below that tier it comes from a separate tool or from recording the derivations yourself, and for most projects the practical middle ground is to record column detail only for the fields in regulated or contested reports.

What happens before the source is invisible. The ingestion tool, the API, the CDC stream, and the landing bucket are all upstream of the first raw table, and the DAG starts after them. When the question is why the data is late or wrong, the answer is usually in that missing prefix.

What consumes the marts is invisible unless declared. Exposures exist precisely for this and are chronically underused. Without them, the graph ends at the mart and every dashboard, export, reverse ETL sync, and ML feature table downstream is a dependency nobody can enumerate during a change review.

# models/marts/exposures.yml
exposures:
  - name: revenue_dashboard
    type: dashboard
    maturity: high
    url: https://looker.internal/dashboards/42
    owner:
      name: Finance Analytics
      email: [email protected]
    depends_on:
      - ref('fct_revenue_daily')
      - ref('dim_customers')

  - name: crm_customer_sync
    type: application
    owner:
      name: RevOps
      email: [email protected]
    depends_on:
      - ref('dim_customers')

Past a few hundred nodes the DAG stops being a diagram and becomes an inventory. Everything is technically there and nothing can be read at a zoom level where labels are legible. The answer is not a better layout engine, it is a second artifact: a curated map per domain, showing the ingestion end, the handful of models that matter, and the named consumers, with the full DAG remaining the reference for engineers.

That domain map is the thing you show in a review, hand to a new analytics engineer, or attach to an audit response. It is also the one that has to carry the parts the DAG cannot see, which means the ingestion prefix and the consumer suffix drawn explicitly.

Connect the repository holding the dbt project to Datadef read-only, on GitHub, GitLab, or Azure DevOps, and the sync regenerates the diagram and an architecture.md daily from the chosen branch or tag. Commits that change nothing structural do not redraw the diagram, and nodes you moved by hand keep their positions across syncs, so the map you arranged for humans stays arranged. Embed it in the dbt docs site, the repo README, or a Confluence page with one markdown line and it updates with the canvas.

A diagram generated from a repository is only as honest as its file selection, so it is worth stating what gets read. Datadef walks the tree once and classifies: dbt_project.yml is its own class, every .sql file under a models directory is a dbt model, and migrations, SQL outside models, container files, CI workflows, and dependency manifests each get their own. Directories that never describe architecture are skipped, node_modules and .terraform among them, as are lockfiles and binary formats. Jupyter notebooks are on the skipped list, so a model that exists only in a notebook does not appear.

Then the caps apply: 40 files, 250KB in total, 30KB per file, longer files cut at a truncation marker. Per-class ceilings stop one class from taking the budget. Under the balanced default a project contributes at most twelve model files and two project files; set the sync to a content focus and models rise to eighteen with plain SQL at fourteen; set it to an architecture focus and models drop to three while infrastructure code rises to sixteen.

The consequence for a 400-model project is worth saying plainly. The diagram is drawn from a sample of the project plus the shape of the tree around it, not from every model. That is the right trade for an architecture picture and the wrong one for an inventory, which is what the DAG and the docs site already are.

What the selector takes from a dbt repo

  dbt_project.yml            always, its own class
  models/**/*.sql            12 files (default focus)
                             18 (content focus)
                              3 (architecture focus)
  migrations/**              sampled from BOTH ends: the
                             first files show the initial
                             schema, the last the direction
                             of change
  .github/workflows/*.yml    the jobs that run the project
  README, docs/architecture* the prose that names things

  never: node_modules, .terraform, lockfiles, .ipynb,
         anything over 400KB

• Keep dbt docs in sync
• A living diagram from dbt
• dbt best practices
• Column-level lineage explained
• Repo to diagram: repository sync