Data Lineage Guide

What to capture in data lineage, and what to leave out

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

Lineage projects rarely fail on technology. They fail on scope: a team decides to capture everything, discovers that everything includes forty years of stored procedures, and ships a graph that is 60 percent complete and therefore trusted by nobody. Deciding what not to capture is the design work, and it is done before the first connector is configured.

Try it free
7 min readFor data leads writing the scope of a lineage implementation

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

138/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

Write down the three questions first

Lineage has no intrinsic definition of done, so it needs a borrowed one. Write the specific questions the graph must answer, with names in them. Not "improve data governance" but "prove how the figure in the quarterly regulatory report was derived", "tell us what breaks when we change the customer table", "settle why the finance dashboard and the sales dashboard report different revenue".

Three questions is usually the right number. Each one names a small set of flows and a small set of fields, and together they define a scope that finishes. It is also the artifact you show when someone proposes adding another twenty source systems: either the addition answers one of the three questions, or it goes on the list for later.

Questions also determine grain. An audit question needs column-level detail on the audited fields and nothing else. A migration question needs complete downstream coverage at table grain and no column detail at all. Buying one grain for the whole estate is how budgets are spent on precision nobody asked for.

Capture rules that survive contact with a real warehouse

Capture end to end before capturing deeply. A shallow path from source system to consumed surface is more useful than a perfect subgraph in the middle, because the questions people ask start at one end and finish at the other.

Capture critical data elements at column grain. In most organisations these are the fields in regulated reports, the revenue and cost measures, and the identifiers used to join across domains. Ten to fifty fields, not thousands. Everything else stays at table grain unless a question forces otherwise.

Capture every boundary crossing. Where data moves between systems, between teams, or out of the warehouse into a BI calculation, an export, or a reverse ETL sync, those hops break most often and are exactly where automated discovery is weakest. Missing them produces a graph that looks complete and answers the change-review question wrongly.

Capture an owner at every hop. A hop without a named owner cannot be acted on, and an unactionable graph is a reference document rather than a control.

Lineage scope statement

  Q1  Prove derivation of REG-114 revenue figure
      grain:   column level on 12 audited fields
      flows:   billing -> warehouse -> reg mart -> report

  Q2  Blast radius for changes to dim_customers
      grain:   table level, complete downstream
      flows:   all consumers incl. dashboards + CRM sync

  Q3  Reconcile finance vs sales revenue
      grain:   column level on 2 revenue measures
      flows:   both dashboards back to shared source

  Out of scope: sandbox schemas, ad hoc analyst tables,
  1:1 staging copies, column detail on wide dimensions.

What to leave out on purpose

One-to-one staging copies. A raw table copied to a staging table with no transformation adds a node and no information. Collapse the pair and note that a raw layer exists.

Temporary and sandbox objects. Analyst scratch tables, backup copies with a date in the name, and anything in a personal schema inflate the graph and rot fastest. Excluding them by naming pattern is one of the highest-value filters available.

Column detail on wide dimensions. A hundred-column customer dimension does not need a hundred derivations recorded. The identifiers and the fields used in reported measures do.

Anything you cannot maintain. Captured lineage that is never corrected becomes wrong lineage, which is worse than absent lineage because people act on it. If no owner can be named for maintaining a region of the graph, that region should not be in scope yet.

On a Datadef canvas the mixing is per link: canvas_add_lineage takes column names when the field detail matters and omits them when it does not, so one map carries audited fields at column grain and the surrounding pipeline at table grain without splitting into two artifacts. The call takes up to a hundred links at once, which is a real mart in a single pass, and it resolves every column name against the columns already declared on the node, so a field that does not exist is refused with the list of the ones that do rather than quietly invented.

Definition of done

A flow is done when someone outside the data team can answer the question it exists for without asking a data engineer. Not when every object is connected, not when the graph is complete, and not when the scan finishes. That test is easy to run: hand the artifact to the auditor, the finance lead, or the new joiner and watch where they get stuck.

Everything they got stuck on is the next unit of work, and everything they walked past without needing was scope you can be glad you skipped. Re-run the test after each of the three questions, and the project ends on a date rather than trailing off.

Grain is a per-edge decision

Column grain on the audited fields, table grain elsewhere, owners everywhere. See what column-level lineage actually shows for where the extra detail earns its maintenance cost.

FAQ

How do I choose critical data elements for lineage?

Start from the reports and decisions that carry consequences: regulated figures, revenue and cost measures, and the identifiers used to join across domains. In most organisations that is ten to fifty fields. If a field being wrong would not trigger an escalation, it does not need column-level lineage.

Should lineage be captured at column level everywhere?

No. Column grain multiplies the graph size and the maintenance cost, and the detail is only actionable where a specific field is disputed, audited, or used as a join key. Table grain everywhere else answers the scoping and impact questions perfectly well.

How far upstream should lineage go?

To the system where the data is created, not to the first table you happen to control. Stopping at the raw layer hides the ingestion hops where lateness and corruption usually originate, and those are the hops incident reviews keep returning to.

Should the BI layer be included in lineage scope?

Yes, at least the named consumers and the calculations that redefine a metric after the warehouse. A large share of disputed numbers is computed in a dashboard tile or a spreadsheet, so a graph that stops at the mart cannot settle the argument it was bought for.

When is a lineage implementation finished?

When the questions it was scoped around can be answered by someone outside the data team without asking an engineer. Completeness of the graph is not the finish line, because a graph can be complete and still fail to answer the question that funded it.