When data lineage is overkill, and what to do instead

One warehouse, one transformation project, one team that both produces and consumes, a schema that changes a few times a quarter, and a model count someone can still hold in their head. In that shape, the people are the lineage. Tracing a broken metric is a ten-minute job because the person who built it is in the room, and the automated graph would confirm what they already knew.

The cost side is not zero either. A lineage platform brings connector configuration, scan scheduling, an access model, and a stewardship queue of wrong or missing edges to correct. For a team of three that is a meaningful fraction of one person, spent to answer a question that currently takes ten minutes.

What that team does need is the thing lineage is usually a proxy for: an owner per critical table, naming that makes the flow obvious, and one diagram of the stack that a new joiner can read. Those cost an afternoon and remove most of the pain people buy lineage to remove.

A second producing team. The moment data is created by people you do not sit with, the informal channel breaks and the cost of a wrong assumption starts landing on someone else.

Consumers you cannot enumerate. If nobody can list what reads a core table without asking around, impact analysis has already failed, and every schema change is a gamble.

A regulated or externally reported number. Auditors want a traceable path from the reported figure to the source records, and reconstructing it after the fact costs far more than capturing it as you go.

A migration. Replatforming a source, a warehouse, or a BI tool turns unknown dependencies into an outage, and a migration is the single most common trigger for a lineage purchase that gets approved instantly.

Repeat incidents with the same shape. If more than one incident this quarter was somebody changing something without knowing who depended on it, the informal system is already over capacity.

Score it once a quarter

  [ ] data produced by a team we do not sit with
  [ ] nobody can list what reads our three busiest tables
  [ ] a figure we report externally or to a regulator
  [ ] a source, warehouse, or BI migration booked this year
  [ ] two or more incidents this quarter caused by an
      unknown dependency

  0-1  owners and one diagram; score again next quarter
  2-3  document the critical flows by hand now
  4-5  the informal system is over capacity, and the
       business case will approve itself

Column-level detail for the regulated fields only. Ten to thirty fields carry the numbers people argue about. Record their derivations properly and keep everything else at table grain. That is one afternoon and covers most audit questions.

Owners on the hubs. Find the handful of tables with many downstream consumers and put a named owner and a team on each. Most of the value teams attribute to lineage is really the routing it enables.

A contract on the tables everyone reads. Two or three widely consumed tables with an agreed schema and a not-null test on the join keys prevent more incidents than a complete graph does, because they fail in CI rather than in a dashboard.

One readable map, placed where people work. Datadef does not crawl your warehouse or its query logs, deliberately: the output is a curated map a finance analyst can read rather than ten thousand harvested edges nobody opens. The canvas holds that line on purpose, warning past thirty data nodes that a bigger picture gets searched rather than read. Draw it on the free canvas with no account, and on paid plans embed it live in the wiki so the version people see is the current one.

The failure that follows a justified purchase is the same one that follows an unjustified one, and it is scope. A programme that connects every system before answering a single question bills for two quarters and returns nothing, while the migration that triggered it happens anyway.

Connect the systems on the path of the question that got the budget approved, and nothing else. For most teams that is the warehouse, the transformation layer, and one BI tool, which is a fraction of the estate and answers the audit or the migration question in weeks rather than quarters. The rest of the estate can be added once something is already working, when its cost is being weighed against a benefit somebody has seen.

And keep the manual work. The flows you documented, the owners you named, and the boundary hops you found are the inputs a rollout needs, and they are the parts scanning cannot infer. Teams that discard the manual pass on purchase day repeat it six months later under a different name.

• What data lineage costs
• What to capture in data lineage
• Data contracts
• Data lineage on one canvas