Data Lineage Guide

How to keep data lineage current instead of redrawing it every quarter

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

Every team that has documented lineage by hand knows the shape of the failure: three weeks of work, six months of slow decay, then somebody makes a decision from a picture that stopped being true in March. The fix is not more discipline. It is placing each part of the map where the change happens, and being explicit about which parts are generated, which are declared, and which rest on somebody remembering.

Try it free
7 min readFor teams whose lineage documentation decays faster than they can redraw it

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

167/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

Discipline is not the fix, placement is

Documentation goes stale when nothing in the change process points at it. An engineer renaming a column touches the model, the tests, and a pull request. If the lineage map is not in any of those three places, keeping it current depends on someone remembering, and remembering is not a control.

So the useful question is not how to make people update the map, it is which artifact sits close enough to the change to be updated as a side effect of doing the work. Everything that cannot be moved there needs an owner and a review date instead, and needs to be honest about it.

This is also why maps grow stale unevenly. The parts derived from code stay right for free. The parts declared in YAML stay right if reviewers care. The parts drawn by hand for systems outside git decay fastest and are usually the most business-critical hops on the picture.

Three tiers of freshness

Tier one, generated from code. Anything whose definition lives in a repository can be regenerated instead of maintained. Connect the repo to Datadef read-only on GitHub, GitLab, or Azure DevOps, pick a branch or tag, and a daily sweep regenerates the diagram and an architecture.md, a link falling due 23 hours after its last sync. Before generating anything the sync hashes what actually feeds the diagram, and an unchanged hash skips generation outright, so a commit touching a README, a test, or a comment costs nothing and redraws nothing. A sync can also be triggered on demand, from the UI or from an MCP client with repo_refresh when your coding agent finishes a change.

Tier two, declared next to code. Consumers, owners, and boundary hops that no parser can infer belong in files that live with the project, such as dbt exposures or a small YAML file listing external consumers. They stay current because changing them is part of the pull request, and reviewers see them.

Tier three, hand drawn. Legacy systems, vendor tools, and manual processes have no source to parse. These hops need a named owner and a review cadence tied to something real, such as a quarterly close or a release train, and they need the review date shown on the artifact.

<!-- README.md: the diagram updates with the canvas, no re-upload -->
![Pipeline architecture](https://datadef.io/api/embed/my-pipeline-a1b2c3d4)

Last structural change: see the sync date on the diagram page.
Hand-drawn hops (SFTP export, finance macro): reviewed quarterly by Finance Data.

Make the age visible

A diagram with no date is assumed current, and that assumption is the actual danger. Anyone reading a picture with a visible last-updated date applies the right discount to it, and a stale date is a prompt rather than a trap.

The strongest version of this is an artifact whose age is structural rather than typed: a live embed that renders the current canvas, so the README, the Notion page, and the Confluence page cannot show a different version from each other. Nobody re-exports an image, so nobody forgets to, and viewers need no account to see it.

Pair that with a freshness expectation per tier. Generated hops are current by construction. Declared hops are current as of the last merge. Hand-drawn hops are current as of their review date. Three sentences on the page, and readers stop treating the whole map with the same trust or the same suspicion.

Regeneration keeps your layout

The usual objection to regenerated diagrams is losing the arrangement you made for humans. Positions you set by hand are preserved across syncs, and unchanged commits do not trigger a redraw. See how repository sync works.

What a regeneration is allowed to change

The reason teams stop regenerating diagrams is rarely accuracy. It is churn: yesterday the map had four zones, today it has six with different names, and the reader has to learn it again. Three rules keep a re-sync a delta rather than a new drawing.

Identity first. Every node carries a source id, derived from the Terraform address or reused from the previous run, and anything that still exists in the source keeps its id, its label, its type, and its zone. The previous diagram is handed back to the generator as an explicit instruction to reuse what survives and to mint a new id only for something genuinely new, with up to sixty existing nodes listed by name.

Then geometry. Positions you set by hand are pinned. The sync records where it left each node, compares against the canvas on the next run, and treats a difference of more than twelve pixels as a deliberate move, restoring exactly those coordinates after the new layout runs. A diagram nobody touched stays free to re-lay out; a diagram somebody arranged for a review holds still.

And what you added yourself survives. Nodes on the canvas with no source id are treated as work a person did by hand and kept, so annotating a generated map does not mean losing the annotation tomorrow.

# what the daily sweep writes to the log
[reposync] daily sweep: 12 due, 3 regenerated, 9 skipped, 0 failed

# the nine skipped had an unchanged structure hash:
#   READMEs, tests, comments, formatting -> no redraw
# for the three that changed, the sync also records how much
# of the previous diagram survived, so a small source change
# that returns a low ratio is visible as a problem.

FAQ

How often should data lineage be regenerated?

Generated lineage should refresh at least daily so it never lags a release, and on demand when a large change lands. Hand-drawn parts should be reviewed on a cadence tied to a real event, such as a quarterly close or a migration milestone, since a calendar reminder alone tends to be ignored.

What if part of my pipeline is not in git?

Draw those hops by hand and mark them differently, with a named owner and a review date on each. Mixing generated and hand-drawn hops on one map is normal and usually unavoidable, since vendor tools and manual processes have no source to parse. Hiding which is which is what makes the whole map untrustworthy, because a reader then applies one discount to every hop.

Does regenerating a diagram destroy manual layout work?

It should not. Node identities derive from the source, so a re-sync updates the picture in place rather than redrawing it. The sync records where it left each node and compares on the next run, treating a move of more than twelve pixels as deliberate and restoring those coordinates after layout. Commits that change nothing structural do not trigger a redraw at all.

How do I tell whether a lineage map is stale?

Show the last update date on the artifact itself and compare it with the last structural change in the repository. If the map predates the most recent architectural commit and nothing regenerated it, treat it as out of date until verified.

Who should own keeping lineage current?

The team that owns the flow, not a central documentation role. Central ownership decays because the owner is never the person making the change, while flow owners update the map as part of shipping and notice immediately when it is wrong.