What data lineage costs, and why the bill lands where it does

Connector coverage comes first. Every source system, transformation engine, and BI tool is separate engineering on the vendor side, which is why quotes rise sharply when a mainframe, an on-premises database, or a niche BI tool is in scope. It is also why the tool that looks cheapest on a modern cloud stack can be the most expensive once the legacy half of the estate is included.

Scanning and compute come second. Automated lineage is produced by repeatedly reading metadata and parsing query history, and that work recurs. Data volume, object count, and how frequently you want the graph refreshed all move this number, and on cloud-metered platforms it lands on your own bill rather than the vendor invoice.

Seats come third. A tool used by a four-person compliance team is priced differently from the same tool rolled out to every analyst and steward. Fourth is implementation: integration, mapping, and the services engagement that gets the first useful graph out. Fifth, and the one that gets left out of business cases, is stewardship time: reviewing what was captured, correcting wrong edges, and declaring the hops automation cannot see.

Time to first answer is a real cost and nobody lines it up against the licence. A programme that spends two quarters connecting sources before it can answer a single question has been billing the whole time and has returned nothing, while the audit that triggered the purchase came and went. Teams that get value early almost always did it by narrowing scope first and connecting the rest afterwards.

Lock-in is the other quiet line item. Lineage that exists only inside one vendor graph, in a proprietary shape, has an exit cost, and that cost sits on the vendor side of the table at every renewal. Lineage that is declared in your repository, in your dbt project, or in a diagram you own moves with you.

The third invisible cost is coverage that reads as complete but is not. A graph missing the BI layer, exports, and reverse ETL still renders a confident picture, and teams make merge decisions against it. The incident that follows is a lineage cost even though it never appears in the lineage budget.

The recurring line that surprises people is scanning, so it is worth knowing what a git-based alternative does instead of it. Datadef reads the repository, not the warehouse: a read-only connection to GitHub, GitLab, or Azure DevOps on a chosen branch or tag, with Terraform parsed file by file and no terraform init, no state file, and no cloud credentials handed over. There is no metastore to scan and no query history to parse, so the per-object and per-volume drivers do not apply at all.

The corpus is capped rather than exhaustive, which is what makes a daily job cheap and predictable: at most 40 files, 250KB in total, 30KB per file, anything longer cut at a truncation marker. Per-class ceilings stop one noisy class from taking the whole budget, so a repository holding 400 dbt models contributes twelve of them under the balanced default and three when the sync is set to an architecture focus, alongside the infrastructure and CI files that carry the shape.

The saving nobody prices is that most syncs do nothing. Each run hashes what actually feeds the diagram, and a commit that touched a README, a test, or a comment produces the same hash, so generation is skipped outright. On a normal repository that is most days, which is the difference between a fixed recurring cost and a cost that follows real architectural change.

Three-year lineage cost worksheet

  licence            vendor quote x 3 years
  connectors         list the systems in scope; legacy and
                     niche ones move the quote most
  scan compute       object count x refresh frequency,
                     billed to YOUR cloud account
  seats              compliance team, or every analyst
  rollout            integration + services, year one
  stewardship        hours/month correcting captured edges,
                     x 36
  exit               cost of re-declaring lineage elsewhere
  ---------------------------------------------------------
  baseline to beat   hours/month currently spent tracing a
                     broken metric by hand, x 36

Scope by question. Two or three questions imply five to fifteen flows, and those flows can be documented and kept current for a fraction of the effort of a full-estate rollout. Add automated discovery when the number of objects outruns what people can maintain, which is a threshold you can measure rather than guess.

Generate what can be generated. Where the definitions live in git, the map can be regenerated from the code instead of maintained by a person, which removes the recurring compute driver entirely for that part of the map.

Draw the rest once, deliberately. A curated lineage canvas costs an afternoon and answers the audit question that the un-launched platform would have answered next quarter. Datadef starts with a seven-day trial that needs no card, the free scratch canvas needs no account at all, and live embeds and repository sync sit on the paid plans.

• Best data lineage tools
• When data lineage is overkill
• What to capture in data lineage
• Generate diagrams from a repository