See it as a diagram
Everything below, as a diagram you can edit. Describe yours and see it in seconds.
No account needed · Editable canvas, not a picture
The five artifacts, and nothing else
One, the layer map. A single diagram showing sources, the bronze, silver, and gold zones, the compute that moves data between them, and the consumers. This is the page everyone opens first and the only one most people ever need.
Two, the object inventory. Which catalogs and schemas exist, what each one is for, and who owns it. This is a table, it is boring, and it is the thing that stops a new team from creating a seventh gold schema.
Three, a contract per published table. Grain sentence, owner, freshness expectation, and the columns consumers may depend on. Only for gold tables that other teams read, which is usually a couple of dozen rather than hundreds.
Four, lineage from source to gold. Generated, never written. Its job is to answer what breaks if this column changes, and a hand-drawn version answers that question wrongly within a month.
Five, the reprocessing runbook. How to rebuild silver from bronze and gold from silver, what the safe order is, and what to tell consumers while it runs. This is the artifact people write during the first incident and should have written before it.
Anything not on this list is optional. Notebooks, design records, and ADRs are useful, but they document decisions rather than the current state, and they should not be maintained as if they described the platform today.
Why lakehouse docs rot faster than warehouse docs
Schema on read removes the moment where a change forces a conversation. In a classic warehouse, a column change is a DDL statement in a migration that somebody reviews. In a lakehouse, a new field can appear in a bronze table because an upstream producer added it, and nothing in the documentation loop notices.
Notebooks bypass the pipeline. A gold table created from a notebook by an analyst under deadline is indistinguishable, to a consumer, from a governed one. It has no owner, no test, and no entry in any inventory, and it will be discovered when it breaks. Tooling does not rescue you here either: Datadef repository sync classifies .ipynb as a binary format and skips it outright, so a transformation that exists only in a notebook is invisible to the generated diagram in exactly the way it is invisible to your catalog.
The platform itself moves. Table formats gain features, engines change defaults, and the recommended layout for storage credentials or external locations shifts between releases. Documentation written against last year platform is not wrong because anyone was careless.
The consequence is that hand-written documentation of current state has a half life measured in weeks. Documentation of intent, such as why bronze holds no business logic, lasts years. Sorting artifacts into those two buckets is most of the work.
Generate what can be generated
Model-level documentation comes from the transformation project itself: descriptions, tests, and dependencies live next to the SQL and are published from it. Table-level metadata comes from the catalog, which already knows what exists and who owns it. Neither of those should be retyped into a wiki.
The architecture diagram is the artifact teams assume must be hand-drawn, and it is the one that goes stale fastest. Connect the repository that declares the platform, whether that is the Terraform that provisions catalogs and storage or the dbt project that builds the layers, choose a branch or tag, and Datadef regenerates the diagram and an architecture.md on a daily sync. Commits that change nothing structural do not redraw, and nodes you positioned by hand keep their positions, so the map stays recognizable rather than reshuffling every morning.
Know what it read, though, because a generated artifact that quietly omits things is worse than a hand-drawn one. The corpus is capped at 40 files, 250KB in total, and 30KB per file, with per-class ceilings so infrastructure code, container files, CI workflows, and models each get a share instead of one class taking the budget. On a large platform repository that means the diagram is drawn from the files that carry the shape plus a summary of the tree around them, which is the right input for a layer map and the wrong one for an inventory. The inventory is artifact two, and it comes from the catalog.
What remains hand-written is small: the purpose sentence per layer, the contracts for published tables, and the runbook. That is a page or two, which a team can actually keep true.
## Published table contract | Field | Value | | --- | --- | | Table | prod.gold.fct_subscription_events | | Grain | one row per subscription, per state change | | Owner | revenue-data (group) | | Freshness | daily by 06:00 UTC, alert at 08:00 | | Stable columns | subscription_id, event_type, event_at, mrr_delta | | Deprecation | 30 days notice in #data-announcements |
Publish where people already read
Documentation nobody can find is documentation nobody trusts. The layer map should appear in the platform wiki, in the repository README, and in the onboarding page, and it should be the same map in all three. A live embed is one markdown line that renders the current diagram wherever markdown or an image is accepted, including Confluence, Notion, and a GitHub README, and viewers need no account.
That single detail changes the maintenance economics. When the diagram is embedded rather than exported, updating it once updates it everywhere, and the usual failure of three different screenshots in three different pages disappears. Export to PNG or JPEG stays available for slides, where a live image is not an option.
Make the runbook the only thing you write during an incident
FAQ
What documentation does a lakehouse actually need?
Why does lakehouse documentation go stale faster than warehouse documentation?
Should the architecture diagram be drawn by hand or generated?
How many gold tables need a written contract?
Where should the documentation live?