Data Pipeline Guide

How to document an ELT pipeline: the transform moved, so the docs move too

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

In a classic ETL pipeline the interesting logic lived in a tool with its own repository and its own diagrams. In ELT it lives in SQL inside the warehouse, spread across a few hundred models, and the documentation problem changes shape with it. There is less to say about the ingestion, which is usually a managed connector, and far more to say about the model layers, the tests, and the contracts with the sources. Here is what to write, what to draw, and where to keep both.

Try it free
7 min readFor analytics engineering teams whose transform logic lives in the warehouse

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

221/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

The one visual difference between ETL and ELT

On a diagram the whole distinction is where the transform box sits. In ETL it sits between the source and the warehouse, as its own zone with its own compute, and the warehouse receives finished tables. In ELT it sits inside the warehouse boundary, because the warehouse is the compute, and the raw landing tables are inside the boundary too.

That placement changes what the diagram is about. An ETL diagram is mostly about the movement, so the tooling and the hops are the subject. An ELT diagram is mostly about the model layers, so the subject is the shape of the transformation DAG and the ingest side compresses to a handful of connector nodes.

The practical consequence for documentation is that an ELT pipeline needs fewer words about how data arrives and many more about what the layers mean, which model owns which business definition, and what the tests guarantee.

What to write, in the order people need it

Start with the source contracts, one short block per source: the objects extracted, the cadence, the schema owner, the expected volume, and whether the source carries personal data. This is the section that gets read during an incident, and it is the section that becomes worthless fastest, so keep it short enough that updating it is not a project.

Then the layer definitions. Say what staging is for, what intermediate is for, and what marts are for, give the naming rule for each so a reader can predict a model name from a table name, and state the default materialization per layer. dbt ships five built-ins, view, table, incremental, ephemeral, and materialized_view, and a layer whose default nobody wrote down is a layer where cost gets decided one pull request at a time. Analytics teams that skip this end up with the same business logic implemented at three different layers.

Then the tests and the failure behaviour. Which tests are blocking, which are warnings, where failed rows go, and who is notified. A test that fails silently is worse than no test, and documenting the severity is what stops that.

Finally the consumers and their freshness commitments. Name each dashboard, sync, or model that depends on a mart, and state the freshness the consumer is promised. That list is also your impact analysis when someone proposes changing a mart.

## Source: Salesforce
- Objects: Opportunity, Account, Contact
- Extraction: managed connector, bulk API, nightly 01:00 UTC
- Schema owner: RevOps (schema changes announced in #revops-changes)
- Volume: ~2M rows/night
- Personal data: yes (Contact.email, Contact.phone)

## Layers
- staging: one model per source object, renamed and typed, no joins
- intermediate: reusable joins and business logic, not exposed to BI
- marts: consumer facing, one folder per domain, documented columns

## Tests
- unique + not_null on every staging primary key (blocking)
- accepted_values on status columns (warning)
- Failed rows land in ANALYTICS.QUARANTINE and page #data-oncall after 2 runs

Keep the doc and the diagram beside the code

Documentation that lives in a separate wiki decays on a predictable schedule, because the person changing the model is not the person opening the wiki. Keeping the written doc in the repository next to the models means it shows up in the same pull request as the change, which is the only mechanism that reliably keeps it honest.

The diagram should live in the same place, which means it has to be a URL rather than an exported image file. Connect the repository to Datadef with read only access on GitHub, GitLab, or Azure DevOps, pick the branch, and a daily sync regenerates the diagram and an architecture.md from what is actually in that branch. A commit that changes nothing structural skips generation altogether, and a node counts as hand-moved once it sits more than 12 pixels from where the previous sync left it, which is what lets an arranged canvas hold still while an untouched one keeps re-laying out.

It is worth knowing what that sync reads before relying on it. It assembles a corpus rather than ingesting the project: at most 40 files, 250KB in total, 30KB per file with the remainder truncated behind a marker, and a per-class cap so no single directory floods the budget. Under the architecture weighting a dbt project contributes at most three model files and one dbt_project.yml, next to as many as 16 infrastructure files and 10 CI files; under the content weighting the balance flips to 18 model files and 14 other SQL files. A four hundred model project is never read model by model in either case, which is exactly why the layer definitions and naming rules above have to be written rather than inferred.

Then embed the diagram in the repository README with one markdown image line. Viewers need no account, and the image follows the canvas, so the README stops being a snapshot of the pipeline as it was when someone last exported a PNG. The same live embed line works in Notion and Confluence for the audiences who never open the repo.

One pull request, both artifacts

When the doc and the diagram both come from the connected branch, a model change and its documentation cannot drift apart quietly. See repository sync.

FAQ

How do you document an ELT pipeline?

Write the source contracts first, one short block per source with objects, cadence, schema owner, volume, and whether personal data is involved. Then define what each model layer is for and its naming rule, then the tests with their severity and where failed rows go, then the consumers with the freshness each is promised. Keep all of it in the repository beside the models.

What is the difference between documenting ETL and ELT?

ETL documentation is mostly about the movement and the transformation tool, because that is where the logic lives. ELT documentation is mostly about the model layers inside the warehouse, because ingestion is usually a managed connector with little to explain and the transformation is a few hundred SQL models that need definitions, naming rules, and tests documented.

Where should ELT pipeline documentation live?

In the repository next to the models, so it appears in the same pull request as the change that makes it wrong. A separate wiki decays predictably because the person editing the model is not the person opening the wiki, and nothing in the review process forces the two to meet.

How do you keep the pipeline diagram from going stale?

Generate it from the connected repository rather than drawing it once. A read only connection to GitHub, GitLab, or Azure DevOps with a chosen branch or tag can regenerate the diagram and an architecture document daily, and embedding the live image in the README means readers always see the current version instead of an exported screenshot.

Should the documentation list every model?

No. A per model list is what the dbt catalog is for, and duplicating it by hand guarantees drift. The written documentation should cover layer definitions, naming rules, source contracts, test severity, and consumers, which are the things the generated catalog cannot express.