Docs & Code Guide

Keep pipeline docs in sync with the DAG: the scheduler already knows the truth

Every orchestrated pipeline has a complete, current description of itself: the DAG the scheduler parses before every run. Hand-drawn pipeline charts are transcriptions of that graph, and transcriptions drift. This guide is about the keep-current loop; for what a good pipeline doc contains in the first place, start with data pipeline documentation.

Try it free
7 min readFor data engineers whose pipeline diagram was accurate two quarters ago

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

245/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

The scheduler holds the only graph that cannot lie

Airflow parses your DAG files continuously and will refuse to run anything that is not in them. Dagster builds its asset graph from the definitions in code and draws it in the UI on every load. Whatever a wiki page claims, the scheduler metadata is the pipeline: task names, dependencies, schedules, retries, all resolved from the code that actually executes.

That makes pipeline documentation an unusual case. The problem is not discovering the truth, which is sitting in the scheduler, but the gap between that truth and the diagram people actually read: the one in the onboarding doc, the architecture page, the incident channel. The keep-in-sync problem is a transcription problem.

What hand-drawn pipeline charts get wrong within weeks

The decay pattern is predictable. First the names go: a task gets renamed in a refactor and the chart keeps the old label, so the on-call engineer greps for a task that no longer exists. Then the structure: a sensor or a backfill branch is added and never drawn, so the chart shows a linear flow where the real DAG fans out. Then whole subgraphs: a new source lands, a deprecated branch keeps its box for a year because deleting from a drawing feels like losing work.

Each error is small. The compound effect is that engineers stop trusting the chart and read the DAG file instead, at which point the documentation has negative value: it costs maintenance and misleads the people who do not know to distrust it.

Render from the scheduler when you need a quick answer

For a fast, guaranteed-accurate picture of a single DAG, ask the scheduler. Airflow renders the parsed graph to an image from the CLI, and the Graph view in the UI shows the same thing per run. Dagster draws the asset graph in its UI, including upstream and downstream assets across jobs.

These renders have a ceiling, and it is worth being clear about it. They are per-DAG, so they cannot show the system around the pipeline: the S3 buckets, the warehouse zones, the CRM at the end of the reverse ETL. They carry no annotations for the reader, and they live in a terminal or an orchestrator UI rather than in the docs your team reads. They are the accurate sketch, not the published diagram.

# Airflow renders the DAG it will actually run (requires graphviz)
airflow dags show orders_nightly --save orders_nightly.png

# List what the scheduler has actually parsed, refactors included
airflow dags list

The keep-current loop: regenerate and embed

The published diagram gets the same treatment as the code: it is regenerated from the source when the source changes. Datadef ships an MCP server, registry name io.datadef/mcp, so an agent like Claude Code or Cursor connected to it can read the dags/ directory after a merge and update the Datadef diagram to match the new structure, keeping the system context and annotations that the scheduler render cannot carry. The first version takes minutes: paste a DAG file or a plain description into the Airflow DAG diagram generator and edit from there. The Airflow-specific version of the loop is detailed in living diagram from Airflow.

Two honest mechanics. Datadef does not watch your repository: nothing detects the merge by itself, the regeneration runs when your CI step or your agent invokes it, which is one command in the pipeline that already runs your tests. And the published copy follows without touching: a project shared public serves its diagram at a permanent image URL that re-renders when the diagram changes, so the copy in your README or wiki is current within minutes of the edit. The agent authenticates with an API key created on a paid plan.

Where the anchor pages are

This page owns the keep-current loop. What to write in pipeline docs is covered by data pipeline documentation; the transform layer has its own loop in living diagram from dbt.

Scope: the diagram stays current, the prose is your job

A pipeline doc is a diagram plus prose: what the pipeline is for, what to do when it breaks, who owns it. The loop above keeps the diagram matching the DAG. It does nothing for the prose, and Datadef does not sync prose documentation; tools like Swimm target that problem. The practical division that works: generate the graph, write the intent, and review the prose whenever the regeneration diff shows the structure changed, because a structural change is exactly when the runbook paragraph is most likely to be wrong too.

For the wider practice across schemas, lineage and warehouse docs, keeping data documentation in sync covers the full surface.

FAQ

How do I keep data pipeline documentation up to date?

Stop transcribing the scheduler by hand. The DAG definition is the accurate graph, so render or generate the diagram from it and regenerate on merge: an agent connected through an MCP server can update the published diagram when the dags/ directory changes. Keep the prose parts, purpose and runbooks, on a review trigger tied to structural changes.

Can pipeline documentation be generated automatically from Airflow?

The graph can. Airflow renders any parsed DAG to an image with airflow dags show, and the Graph view in the UI is always current. What those renders lack is system context, annotations and a home in your docs, which is why teams generate a published diagram from the DAG source and regenerate it on merge instead of screenshotting the UI.

Why is a hand-drawn pipeline flow chart always wrong?

Because the pipeline changes through code review and the chart does not. Task renames, added sensors, new branches and removed sources all land without the chart in the diff. The decay starts with stale labels within weeks, and it ends with engineers reading the DAG file instead of the chart, which makes the chart worse than nothing.

Does this work for Dagster as well as Airflow?

Yes. Dagster builds its asset graph from code definitions and shows it in the UI, so it is the same situation: an accurate per-graph render in the tool, and a published system diagram that needs a regenerate-on-merge loop to stay current. The loop reads the definitions instead of dags/ but is otherwise identical.

Does the published diagram update by itself when the DAG changes?

No, and claims of full automation usually hide a scheduled job. Datadef does not watch repositories; regeneration runs when a CI step or an agent invokes it after the merge. What is automatic is propagation: once the diagram is edited, the embedded copy in your docs follows within minutes because it is served from a live image URL rather than a pasted file.