A living diagram from Airflow: dynamic DAGs mean you diagram what the scheduler parsed, not what the file says

A DAG file in a mature Airflow project rarely lists its tasks. It loops over a config of tables to sync, builds task groups from a dict, branches on environment variables. Run the same file against a different config and you get a different DAG, which is the point of writing it that way. Static parsing of the file therefore cannot tell you what runs tonight; only the parsing Airflow itself performs can.

Airflow keeps the result of that parsing: the DagBag in memory, and serialized DAG representations in the metadata database that the webserver reads. Anything that claims to show your real pipelines has to come from this parsed layer.

The CLI exposes exactly the artifact a diagram needs. airflow dags show <dag_id> prints the parsed DAG, tasks and dependencies as Airflow resolved them, in Graphviz DOT notation, and --save renders it to an image if graphviz is installed. Because the command goes through DAG parsing, dynamically generated tasks appear exactly as the scheduler sees them, config expansion included.

The DOT text is also a fine prompt: Datadef's AI import accepts pasted structured text, so the output of dags show is a working input for a first generation, before any agent wiring. The pure generation intent, one DAG into one picture, lives at the Airflow DAG diagram generator.

# The parsed graph, printed as Graphviz DOT
airflow dags show nightly_warehouse_load

# Rendered to an image (requires graphviz)
airflow dags show nightly_warehouse_load --save nightly_warehouse_load.png

Inside one DAG, Airflow's own graph view is genuinely good, and duplicating it adds nothing. The map readers lack is between DAGs: the TriggerDagRunOperator that chains the reporting DAG to the warehouse load, the ExternalTaskSensor waiting on another team's pipeline, dataset-driven scheduling (datasets since Airflow 2.4, renamed assets in Airflow 3) wiring producers to consumers implicitly.

Those edges are scattered across DAG files and invisible in any single DAG's view, which is why the platform-level diagram earns its place in the wiki: every pipeline as a node, trigger and sensor and dataset edges between them, grouped by team or domain. That is the diagram that answers "what happens downstream if tonight's load is late", and it pairs with the wider practice in data pipeline documentation.

DAG deployments are merges to the dags/ folder, so that is where the loop hooks in. A CI job on changes under dags/ parses the affected DAGs, airflow dags show per changed DAG id, and hands the output to an agent connected to Datadef's MCP server (registry io.datadef/mcp), which updates the pipeline diagram: new tasks in their groups, removed pipelines dropped, cross-DAG edges redrawn. The agent connection needs an API key, available on paid plans; setup is in the MCP diagram server guide.

Datadef does not watch the repository or your Airflow instance, and no diagram regenerates on its own. The loop is one job in the CI you already run on the DAG folder, and the wiki embed handles the display side: an updated diagram appears in the embedding page within minutes.

The parsed graph is a schedule-time truth, not a run-time one: branch operators choose paths per run, and mapped tasks expand per run, so the diagram shows the possible shape, not one execution. That is the right thing for a wiki, but say so in the caption.

The embed URL exists only for projects shared public, a real consideration when task names reveal internal systems. And the scope is the diagram: DAG docstrings, runbooks, and operational prose are out of scope here, covered instead by the workflow in keep pipeline docs in sync.

• Airflow DAG diagram generator
• Keep pipeline docs in sync
• Data pipeline documentation
• The MCP diagram server