See it as a diagram
Everything below, as a diagram you can edit. Describe yours and see it in seconds.
No account needed · Editable canvas, not a picture
1. The Documentation Drift Problem
Every data team has experienced this: a new engineer asks "what does this table do?" You send them to the wiki. They come back 10 minutes later: "The docs say this column is required, but it's null in 40% of rows." The docs were last updated 8 months ago.
The Vicious Cycle
Docs get out of sync → People stop trusting them → They stop using them → No one maintains them → They become even more outdated. Breaking this cycle requires changing how documentation is created and maintained, not just asking people to "please keep docs updated."
Why Documentation Drifts
Docs Live Separately from Code
Your documentation is in Confluence. Your schemas are in SQL. When you change the SQL, you have to remember to update the doc. You usually forget.
No Validation or Enforcement
There's no test that fails when docs are outdated. PRs get merged without documentation updates. Nobody notices until someone asks a question.
Manual Updates Don't Scale
With 500+ tables and 5 deploys a day, manually updating docs is impossible. Yet most teams still try.
Incentives Are Misaligned
Engineers are rewarded for shipping features fast, not for maintaining docs. Documentation becomes the thing you do "when you have time" (never).
The Solution
The only way to keep documentation in sync is to make it automatic. Extract metadata from your systems, integrate docs into your development workflow, and make staleness visible. Documentation should be a byproduct of building data products, not a separate task.
2. The Cost of Stale Documentation
Outdated docs aren't just annoying—they're expensive. Here's what happens when documentation drifts from reality.
Wasted Time
Engineers spend hours hunting through code and Slack to understand what data means instead of building.
Cost: 5-10 hours/week per engineer on a 10-person team = 50-100 hours/week
Wrong Assumptions
Analysts build dashboards based on outdated docs. They make incorrect assumptions about data meaning.
Cost: Bad decisions, lost stakeholder trust, rework
Slow Onboarding
New hires can't trust the docs, so they have to learn everything from senior engineers via Slack.
Cost: 2-3 months to productivity instead of 2-3 weeks
Duplicate Work
Teams rebuild tables that already exist because they can't find or understand existing assets.
Cost: Redundant pipelines, storage, and compute
Real-World Impact
A fintech company found:
After implementing automated documentation, time to onboard new analysts dropped from 12 weeks to 3 weeks. Their data team velocity increased by 40%.
A SaaS company discovered:
They had 3 different tables with "monthly revenue" that gave different numbers. Reason: no one knew which was canonical because docs were outdated.
From experience
I once spent 6 hours debugging a dashboard only to discover the docs said "revenue" but the field actually contained "gross merchandise value." The docs were written in 2019. The column definition changed in 2021. Good documentation could have saved that entire day.
3. Automation-First Approach
The key insight: documentation should be extracted, not written. Instead of asking engineers to maintain docs manually, pull metadata automatically from your systems and supplement it with human context.
The Three-Layer Documentation Model
Auto-Generated Metadata (85%)
Extracted automatically from your systems. Always current because it's pulled from the source of truth.
Embedded Context (10%)
Written by engineers in their code, extracted automatically during deployment.
Human Annotations (5%)
High-value context that can't be extracted. Add sparingly via a catalog UI.
How Automation Works in Practice
Schema Auto-Discovery
Data catalogs scan your warehouse every hour to detect schema changes. New columns appear in docs automatically. Deleted columns get marked deprecated.
Tools: Atlan, Alation, Collibra, Select Star
Lineage Auto-Tracking
Parse SQL in your pipelines to build lineage graphs. When someone changes a transformation, lineage updates automatically.
Tools: dbt lineage, SQLLineage, DataHub, OpenLineage
Docs-as-Code
Write descriptions in YAML files next to your SQL. CI/CD validates and publishes them. Docs live in git, versioned with your code.
Tools: dbt, DataHub YAML, Airflow DAG docs
Pro Tip
Shift Documentation Left
The best time to write documentation is when you're writing the code—context is fresh and you're already in the file. Add a description field to your dbt model YAML. Write a docstring for your Python transform. Future you (and your team) will thank you.
4. What to Automate vs. Document Manually
Not everything should be automated. Not everything should be manual. Here's how to decide what goes where.
Always Automate
- • Schema information: column names, types, nullability
- • Lineage: upstream/downstream dependencies
- • Usage stats: query frequency, popular columns
- • Freshness: last updated timestamp
- • Volume metrics: row counts, data size
- • Data quality: null rates, uniqueness
- • Query patterns: common joins, filters
Document Manually
- • Business context: what this data represents
- • Calculation logic: how metrics are computed
- • Known caveats: data quality issues, edge cases
- • Ownership: who to ask about this data
- • SLAs: expected freshness, quality standards
- • Use cases: what dashboards/reports use this
- • Change history: why this field was added
| Information Type | Approach | Reason |
|---|---|---|
| Table schema | Auto-extract | Always accurate, never drifts |
| Column descriptions | Docs-as-code | Lives with SQL, versioned in git |
| Data lineage | Auto-extract | Parse SQL, always current |
| Business meaning | Manual | Requires human context |
| Usage frequency | Auto-extract | Pull from query logs |
| Known issues | Manual | Tribal knowledge, add as discovered |
| Last refresh time | Auto-extract | Pull from warehouse metadata |
| Calculation caveats | Docs-as-code | Write in dbt description, extract |
Example: dbt model with embedded docs
# models/marts/revenue/monthly_revenue.sql
{{
config(
materialized='table',
description='Monthly revenue aggregated from orders. **Caveat**: Excludes refunds processed >30 days after order.'
)
}}
-- Column-level docs in schema.yml
version: 2
models:
- name: monthly_revenue
description: |
Monthly revenue by product category.
**Calculation**: SUM(order_total) WHERE order_status = 'completed'
**Owner**: [email protected]
**SLA**: Updates daily at 6 AM UTC
columns:
- name: month
description: Calendar month (YYYY-MM-01)
- name: category
description: Product category from dim_products
- name: revenue
description: |
Total revenue in USD.
**Note**: Does not include tax or shipping.5. Integrating Documentation into Your Workflow
The secret to keeping docs current: make documentation part of the development process, not something that happens after. Here are the key integration points.
1. PR Template Checks
Add a documentation section to your PR template. Reviewers check that new tables have descriptions.
Example GitHub PR template:
## Documentation Checklist - [ ] dbt model has description - [ ] New columns have descriptions - [ ] Calculation logic is documented - [ ] Owner is specified in schema.yml - [ ] Known caveats are noted
2. CI/CD Validation
Automated checks in your build pipeline ensure docs meet standards before merging.
Example dbt test for undocumented models:
# Check that all models have descriptions
SELECT
model_name
FROM {{ ref('dbt_models') }}
WHERE description IS NULL
OR description = ''
OR description = 'TODO'
-- This test will fail if any models are undocumented3. Staleness Alerts
Automated reminders when high-traffic tables haven't been reviewed in 6+ months.
Weekly Slack reminder:
These high-usage tables haven't been reviewed in 6+ months:
•
orders_mart - last reviewed 8 months ago•
customer_ltv - last reviewed 10 months agoPlease review and update if needed.
4. Quarterly Doc Sprints
Dedicate 1 day per quarter for the team to update critical documentation.
What to review:
- • Top 20 most-queried tables (are docs still accurate?)
- • Recently deprecated tables (are they marked as such?)
- • High-priority dashboards (is lineage documented?)
- • Onboarding docs (do they reflect current reality?)
Pro Tip
Make Documentation Visible
Add a "Documentation Health" dashboard to your team metrics. Track: % of tables documented, average staleness, PR merge rate with docs. What gets measured gets improved.
6. Tools for Documentation Automation
The modern data stack has excellent tools for keeping docs in sync. Here's how they fit together.
| Tool | What It Automates | Best For | Pricing |
|---|---|---|---|
| dbt Docs | Model lineage, schema, descriptions | Transformation layer | Free (OSS) |
| Atlan | Schema discovery, lineage, usage | Enterprise catalog | Enterprise |
| Select Star | Lineage, usage, popularity | Usage-based discovery | Paid |
| DataHub | Metadata ingestion, lineage | OSS metadata platform | Free (OSS) |
| Datadef | Visual diagrams, lineage | Visual documentation | Free tier + paid |
| Alation | Full catalog with ML enrichment | Large enterprises | Enterprise |
Recommended Tool Stack by Team Size
Small Teams (2-5)
- • dbt docs for transformations
- • README files in git for high-level
- • Datadef for visual diagrams
Focus on lightweight, free tools
Mid-Size (6-20)
- • dbt docs + dbt Cloud
- • DataHub or Select Star
- • Datadef for stakeholder docs
Add a catalog for discovery
Large (20+)
- • Atlan or Alation
- • dbt Cloud integration
- • DataHub for custom metadata
Enterprise catalog with governance
From experience
Start simple. Most teams over-invest in expensive catalogs before they have basic docs-as-code working. Get dbt descriptions in place first, prove value with your team, then expand to a catalog if needed. You can always graduate to enterprise tools later.
7. Best Practices for Keeping Docs in Sync
Automate the structural parts
Schema, lineage, usage stats should be auto-extracted. Never ask humans to maintain information that systems know.
Write docs in code, not wikis
Use dbt YAML, SQL comments, Python docstrings. Docs that live next to code stay in sync. Docs in Confluence don't.
Make docs part of PR reviews
Add documentation checks to your PR template. Reviewers verify that new tables and columns are described.
Set up staleness alerts
Automated Slack reminders when high-usage tables haven't been reviewed in 6+ months. Catch drift proactively.
Focus on high-value tables first
Document the 20% of tables that get 80% of queries. Don't try to document everything at once.
Use CI/CD to enforce standards
Fail builds if critical models lack descriptions. Make good docs non-negotiable for high-priority data.
Embed ownership in metadata
Every table should have an owner field. When docs are unclear, people know who to ask.
Run quarterly doc sprints
Dedicate 1 day per quarter for the team to update and improve documentation. Make it a team ritual.
Measure documentation health
Track % of tables documented, staleness, and PR compliance. What gets measured gets maintained.
Make docs discoverable
If people can't find your docs, they might as well not exist. Integrate with Slack, IDE extensions, catalog UI.
Anti-Patterns to Avoid
- • Maintaining documentation in a separate wiki/Confluence
- • Asking junior engineers to "go document everything"
- • No validation that docs match reality
- • Documentation as an afterthought, not part of PRs
- • Over-documenting trivial tables, under-documenting critical ones
Golden Rule
If It Can Be Automated, Automate It
Humans are terrible at maintaining documentation manually. We forget, we get busy, we deprioritize it. Automation doesn't forget. Extract what you can automatically, embed what you must write in code, and manually document only the high-value context that can't be extracted.
8. Frequently Asked Questions
Why does data documentation become outdated so quickly?
Documentation drifts because it lives separately from the code. When engineers update pipelines, they focus on making the change work, not updating docs. Without automation or integration into the development workflow, docs become stale within weeks.
How can I automate data documentation updates?
Use tools that extract metadata directly from your stack: dbt docs for transformations, data catalogs that scan your warehouse schema, lineage tools that track dependencies, and CI/CD integrations that validate docs on every PR. The goal is to make documentation a byproduct of normal development, not a separate task.
What should be documented vs. auto-generated?
Auto-generate: schemas, lineage, query patterns, usage stats, freshness. Document manually: business context, calculation logic, known caveats, data quality issues, ownership. The rule: automate the "what" (structure), document the "why" (context).
How do I get my team to maintain documentation?
Make it part of the workflow, not an afterthought. Add documentation checks to PR templates, set up automated reminders for undocumented tables, recognize teams with good docs, and most importantly: demonstrate the value by showing how good docs save debugging time.
What's the ROI of investing in documentation automation?
Teams with automated documentation report 40% faster onboarding, 50% less time debugging, and 30% fewer duplicate tables. The payback period is usually 2-3 months. The cost of bad docs (wasted time, wrong decisions, duplicate work) far exceeds the investment in automation.
Should I use a data catalog or just stick with dbt docs?
Start with dbt docs if you have <20 people. Add a catalog (DataHub, Select Star, Atlan) when: you have 100+ tables, multiple tools beyond dbt, need usage analytics, or have compliance requirements. Catalogs add discovery and governance; dbt docs are great for engineering-focused lineage.
Document Your Data Architecture Visually
Create clear, always-current diagrams of your data platform. Show your team and stakeholders exactly how data flows through your stack.