Keep a data catalog in sync: crawlers handle technical metadata, the human context is what drifts

Every serious catalog, DataHub, Amundsen, Collibra, Alation, the cloud-native ones, runs scheduled ingestion against your warehouse, BI tool and orchestrator. New tables appear, dropped columns disappear, query-log lineage updates, and none of it needs a human. This layer genuinely stays in sync, and it is worth saying so plainly because it defines what is left over.

What the crawler cannot ingest is meaning: why the table exists, which of the three revenue tables is the certified one, who answers questions about it, and whether the gotcha in the currency column still applies. That context is the reason the catalog exists, it arrives only through curation, and it starts drifting the day it is written.

The context layer needs a number, and description coverage is the one that works: the share of tables and columns carrying a real description, tracked over time, cut by schema or domain. It is measurable directly in the warehouse, and if you push descriptions down as native comments, from dbt with persist_docs, or through the catalog's write-back, one information_schema query audits the whole estate. Falling coverage in a schema is the early signal that a team has stopped curating what it ships.

Set the target by tier, not globally: certified, tier-one datasets at or near full column coverage, the long tail at table-level descriptions only. A global percentage invites gaming the number on tables nobody queries.

-- Description coverage per schema (Snowflake)
SELECT table_schema,
       COUNT(*) AS columns_total,
       COUNT_IF(comment IS NOT NULL AND comment <> '') AS columns_documented,
       ROUND(100.0 * COUNT_IF(comment IS NOT NULL AND comment <> '')
             / COUNT(*), 1) AS coverage_pct
FROM information_schema.columns
WHERE table_schema NOT IN ('INFORMATION_SCHEMA')
GROUP BY table_schema
ORDER BY coverage_pct ASC;

Backfill campaigns produce a coverage spike and a slow decay, because they treat curation as a project instead of a loop. The loop version: when the crawler picks up a new table or column without a description, a curation task goes to the owning team, the team that shipped the change, not a central data librarian who lacks the context to write anything true. Most catalogs can drive this from ingestion events or a scheduled query against their own metadata.

Ownership drift needs the same treatment, because a catalog entry owned by someone who left is unowned with extra steps. Sync owner fields against the directory or the group membership, and flag entries whose owner no longer exists as loudly as entries with no owner at all.

Catalog adoption runs on trust, and trust is asymmetric: one analyst finding one certified-looking entry that describes a table dropped last quarter undoes fifty accurate ones, because the lesson learned is "check with the data team anyway", which is the exact workflow the catalog was bought to replace. Staleness in a catalog is not clutter, it is the adoption killer.

The countermeasure is aggressive deprecation: archive entries for datasets with no queries in ninety days, mark deprecated tables deprecated in the catalog the day the migration starts, and let certification expire so it means "someone recently vouched for this" rather than "someone once did". An emptier, truer catalog beats a complete, lying one.

This page is the staying-current loop. Choosing a catalog, connecting sources and structuring domains is the setup problem, and it has its own anchor in data catalog best practices; the broader discipline across pipelines, schemas and lineage lives in keeping data documentation in sync, with the dictionary-specific variant in keep a data dictionary in sync with the schema.

One more artifact drifts alongside the entries: the diagram of the catalog platform itself, the picture of sources, crawlers, metadata store and curation flow that appears in every onboarding deck. Generate it once with the data catalog architecture generator, and when the platform changes, update it in Datadef by hand or through an agent connected to the Datadef MCP server, registry name io.datadef/mcp. Datadef does not watch your stack; the update is one agent call, its API key comes with a paid plan, and a project shared public serves an image URL that follows edits within minutes, so the deck and the wiki stop disagreeing about your own platform.

• Data catalog best practices
• Keeping data documentation in sync
• Keep a data dictionary in sync with the schema
• Data catalog architecture generator