A living diagram from Kafka topics: list topics, read the schema registry, and map who produces and consumes what

Start with what exists. kafka-topics.sh --list enumerates every topic on the cluster, and the schema registry's subjects endpoint enumerates every registered contract; under the default TopicNameStrategy a subject named orders.created-value binds a schema to the orders.created topic. Between the two you have the nodes of the diagram and, where schemas exist, what flows through each one.

The subjects list is also a quality signal in itself: topics with no registered subject are the ones carrying unversioned payloads, exactly the edges where a data contract is missing.

kafka-topics.sh --bootstrap-server broker:9092 --list

# Every reader, per group: which topics, which partitions, what lag
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --all-groups

# Every registered contract
curl -s http://schema-registry:8081/subjects

Here is the asymmetry that shapes the whole exercise. Consumers announce themselves: committed offsets mean kafka-consumer-groups.sh --describe --all-groups lists every group and exactly which topics and partitions it reads, with lag as a bonus. The consume edges of your diagram come straight from the cluster, no code reading required.

Producers register nothing. A service that writes to a topic leaves no standing record in the cluster, so the produce edges need indirect sources: ACLs are the best one, since kafka-acls.sh --list shows which principals hold WRITE on which topics in a locked-down cluster. Failing that, it is client configs and code search. A diagram tool cannot fix this asymmetry; it can only be honest about which edges are observed and which are declared.

If your topics follow a domain.entity.event convention, orders.created, payments.captured, the diagram's grouping is already decided: one zone per domain, topics inside it, services on the edges. This is where a curated diagram beats any auto-layout of the full topic list.

The full list also needs filtering. A real cluster carries internal topics, __consumer_offsets, connect-configs and friends, plus the -changelog and -repartition topics Kafka Streams creates per application. They are implementation detail, and a wiki diagram that includes them buries the fifty topics that matter under two hundred that do not.

If topics are managed declaratively, a Terraform provider, a GitOps topology file, then topic changes arrive as merges, and the diagram loop hooks there: when CI applies the topology change, one added step has an agent connected to Datadef's MCP server (registry io.datadef/mcp) update the diagram, new topics into their domain zones, consumer edges refreshed from the cluster describe output. The agent connection needs an API key, available on paid plans; the MCP diagram server guide covers setup.

Datadef does not watch the cluster or the repo; the regeneration is the one command you add. For clusters managed by hand, a scheduled job that re-reads topics, subjects, and groups weekly is the honest fallback. Either way the wiki embed refreshes itself within minutes of a diagram update, and the first version of the diagram is quickest through the Kafka streaming architecture generator.

Consumer group output describes active and recently active groups: committed offsets expire after the broker's offsets.retention.minutes, seven days by default, so a consumer that has been down for weeks quietly vanishes from the map. Treat the describe output as "who reads now", not "who has ever read".

Produce edges built from ACLs are permissions, not activity: WRITE granted is not WRITE used. And the embed that keeps the wiki current requires the Datadef project to be shared public, which for a diagram naming internal services and topics deserves a deliberate yes. The wider pipeline documentation practice around this diagram is covered in keep pipeline docs in sync.

• Kafka streaming architecture generator
• Data contracts
• Keep pipeline docs in sync
• The MCP diagram server