Kafka Guide

Kafka services diagram: the event flow your repository declares

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

Confluent Stream Lineage and the observability tools draw the topology from live traffic, which is the right answer when the cluster is running and the wrong one when it is not. Before deployment, during a design review, or on a branch that adds a consumer, the only source is the repository: the compose file that starts the broker, the deployment manifests that set the bootstrap servers, and the schema files the services share.

Try it free
7 min readFor teams building event-driven services who need the topology before the traffic

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

175/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

Where the topology is written down

Three classes of file carry it. The container class takes the compose file that runs Kafka or Redpanda locally, plus every Dockerfile, matched on the name prefix, six of each. The manifest class takes YAML under a directory named k8s, kube, kubernetes, manifests, charts, or helm, twelve files, which is where a Strimzi cluster definition, a topic custom resource, or a chart values file lives. The API class takes .proto files, GraphQL schemas, and OpenAPI documents, ten files. Choosing the architecture focus at the first sync raises the manifests to fourteen and lowers the contracts to four.

Between them, most of the topology is declared. A KafkaTopic custom resource names the topic, its partitions, and its retention. A deployment environment block names the bootstrap servers, the consumer group, and usually the topic the service subscribes to. A shared proto file names the event payload that two services agree on.

The gap to be clear about: topic names constructed in application code are not visible, because application source is not part of the classified corpus. A service that reads its topic from an environment variable declared in a manifest is drawn accurately. A service that builds the topic name from a constant in a Java or Go file is drawn from whatever its configuration does declare.

Event topology from files
  docker-compose.yml            broker, schema registry, connect, local sinks
  k8s/base/kafka-topics.yaml    topic resources, partitions, retention
  k8s/base/orders-deploy.yaml   bootstrap servers, group id, topic env vars
  charts/connect/values.yaml    connectors and their sinks
  proto/events/order_v2.proto   the event contract two services share
  .github/workflows/*.yml       what deploys each service

How the diagram is arranged

Event-driven systems read badly when they are drawn as a mesh. The generated canvas puts the producers on one side, the topics in the middle as their own row, and the consumers and sinks on the other, so the direction of flow is the direction of the page. Edges are labelled with the topic and, where the files declare it, the consumer group.

Components render with the marks they deserve: the broker with the Kafka or Redpanda icon, the schema registry with its own, connectors with the systems they sink into, and each service with the runtime its Dockerfile or dependency manifest names. Real icons are the difference between a diagram people recognize and a set of labelled rectangles.

Where the same topic is consumed by four services, that is four edges out of one topic node, not four copies of the topic. Keeping the topic single is what makes a fan-out visible at a glance, and it is the first thing hand-drawn event diagrams get wrong.

A view that exists before the cluster does

No broker connection, no consumer group to register, no admin credentials. The sync reads the repository with read-only access on GitHub, GitLab, or Azure DevOps, which means the topology can be drawn for a branch that adds a consumer, a design under review, or a client system you have no runtime access to.

Runtime lineage tools and this view answer different questions and both are worth having. Live tooling shows what is actually flowing right now, including the producer nobody documented. The repository view shows what the team has agreed to build, which is the version you need in a design review and the one that can be compared against production later.

On the first sync the counts are stated in one deterministic sentence, no model involved. An event platform repo usually reads as something like: Mostly Kubernetes manifests and Docker Compose files (21 files) with 9 data model and API files, which points at the architecture focus and the runtime view.

Avro schema files

Protocol buffer, GraphQL, and OpenAPI definitions are classified as API contracts. Avro .avsc files are not in that class, so their content is not read, though their directories appear in the tree summary. See how repository sync works.

Keeping the map alive

Topologies grow one consumer at a time, and each of those changes is a manifest or a compose edit, which is precisely what the daily sync notices. Commits that do not move the branch head are skipped, node identity carries across regenerations, and nodes you positioned by hand stay where you left them.

Embed the current map in the platform README or the service catalog page as a live image, and an agent can trigger a refresh over MCP the moment a new consumer merges, so the map and the code land in the same session.

FAQ

Does it connect to my Kafka cluster?

No. There is no broker connection, no consumer group registration, no admin client, and no credential of any kind for the cluster. The topology comes from the repository files read with read-only access on GitHub, GitLab, or Azure DevOps, so it can be drawn for a branch that adds a consumer, for a design under review, or for a client system you will never be given runtime access to.

Where do the topic names come from?

From the files that declare them: topic custom resources under a cluster-named directory, environment variables in deployment manifests and compose services, and connector configuration. Topic names assembled in application source are not visible, because application code is not part of the classified corpus.

Are event schemas read?

Protocol buffer files, GraphQL schemas, and OpenAPI documents are classified as API contracts and are read, so a shared proto between two services becomes a real edge. Avro .avsc files are not in that class, so their directories appear in the tree summary without their contents being fetched.

How does this compare to stream lineage tools?

They observe live traffic and show what is flowing now, including undocumented producers. A repository view shows what the code declares, works before anything is deployed, and can be regenerated on every branch. The two are complementary rather than competing.

Does a fan-out topic get drawn once or many times?

Once. A topic consumed by several services is a single node with an edge to each consumer, which is what makes the fan-out readable. Duplicating the topic per consumer is the most common mistake in hand-drawn event diagrams.