Streaming Diagram Guide

Kafka streaming architecture diagram: topics, consumer groups, and the sinks

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

Kafka diagrams break in a specific way: someone draws a topic as if it were a service, gives it a box the size of a database, and the picture stops describing the system. A topic is a named log inside one cluster. Getting that right changes the whole layout, and everything else in a streaming diagram follows from it.

Try it free
7 min readFor data and platform engineers documenting an event streaming pipeline

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

305/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

A topic is not a service

Draw the cluster once, as a container, and put topics inside it as lanes or as small labelled nodes. Producers sit on the left, consumer groups on the right, and each arrow carries the topic name. That single change removes the most common misreading of a Kafka diagram, which is that each topic is separately deployed and separately operated.

Two properties belong on every topic label: partition count and retention. Partitions determine the maximum useful parallelism of a consumer group and the ordering guarantee, which is per partition and never global. Retention determines whether a consumer that falls behind recovers or loses data. Almost every operational question about a topic reduces to one of those two numbers, so put them on the picture.

Compacted topics deserve a marker of their own. A compacted topic is closer to a keyed table than to a log, downstream consumers treat it differently, and a reader cannot infer it from the arrow.

The parts around the broker that always get left out

Schema Registry is the contract between producers and consumers, and it is the piece that turns a topic from a byte stream into an interface. Draw it beside the cluster with dashed edges to producers and consumers, because schemas are metadata rather than data, and put the compatibility mode on it. Backward compatible and full compatible are different promises to downstream teams.

Kafka Connect is a worker tier, not an arrow. Source connectors and sink connectors run on Connect workers with their own scaling and their own failure modes, so drawing a connector as a plain line from a database to a topic hides a whole deployment. Give Connect a lane and put the connectors in it.

Consumer groups are the unit of parallelism and, more usefully, the unit of ownership. Name the group after the downstream system and put the owning team on it. Add the dead letter topic wherever one exists, since that is the first place anyone looks when messages disappear. For cross-region setups, draw the replication mechanism explicitly rather than a line between two clusters.

Where the stream lands

Processing is usually Flink or Spark Structured Streaming, occasionally Kafka Streams running inside the consuming application. That last case is worth distinguishing on the diagram, because it means there is no separate processing cluster to operate and the state lives in the app.

Sinks split by purpose: ClickHouse or Druid for real time analytics with second-level freshness, Snowflake or BigQuery for the warehouse on a batch cadence, Elasticsearch for search, and object storage for the archive and replay. Draw the cadence on the sink edge, because a stakeholder reading the diagram wants to know how fresh the dashboard is, and the answer lives on that edge.

Put the delivery guarantee on the sink edge too. At least once with idempotent writes and exactly once through a transactional sink lead to different downstream designs, and this is the one place a diagram can state it without a paragraph.

The icons, including the managed equivalents

The data tool set here is 32 marks and it is worth knowing what is in it, because these are the logos no cloud vendor draws: Kafka, Flink, Spark, Hadoop, ClickHouse, Druid, Cassandra, Elasticsearch with Kibana and Logstash, Snowflake, Databricks, dbt, Airflow, Dagster, Prefect, Fivetran, Airbyte, Segment, RudderStack, Looker, Tableau, Metabase, Superset, Monte Carlo, Grafana, Prometheus, Datadog, Splunk, Sumo Logic and New Relic. Pulsar sits in the wider tech-stack set rather than this one.

The managed equivalents each have their own official mark, and that matters for accuracy rather than tidiness. Amazon Managed Streaming for Apache Kafka and Kinesis Data Streams are separate AWS icons and are not interchangeable with each other. Azure Event Hubs speaks the Kafka protocol but is its own service with its own icon, its own partition semantics and its own capture behaviour. Google Cloud Pub/Sub is a different model entirely, with subscriptions rather than consumer group offsets. Using the Apache Kafka mark for Event Hubs or Pub/Sub tells a reader something untrue about how offsets and replay work, and it is the kind of error a reader who operates the thing spots immediately.

Describe the pipeline and the canvas comes back with the cluster as a zone, the topics inside it, and labelled edges. Export to PNG or JPEG, or keep a live embed in the platform README. For the model where the topics themselves drive the picture, see living diagrams from Kafka topics.

FAQ

How should Kafka topics appear on an architecture diagram?

As lanes or small labelled nodes inside a single cluster container, not as standalone service boxes. A topic is a named log within one cluster, so drawing each one as its own deployed component misrepresents both the operations and the failure domain. Put partition count and retention on every topic label.

Should consumer groups be on the diagram?

Yes. The consumer group is the unit of parallelism and the unit of ownership, so naming the group after the downstream system and adding the owning team answers who is affected when a topic changes. Group-level lag is also the metric on-call watches, which makes the group the right node to point an alert at.

Where do Kafka Connect and the Schema Registry belong?

Kafka Connect belongs in its own lane as a worker tier holding the source and sink connectors, because it has independent scaling and failure modes. The Schema Registry sits beside the cluster with dashed metadata edges to producers and consumers, labelled with its compatibility mode.

Is Azure Event Hubs drawn as Kafka?

No. Event Hubs offers a Kafka-compatible endpoint, but it is a distinct service with its own official Azure icon, its own partition and consumer group semantics, and its own capture behaviour. Using the Apache Kafka mark for it implies a portability and an operational model that do not fully carry over, which is exactly the assumption a reader will make from the logo alone.

How do I show delivery guarantees on a streaming diagram?

As a label on the sink edge. At least once with idempotent writes, exactly once through a transactional sink, and at most once produce different downstream designs, and the edge into the sink is the one place a diagram can state that without adding prose.