Data Pipeline Guide

Batch vs streaming on a pipeline diagram: conventions that survive one canvas

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

Very few real platforms are purely batch or purely streaming. Most are a hybrid where a Kafka topic and a nightly extract end up writing to the same gold table, and the diagram has to make that legible without a legend nobody reads. A small set of drawing conventions does the job: line style carries the processing mode, edge labels carry the timing, and the boundary node where continuous becomes scheduled is drawn explicitly rather than implied.

Try it free
6 min readFor platforms running streaming and batch paths side by side

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

239/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

Line style for mode, labels for timing

Use a solid edge for a continuous hop and a dashed edge for a scheduled one. It is a two value convention, it needs no legend once a reader sees one labelled example, and it reads correctly even after the diagram has been shrunk into a wiki page.

Then let the labels carry the timing detail that line style cannot. On a batch edge, write the schedule the way the orchestrator expresses it, for example nightly 02:00 UTC or every 15 minutes, because that string is searchable in the DAG code. On a streaming edge, write the topic name and the latency target, for example orders.v2, p99 under 5s. Throughput belongs there too when it drives capacity decisions.

Resist a third line style. A diagram with solid, dashed, dotted, and thick edges becomes a puzzle. If you need a third mode, it is usually micro-batch, and micro-batch is better handled with a label than with a stroke pattern.

Micro-batch, CDC, and the mode boundary

Micro-batch is neither mode, and pretending otherwise causes arguments. A Structured Streaming query with a five minute processing time trigger is scheduled from the operator point of view and continuous from the code point of view. Draw it as a solid edge, since it is always running, and label it micro-batch, 5 min trigger. That single label prevents someone assuming sub-second latency they are not getting.

Change data capture has the opposite trap. The transport is continuous, but the consumption is very often scheduled: Debezium writes to a topic all day, and a job drains that topic on a schedule. Two hops, two line styles, with the topic as the node between them. Drawing it as one arrow from Postgres to the warehouse hides the exact place where the latency actually lives.

Wherever a solid edge meets a dashed edge, there is a node that changes the mode: a topic, a landing table, a queue, a staging area. Name that node. It is the buffer that absorbs a producer outage, and it is the first thing anyone inspects when latency grows.

orders_service ==[orders.v2, ~2k msg/s, p99 <5s]==> Kafka orders.v2      (solid: continuous)
Kafka orders.v2 --[Spark micro-batch, 5 min trigger]--> lake.bronze_orders   (solid, labelled)
Postgres        - -[JDBC extract, nightly 02:00 UTC]- -> lake.bronze_customers (dashed: scheduled)
lake.bronze_*   - -[dbt run, hourly]- -> lake.gold_orders_daily              (dashed: scheduled)

Lambda, kappa, and the backfill lane

If a streaming path and a batch path both write to the same serving table, that convergence is the single most important fact on the diagram and it should be impossible to miss. Draw both lanes arriving at the same node, and put the reconciliation rule on the node itself: streaming path is provisional, batch path overwrites the last 48 hours. Readers who understand lambda architecture will recognize it, and readers who do not will still understand what happens.

A kappa style platform, where the batch path is a replay of the same stream rather than a separate pipeline, should look different: one lane, plus a replay arrow returning to the processing job from the retained log. Drawing kappa as two lanes makes it look like lambda and invites the wrong questions in review.

Draw the backfill path once, and mark it. Every real pipeline has one, most diagrams omit it, and it is the path that runs during the worst week of the quarter. One dashed arrow from the source or the archive into the processing node, labelled backfill, manual trigger, is enough.

Two lanes, one canvas

Datadef edges take labels and zones can hold each lane, so the streaming and batch paths stay visually separate while converging on the shared serving nodes. The Kafka, Spark, Flink, and warehouse marks come from a registry of over two thousand icons, 438 of them AWS, 624 Azure, and 226 GCP.

Every label should point at a setting somebody can open

A drawing convention is only worth having if it can be falsified. Each label above corresponds to a setting in a file, so a reviewer can check the diagram against the repository rather than against their memory of a design meeting.

For Spark the setting is the trigger on the writer. processingTime with an interval is the micro-batch case, availableNow drains everything currently available and then stops, and continuous runs a continuous query with a checkpoint interval. Trigger.Once has been deprecated since Spark 3.4 in favour of Trigger.AvailableNow, so an edge still annotated "one-time trigger" usually marks code nobody has revisited. Put the trigger on the edge and the checkpointLocation on the node, because that path is the first thing anyone needs when a streaming job has to be restarted.

For a scheduled hop the setting is the orchestrator schedule, and its string belongs on the edge verbatim rather than translated into prose. For a CDC hop there are two settings, the connector config and the consumer schedule, which is exactly why that hop is two edges rather than one. Where a label has no setting you can point at, it is a belief rather than a fact, and it is better to find that out before somebody sizes a cluster against it.

FAQ

How do you show batch and streaming on the same diagram?

Use line style for the mode and labels for the timing: a solid edge for a continuously running hop, a dashed edge for a scheduled one. Put the schedule string on batch edges and the topic name with a latency target on streaming edges. Where a solid edge meets a dashed one, name the buffering node, because that is where the latency and the failure absorption actually live.

How should micro-batch be drawn?

As a solid edge, because the query is always running, with an explicit label such as micro-batch, 5 minute trigger. Drawing it as pure streaming implies sub-second latency the trigger interval does not provide, and drawing it as batch hides the fact that there is a long lived job to operate. Take the label from the writer trigger, which is processingTime, availableNow, or continuous.

Where does change data capture belong on a diagram?

Usually as two hops, not one. The capture side is continuous and writes to a log or topic, and the consumption side is often scheduled. Drawing the topic as a node between a solid edge and a dashed edge shows where the real latency sits, which a single source to warehouse arrow hides.

How do I draw a lambda architecture without confusing readers?

Draw both lanes arriving at the same serving node and write the reconciliation rule on that node, for example streaming results are provisional and the nightly batch overwrites the last 48 hours. The convergence point is the whole idea of lambda, so it should be the most visually obvious element.

Should backfill paths appear on the diagram?

Yes, once, as a marked arrow from the source or the archive into the processing step, labelled as a manually triggered backfill. It is omitted from most diagrams and it is the path that gets used during incidents and migrations, which is exactly when people go looking at the diagram.