Real-Time Analytics Architecture Generator

Real-time processes data as it arrives (sub-second to seconds latency) using streaming engines like Flink or Spark Streaming. Batch processes accumulated data periodically (hourly, daily). Visualize lambda architecture combining both: hot path for real-time, cold path for accurate batch processing.

Streaming source (Kafka, Kinesis) → stream processor (Flink, Spark Streaming) → real-time database (ClickHouse, Druid, TimescaleDB) → visualization (Grafana, custom dashboards). Show windowing for aggregations, stateful processing for complex analytics, and materialized views for query performance.

ClickHouse for OLAP queries, Druid for time-series, TimescaleDB for IoT data, Redis for caching metrics. Show write-optimized ingestion layer, columnar storage for fast aggregations, and in-memory caching for frequently accessed metrics. Include data retention and rollup policies.

WebSocket connections from dashboard to backend, server pushes updates as events occur. Or dashboard polls REST API on interval. Visualize event stream → processing → dashboard state management → UI updates. Include mechanisms for handling late-arriving data and out-of-order events.

Partitioned message brokers (Kafka topics), horizontal scaling of stream processors, sharded databases for writes. Show data ingestion → partitioning by key → parallel processing → aggregation → storage. Include backpressure handling, auto-scaling based on lag, and monitoring for bottlenecks.