Platform Engineering Guide

Kubernetes Data Architecture

Kubernetes is the de facto platform for compute. But running data workloads — Spark, Airflow, stateful databases — on K8s has unique challenges. This guide covers what works, what doesn't, and practical patterns.

Diagram Your K8s ArchitectureStreaming Architecture
18 min readFor Platform & Data Engineers

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

89/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

What to Run on K8s (and What Not To)

Run on K8s

  • Airflow (KubernetesExecutor)
  • Spark jobs (batch processing)
  • dbt runners
  • Flink stream processors
  • ML training jobs
  • Redis, ClickHouse (analytics)

Use Managed Instead

  • Production PostgreSQL/MySQL (use RDS/Cloud SQL)
  • Kafka (use MSK/Confluent Cloud)
  • Elasticsearch clusters (use Elastic Cloud)
  • Data warehouses (Snowflake, BigQuery)

Spark on Kubernetes

Since Spark 3.0, K8s is a first-class cluster manager alongside YARN. Use the Spark Operator for declarative job management.

Driver Pod

Coordinates the Spark application. Runs one pod with configurable CPU/memory. Use pod templates for custom configs.

Executor Pods

Scale dynamically based on workload. Dynamic allocation with spark.dynamicAllocation.enabled=true spins pods up/down automatically.

Storage

Read from S3/GCS/ABFS via Hadoop connectors. Shuffle data uses local SSD (emptyDir) or remote storage (S3 shuffle).

Images

Custom Docker images with your Spark app, Python deps, and configs. CI/CD builds and pushes images per deployment.

Airflow on Kubernetes

The KubernetesExecutor is the recommended executor for production Airflow on K8s. Each task runs as an isolated pod.

Isolation

Each task gets its own pod, resources, and Docker image. No interference between tasks.

Auto-Scaling

Pods spin up for tasks, terminate when done. No idle workers eating cost.

Helm Chart

Deploy with the official apache-airflow Helm chart. Manage config via values.yaml.

Architecture Patterns

Namespace-Per-Team

Each data team gets a K8s namespace with resource quotas, RBAC policies, and dedicated node pools. Prevents noisy neighbors.

Spot Instances for Batch

Use spot/preemptible nodes for Spark jobs and ML training. Node pools with taints ensure only fault-tolerant workloads land on spot.

GitOps with ArgoCD

Define Spark jobs, Airflow DAGs, and Flink applications as K8s manifests in Git. ArgoCD syncs desired state automatically.

FAQ

Should you run databases on Kubernetes?
Use managed services (RDS, Cloud SQL) for production databases. K8s with StatefulSets works for dev/test and analytics databases (Redis, ClickHouse).
How do you run Spark on Kubernetes?
Use spark-submit with --master k8s:// or the Spark Operator. K8s handles pod scheduling, resource isolation, and auto-scaling. Native support since Spark 3.0.
What is the best way to run Airflow on K8s?
KubernetesExecutor: each task runs as a separate pod with perfect isolation and auto-scaling. Deploy via the official Helm chart.

Diagram your K8s data platform

Map pods, services, and data flows with AI-powered diagrams and Kubernetes icons.

Try Datadef Free