AWS Architecture Guide

AWS Data Platform Architecture Diagram

The reference architecture drawn service by service: DMS and Kinesis ingestion, the three S3 data lake zones, Glue jobs and the Glue Data Catalog, Redshift and Athena on the serving side. The diagram is below, then the detail for each layer.

18 min readFor Data Engineers & ArchitectsAWS-specific examples

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

185/20003 credits left
Try:

No account needed · Editable canvas, not a picture

The short answer

An AWS data platform is five layers, and the diagram should show them left to right. Ingestion: AWS DMS for database change data capture, Kinesis Data Firehose for events, AppFlow or Lambda for SaaS APIs. Storage: one S3 data lake split into raw/, processed/ and curated/ prefixes. Transformation: Glue jobs between those prefixes, with a Glue crawler registering each zone in the Glue Data Catalog. Serving: Athena for ad hoc SQL straight on S3, Redshift for BI joins and concurrency. Consumption: QuickSight, SageMaker and operational apps. Everything else (Lake Formation, KMS, Step Functions, CloudWatch) is drawn as a band across the five layers rather than as another box in the flow.

Reference AWS data platform architectureFive layers left to right: sources (RDS, SaaS APIs, app events), ingestion (AWS DMS, Amazon AppFlow, Kinesis Data Firehose), an Amazon S3 data lake with raw, processed and curated prefixes separated by AWS Glue ETL jobs and registered in the Glue Data Catalog, a serving layer of Amazon Athena and Amazon Redshift, and consumers QuickSight, SageMaker and operational apps. A band underneath covers Glue Data Catalog, Lake Formation and IAM, KMS encryption, and Step Functions with CloudWatch.1. SOURCES2. INGESTION3. S3 DATA LAKE4. SERVING5. CONSUMERSAmazon RDS / Auroraorders, customersSaaS APIsSalesforce, StripeApp and web eventsclickstream, task eventsAWS DMSCDC, no source downtimeAmazon AppFlowscheduled SaaS pullsKinesis Data Firehosebuffers, then writes to S3Amazon S3 data lakeone bucket per environmentraw/ (bronze)as delivered, JSON + gzip, dt=YYYY-MM-DDGlue ETL job, bookmark onprocessed/ (silver)typed and deduplicated, Parquet + SnappyGlue ETL job, partitions rewrittencurated/ (gold)business tables, one row per grainAWS Glue Data Cataloga crawler registers one table per zoneAmazon AthenaSQL on S3, billed per scanAmazon RedshiftCOPY to staging, then MERGERedshift Spectrum readscurated/ in place, no loadAmazon QuickSightexec and ops dashboardsAmazon SageMakertraining sets from curated/Operational appsreverse ETL, internal APIsDrawn as a band across all five layers, not as another box in the flowGlue Data Catalogone table definition per zoneLake Formation + IAMwho reads which prefixAWS KMSSSE-KMS on every bucketStep Functions + CloudWatchschedule, retry, alarm
The reference AWS data platform: five layers, one bucket, two Glue hops, two query engines. Everything below is the detail behind one of these boxes.

1. AWS Data Platform Architecture, Layer by Layer

Almost every AWS data processing architecture is a variation on the five layers in the diagram above. What changes between companies is which service fills each slot, not the shape. Here is what each layer does and the detail worth writing on the box.

1

Ingestion: get the bytes into S3, unchanged

AWS DMS replicates relational sources with change data capture, so the lake tracks updates and deletes rather than re-copying whole tables. Kinesis Data Firehose takes events and buffers them by size or by interval before writing an object, which is why the raw zone ends up with a few large files instead of millions of tiny ones that make every later query slow. AppFlow or a scheduled Lambda pulls SaaS APIs. None of these three should transform anything: the value of the raw zone is that it is replayable.

2

Storage: one S3 bucket, three prefixes

Not three buckets. One bucket per environment with raw/, processed/ and curated/ prefixes keeps bucket policies, replication rules and lifecycle configuration in one place. Partition every prefix on the ingest date with Hive-style keys (dt=2026-08-21) so both Athena and Glue can prune without reading the objects.

3

Transformation: Glue jobs between the prefixes

Each arrow between two zones is one Glue job with a name you can page on. Turn job bookmarks on so a rerun processes only new partitions. Write Parquet with Snappy compression from processed/ onward, because Athena is billed on bytes scanned and columnar plus partition pruning is where that number falls by an order of magnitude.

4

Serving: Athena on the lake, Redshift for concurrency

Athena queries curated/ directly through the Glue Data Catalog with nothing to provision. Redshift is where you load when dashboards need consistent response time under concurrency, or when the joins get wide. Redshift Spectrum sits between the two: Redshift compute reading the same S3 prefix in place.

5

Consumption: dashboards, models, and apps reading back

QuickSight for BI, SageMaker training sets built from curated/, and operational systems reading the platform back through reverse ETL or an API. Draw the consumers, because they are what makes a schema change expensive, and a diagram that stops at the warehouse hides that.

Pro Tip

Governance is a band, not a box

Lake Formation, KMS, IAM and CloudWatch touch every layer. Putting them in the flow as boxes adds arrows to everything and the diagram stops being readable. Draw them as a strip under the five layers, as in the diagram above, and annotate which prefix each policy covers.

2. Core AWS Data Services and When to Use Each

AWS has 200+ services, but modern data platforms are built on a core set of 10-15. Here's what each service does and when to use it in your diagram. If you are deciding what to leave out, the guide on which AWS services to show in a diagram goes further.

Amazon S3

Storage

Object storage for your data lake. Stores raw, processed, and curated data at scale.

Use when: Storing any data at scale. Organize with prefixes: s3://bucket/raw/, s3://bucket/processed/, s3://bucket/curated/

AWS Glue

ETL + Catalog

Serverless ETL for transforming data. Glue Data Catalog provides metadata for tables.

Use when: Building ETL pipelines that don't require real-time processing. Glue crawlers auto-discover schemas in S3.

Amazon Athena

Query

Serverless SQL queries on S3 data. Billed per query, no cluster management.

Use when: Ad-hoc analysis, data exploration, cost-sensitive workloads. Works with Glue Data Catalog.

Amazon Redshift

Warehouse

Columnar data warehouse for BI and analytics. Optimized for complex queries and dashboards.

Use when: High-performance dashboards, predictable workloads, complex multi-table joins.

Amazon Kinesis

Streaming

Real-time data streaming. Kinesis Data Streams for custom apps, Firehose for direct S3/Redshift delivery.

Use when: Real-time ingestion from clickstreams, IoT, logs. Use Firehose for simple S3 delivery.

AWS Lambda

Compute

Serverless functions triggered by events (S3 uploads, Kinesis streams, schedules).

Use when: Lightweight transformations, file processing, API calls. Cheaper than Glue for small workloads.

AWS DMS

Replication

Database Migration Service, used here for ongoing change data capture from RDS, Aurora or an on-premises database into S3.

Use when: The lake needs updates and deletes, not nightly full copies. Draw the replication instance and its VPC, since that is where the failures happen.

ServiceCategoryBilled byWhen to Use
S3StorageGB storedAlways (data lake foundation)
GlueETLDPU-hourBatch transformations >5 min
LambdaComputeInvocationEvent-driven, <15 min tasks
AthenaQueryTB scannedAd-hoc queries, exploration
RedshiftWarehouseNode-hour or RPUHigh-perf BI dashboards
KinesisStreamingShard-hourReal-time ingestion
DMSReplicationInstance-hourCDC from relational sources
Step FunctionsOrchestrationState transitionMulti-job pipelines with retries

3. AWS Data Lake Architecture Diagram: the S3 Layout to Draw

An S3 data lake architecture diagram that only shows a bucket labelled "Data Lake" is not worth drawing. The useful version shows the prefixes, what format lives in each one, and the partition key, because those three facts are what someone debugging a slow query or a missing day actually needs.

ZonePrefixFormatWhat lives there
Bronzeraw/source/table/dt=JSON, CSV, gzipExactly what arrived. Never edited, never deleted, replayable.
Silverprocessed/domain/table/dt=Parquet + SnappyTyped columns, deduplicated on the business key, late arrivals merged.
Goldcurated/mart/table/Parquet, often IcebergOne row per business grain. The only zone analysts should query.

Five things to put on the S3 boxes

The prefix, not just the bucket

s3://acme-lake-prod/raw/salesforce/opportunity/ tells someone where to look. "Raw bucket" does not.

The partition key

dt=YYYY-MM-DD is the default. If queries filter on region or tenant more often than on date, partition on that instead and say so on the diagram.

The file format and compression

JSON in raw, Parquet with Snappy from processed onward. This is the single biggest lever on Athena scan volume.

The lifecycle rule

Raw objects moving to S3 Glacier Instant Retrieval after 90 days is a fact people forget until a replay takes hours. Annotate it.

The catalog table it maps to

Each prefix is one Glue Data Catalog table. Write the database and table name so the diagram connects to what Athena sees.

One bucket or three

Three separate buckets look tidier on a diagram and cost more to operate: three sets of bucket policies, three replication configurations, three places for a permission to drift. One bucket per environment with prefixes per zone is the layout most teams end up at. The exception is when a zone needs a different retention or a different KMS key, at which point the split earns itself. The medallion architecture guide covers the bronze, silver and gold naming in more depth.

4. AWS Glue Architecture Diagram: What to Draw Around a Glue Job

A Glue architecture diagram is not one box. Every Glue job has four things attached to it, and leaving them out is why so many Glue diagrams are useless during an incident.

The crawler

Points at an S3 prefix, infers the schema and writes a table into the Glue Data Catalog. Draw it once per zone, on a dotted line, not in the data flow.

The Data Catalog table

Database plus table name. Athena, Redshift Spectrum and EMR all read this same entry, which is why it belongs at the side of the diagram rather than inside one branch.

The trigger

A schedule, an EventBridge rule on S3 object creation, or a Glue workflow step. Without it nobody can answer "why has this not run".

The target

The S3 prefix or the Redshift table the job writes, plus whether it appends, overwrites the partition, or merges.

Glue, Lambda or EMR

ChooseWhen the jobWhat to label on the box
LambdaFinishes in under 15 minutes and touches one file at a timeMemory, timeout, the S3 event that triggers it
Glue (Spark)Joins or shuffles across a whole partition, or runs past a Lambda timeoutJob name, worker type, bookmark on or off
Glue (Python shell)Is a small script that needs longer than Lambda allows but no SparkJob name, schedule
EMRNeeds cluster tuning, a specific Spark version, or long-running sessionsCluster name, instance types, whether it is transient

Job bookmarks change what the diagram means

With bookmarks on, a Glue job processes only what arrived since the last successful run, so the arrow on your diagram means "new partitions". With bookmarks off it means "everything, every time". Two very different pipelines, identical boxes. Write it on the arrow.

5. Redshift Architecture Diagram: Getting Task Events into Redshift

The most common Redshift diagram question is the concrete one: an application emits events, a task starting, a task completing, a task failing, and they have to end up queryable in Redshift. There are three paths, and the diagram should make clear which one you took.

Path 1: Firehose to S3, Glue, then COPY (the default)

The producer writes each task event to Kinesis Data Firehose. Firehose buffers and drops batched objects into raw/events/task_event/dt=. A Glue job types and deduplicates them into Parquet under curated/fact_task_event/. Redshift then runs COPY from that prefix into a staging table and MERGE into the fact table.

Draw: the staging table as its own box. It is where duplicate loads get caught, and a diagram that jumps straight from S3 to the fact table hides the step everybody debugs.

Path 2: Redshift streaming ingestion (near real time)

Redshift reads a Kinesis Data Stream or an MSK topic directly through an external schema, and a materialized view over that schema is what analysts query. No S3 hop, no Glue job, latency measured in seconds rather than in the Firehose buffer interval.

Draw: the materialized view and its refresh, plus the stream retention period. The lake no longer holds a replayable copy on this path, which is a trade the diagram should show rather than hide.

Path 3: Leave it in S3, query with Spectrum

Nothing is loaded. Redshift Spectrum reads the curated prefix in place through the Glue Data Catalog, joined against dimension tables that do live in Redshift. Good for high volume event history that is queried occasionally.

Draw: an external schema box between Redshift and S3, and mark which tables are local and which are external. That distinction drives query performance more than anything else in the picture.

Athena or Redshift for the query layer

Athena is the right default for exploration and for anything read a handful of times a day: nothing to size, billed on bytes scanned, reading the same catalog tables the Glue jobs wrote. Redshift earns its place when dashboards need consistent response time under concurrency, when joins span many wide tables, or when the same aggregate is recomputed hundreds of times a day and materializing it is cheaper than rescanning. Most platforms end up with both, and the diagram should show them side by side reading the same curated zone, not stacked as if one replaced the other.

From experience

Every team I have seen regret their event pipeline regretted the same thing: writing events into Redshift row by row from the application. It works at ten events a second and falls over at a thousand. Buffer first, load in batches, and put the buffer on the diagram so the next person does not remove it.

6. Four AWS Data Architecture Patterns to Copy

Don't start from scratch. These patterns solve most use cases. Pick one, customize it, then diagram it.

Pattern 1: Medallion Data Lake

The most common pattern, and the one in the diagram at the top of this page. Data progresses through three quality zones: Bronze (raw), Silver (cleaned), Gold (aggregated).

Architecture flow:

Bronze: S3 raw bucket, Glue crawler, Athena queries
Silver: Glue ETL jobs, S3 processed bucket, partitioned Parquet
Gold: aggregations, Redshift Spectrum or Athena, BI dashboards

Best for: Teams wanting clear data quality boundaries and flexible consumption (both Athena and Redshift can read Gold layer).

Pattern 2: Streaming + Batch Hybrid

Real-time ingestion with batch processing. Combines the speed of streaming with the reliability of batch.

Architecture flow:

Streaming: Kinesis Data Streams, Lambda or Managed Service for Apache Flink, S3
Batch: scheduled Glue jobs, transform and aggregate, Redshift
Serving: Redshift for dashboards, Athena for exploration

Best for: Real-time dashboards (streaming) + historical analysis (batch). Common in e-commerce and SaaS.

Pattern 3: Serverless Analytics (the AWS Lambda diagram)

Pure serverless with no cluster management, and the pattern people mean when they search for an AWS Lambda architecture diagram on the data side.

Architecture flow:

Ingestion: API Gateway + Lambda, S3 raw bucket
Transform: S3 event, Lambda functions, Parquet in S3
Query: Athena + QuickSight (no Redshift cluster)

Best for: Startups, cost-conscious teams, or workloads with unpredictable query patterns. The AWS serverless architecture diagram guide covers the event-driven side in detail.

Pattern 4: Customer Data Platform (CDP) on AWS

The same five layers with identity resolution added in the middle, which is the piece that makes a CDP architecture different from a plain analytics platform.

Architecture flow:

Collect: behavioural events through Kinesis, CRM and support records through AppFlow
Resolve: a Glue job stitching device, email and account identifiers into one profile key
Serve: the profile in Redshift for analysis and DynamoDB for low latency lookups

Best for: Teams activating customer data into messaging and personalization. Draw the identity resolution job as its own box: it is the part that breaks, and the part auditors ask about.

Pro Tip

Start with Medallion, Add Complexity Later

The medallion pattern gives you clear zones for debugging and quality gates. You can always add real-time streams later. Teams that start with complex architectures often regret it when debugging gets hard.

7. How to Draw an AWS Data Flow Diagram, Step by Step

Follow these steps to produce a diagram that's both accurate and readable. Start simple, then add detail. The rule that keeps a data flow diagram honest: every arrow is a movement of data, not an API call, so a scheduled pull from a SaaS API still points toward S3.

1

Map data sources and consumers

Start at the boundaries. What data comes in? Where does it go out? List: databases, APIs, files, SaaS tools. List: dashboards, ML models, operational apps.

Action: Draw boxes for sources (left) and consumers (right).

2

Define your storage layers

Decide on your S3 bucket structure. Most teams use: raw/, processed/, curated/ or bronze/, silver/, gold/.

Action: Add S3 buckets in the middle. Label them clearly with prefixes.

3

Add ingestion paths

How does data get from sources to S3? Options: AWS Database Migration Service, Lambda, Glue, Kinesis Firehose, third-party tools (Airbyte, Fivetran).

Action: Draw arrows from sources to raw S3. Label with service names.

4

Map transformation flows

Show how raw data becomes clean data. Usually Glue jobs or Lambda functions. Include: job names, triggers (schedule/event), output format (Parquet/CSV).

Action: Draw Glue/Lambda boxes between S3 layers. Show progression.

5

Add serving layer

How do consumers query data? Athena for ad-hoc, Redshift for dashboards, Redshift Spectrum for hybrid. Show which tables/views are exposed.

Action: Add Athena/Redshift boxes. Draw arrows to consumers.

6

Include governance and security

Show IAM roles (who can access what), encryption (S3 SSE, Redshift), VPC boundaries, CloudTrail logging. This is critical for compliance.

Action: Add security annotations: KMS keys, IAM role names, VPC ID.

7

Add metadata and monitoring

Document: Glue Data Catalog tables, CloudWatch alarms, Step Functions workflows, data quality checks (Lambda or Glue).

Action: Annotate with catalog table names, alarm thresholds, SNS topics.

Generating it instead of drawing it

Two shortcuts are worth knowing. The first is describing the platform in a sentence and letting the AWS architecture generator draw it with the real service icons, then editing the canvas. The second applies if the platform is defined in Terraform, which most AWS data platforms are: Datadef's repository sync connects to GitHub, GitLab or Azure DevOps read only, parses each .tf file directly with no terraform init, no state file and no cloud credentials, draws modules as zones, and redraws daily. A commit that changes nothing structural leaves the diagram alone, and nodes you moved by hand keep their position. The AWS Terraform diagram generator guide walks through it.

From experience

I always start by sketching on a whiteboard with the team. Get everyone to agree on the flow before you make it pretty in a tool. 30 minutes of whiteboarding saves hours of rework.

8. Why the AWS Data Platform Diagram Is Worth Keeping

Your AWS data platform is invisible. Services run in the cloud, data flows through APIs, and failures happen silently. A good architecture diagram is the map that makes the invisible visible: showing how data moves, where it's stored, and who can access it.

The Cost of Poor Documentation

Without clear diagrams, teams spend a large share of their week hunting through CloudFormation stacks, checking S3 bucket policies, and trying to figure out which Glue job writes to which table. On a senior engineer that archeology is measured in hours per day, not minutes.

A good AWS diagram delivers three outcomes:

Faster Debugging

When data is missing, the diagram shows exactly which S3 bucket, Glue job, and Redshift table to check. No more guessing.

Onboarding

New engineers understand the entire platform in 30 minutes instead of 3 months. Show them the diagram, not CloudFormation YAML.

Security Review

Auditors and security teams can see IAM boundaries, encryption zones, and network isolation at a glance.

From experience

I've joined teams where the only documentation was "check the console." It took weeks to understand what was running. The teams with clear diagrams? I was productive on day one. Documentation isn't overhead: it's velocity.

9. AWS Diagram Best Practices

The difference between a useful diagram and shelf-ware is these details. Make diagrams that teams actually use. Start with the official service icons (our AWS icons library has the full downloadable set) so every service is recognizable at a glance.

Do This

  • Use real resource names: "s3://prod-raw-data" not "Raw Bucket"
  • Show data formats: Parquet, JSON, CSV, Avro
  • Include IAM role names for security review
  • Label partition keys: year/month/day, user_id, region
  • Add data volume estimates: 10 GB/day, 1M rows/hour
  • Show SLA/schedule: Hourly, Daily 3 AM UTC, Event-driven
  • Link to code repos: GitHub links for Glue jobs

Avoid This

  • • Generic labels like "Database" or "ETL Process"
  • • Showing every single table (group related tables)
  • • Missing arrows (which direction does data flow?)
  • • Outdated diagrams (sync with reality or delete)
  • • Screenshots pasted into a wiki that nobody can edit
  • • Missing security zones (VPC, encryption, IAM)
  • • No contact/owner information

Layering Strategy for Complex Diagrams

🌐

Layer 1: High-Level

The view showing sources, data lake, serving and consumers. For executives and new hires.

🔧

Layer 2: Logical

Service-level detail: specific S3 buckets, Glue jobs, Redshift clusters. For data engineers.

⚙️

Layer 3: Physical

Resource IDs, IAM ARNs, VPC details, CloudWatch alarm names. For debugging and operations.

Pro Tip

Keep One Source of Truth

Store diagrams where your code lives (GitHub, GitLab) or use a tool with version history. A flat screenshot in a wiki goes out of sync the first time the pipeline changes. A live embed is one markdown line that renders the current canvas in a README, Notion or Confluence, and it is readable without an account.

10. Tools for AWS Diagrams

ToolBest ForAWS IconsDiagram comes fromCollaboration
DatadefAI-generated diagramsFull libraryText prompt or repo syncReal-time
LucidchartProfessional diagramsOfficial iconsManual canvasGood
draw.ioFree and flexibleImport libraryManual canvas or XMLBasic
MiroBrainstormingManual importManual canvasExcellent
CloudCraftAWS-specific 3DNativeLive AWS account scanBasic
Terraform GraphIaC auto-generatedFrom codeterraform graph outputGit-based

Manual Diagramming

You design the diagram from scratch using drag-and-drop tools.

Best for: Custom diagrams, presentations, precise control over layout and styling.

AI-Generated

Describe your architecture in text, AI generates the diagram.

Best for: Quick drafts, onboarding docs, iterating fast on architecture options.

A wider comparison, tested on the same architecture, is in the best AWS architecture diagram tools guide.

11. AWS Diagram Design Checklist

Clear data flow direction

Arrows show left-to-right or top-to-bottom flow. No circular dependencies unless intentional (feedback loops).

Real resource names

Use actual S3 bucket names, Glue job names, Redshift cluster IDs. Makes the diagram operational, not just conceptual.

Security boundaries marked

Show VPC boundaries, IAM roles, encryption (KMS keys), and network isolation. Critical for audits.

Data formats specified

Label file formats (Parquet, JSON, CSV), compression (Snappy, Gzip), and partitioning schemes.

Processing schedules noted

Indicate event-driven, hourly, or daily 3 AM UTC. This is what answers the question of why data is late.

Volume and scale estimates

Add data volume (GB/day, rows/sec) and query concurrency. Helps with capacity planning.

Monitoring and alerting

Show CloudWatch alarms, SNS topics, and what triggers alerts (job failures, data quality, latency).

Version and owner

Date the diagram, add contact info, link to runbook. Prevents the question of whether it is still accurate.

If the platform spans several AWS accounts, one per environment or one per domain, draw the account boundary before anything else. The multi-account AWS architecture diagram guide covers how to keep that readable, and keeping an AWS diagram up to date covers what to do after it ships.

Pro Tip

Test Your Diagram with New Hires

If a new engineer can't understand your data flow from the diagram in 15 minutes, it's too complex or too vague. Simplify or add layers. The best diagrams are self-explanatory.

12. Frequently Asked Questions

What does an AWS data platform architecture look like?

Five layers, drawn left to right. Ingestion with AWS DMS for database change data capture, Kinesis Data Firehose for events and AppFlow for SaaS records. Storage in one S3 data lake split into raw, processed and curated prefixes. Transformation with Glue jobs between those prefixes. Serving through Athena for ad hoc SQL and Redshift for BI. Consumption by QuickSight, SageMaker and operational apps.

What is the AWS data lake architecture on S3?

One bucket per environment with three prefixes inside it. raw/ holds files exactly as delivered, usually JSON or gzip, partitioned by ingest date. processed/ holds typed and deduplicated Parquet with Snappy compression. curated/ holds business tables at a defined grain. A Glue crawler registers each prefix as a table in the Glue Data Catalog so Athena and Redshift Spectrum can read it.

What does an AWS Glue architecture diagram show?

Four things around every job: the crawler that discovers the schema, the Glue Data Catalog database and table it writes, the trigger that starts the job (a schedule, an EventBridge event or a Glue workflow) and the S3 prefix or Redshift table it outputs to. Draw the catalog once at the side, because Athena, Redshift Spectrum and EMR all read the same entry.

When should I use AWS Glue instead of Lambda?

Use Lambda when the work finishes inside its fifteen minute limit and fits one file at a time: format conversion, validation, small enrichments. Use Glue when the job has to shuffle or join across a whole partition, when it runs longer than a Lambda timeout, or when you want Spark, job bookmarks and Data Catalog integration without operating a cluster.

How do I store task events in Redshift?

Do not insert events row by row. Send them to Kinesis Data Firehose, let it buffer and write batched objects into an S3 raw prefix, transform them with a Glue job into typed Parquet, then COPY that prefix into a Redshift staging table and MERGE into the fact table. For near real time, Redshift streaming ingestion reads a Kinesis Data Stream through a materialized view.

Should I use Redshift or Athena for analytics?

Use Athena for ad-hoc queries, exploration, and cost-sensitive workloads billed per query. Use Redshift for high-performance dashboards, complex joins, and predictable high-volume workloads. Many teams use both: Athena for data scientists reading the lake directly, Redshift for production BI where concurrency and response time are the constraint.

What is the medallion architecture in AWS?

The medallion architecture organizes data into three layers in S3: Bronze (raw ingested data), Silver (cleaned and conformed data), and Gold (aggregated business-level data). On AWS these map to the raw, processed and curated prefixes of one bucket, with a Glue job between each pair. The pattern gives clear data quality progression and separates concerns across the pipeline.

How do I draw an AWS data flow diagram?

Put sources on the left and consumers on the right, then fill the middle: ingestion (Kinesis, Lambda, Glue or DMS), storage (S3 prefixes by zone), transformation (Glue or EMR), serving (Redshift or Athena), consumers (dashboards, ML). Include bucket names, IAM roles and Glue job names so the diagram is operable, and label each arrow with what moves and how often.

How do I generate an AWS architecture diagram automatically?

Two routes. Describe the platform in a sentence and let a generator draw it with the real AWS service icons, then edit the canvas. Or point a tool at the infrastructure code: Datadef connects to a GitHub, GitLab or Azure DevOps repository read only, parses each Terraform file without state or cloud credentials, and redraws the diagram daily.

What is a CDP architecture on AWS?

A customer data platform on AWS is the same five layers with identity resolution in the middle. Behavioural events arrive through Kinesis, SaaS records through AppFlow, and both land in S3. A Glue job stitches identifiers into one profile key. The unified profile sits in Redshift for analysis and DynamoDB for low latency lookups, then feeds activation tools.

What are the core AWS services for a data platform?

The core AWS services for modern data platforms are: S3 (data lake storage), AWS Glue (ETL and data catalog), Athena (serverless SQL queries), Redshift (data warehouse), Kinesis (streaming data), Lambda (serverless compute), DMS (database replication) and Step Functions (workflow orchestration).

How often should I update my AWS architecture diagram?

Update diagrams when you add new data sources, change storage structure, modify ETL logic, or add security controls. Treat diagrams as living documentation, review quarterly and update immediately after major changes. Outdated diagrams are worse than no diagrams, because people act on them.

Should I include cost estimates in my AWS diagram?

Yes, especially for budget reviews. Annotate each box with what drives spend: S3 storage volume, Glue DPU-hours, Redshift node-hours, Athena bytes scanned. It shows stakeholders where the money goes and justifies optimization work, and it makes the argument for converting a zone to Parquet legible to people who never write a query.

Generate Your AWS Data Platform Diagram

Describe your AWS architecture in plain English and get an editable diagram with the real service icons in seconds.