Building a Data Lake on S3
Design an S3-based data lake with a landing, processing, and curated zone, apply bucket policies, and organise data by partition for query efficiency.
Building a Data Lake on S3 is a free AWS Solutions Architect lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AWS Solutions Architect learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is a Data Lake?
A data lake is a centralised repository that stores structured, semi-structured, and unstructured data at any scale. Unlike a data warehouse, a data lake stores data in its raw, native format until it is needed for analysis. Amazon S3 is the most common foundation for data lakes on AWS because of its durability, scalability, and integration with analytics services.
Data Lake Zones Architecture
A well-designed S3 data lake uses three logical zones: the Landing Zone (raw ingest, untouched), the Processing Zone (cleansed and transformed), and the Curated Zone (analytics-ready, business-consumable). Each zone is typically a separate S3 prefix or bucket. This pattern is sometimes called a medallion architecture (bronze, silver, gold).
# Example zone structure inside one S3 bucket
# s3://my-data-lake/
# landing/ <- raw ingest from source systems
# processing/ <- cleansed, validated data
# curated/ <- aggregated, analytics-readyCreating the S3 Bucket Structure
Use the AWS CLI to create a versioned, encrypted S3 bucket and apply prefixes for each zone. Enable versioning so that reprocessing can always go back to the raw source, and enable server-side encryption (SSE-S3 or SSE-KMS) for data at rest. Block all public access to keep data private.
aws s3api create-bucket \
--bucket my-data-lake-123 \
--region us-east-1
aws s3api put-bucket-versioning \
--bucket my-data-lake-123 \
--versioning-configuration Status=Enabled
aws s3api put-bucket-encryption \
--bucket my-data-lake-123 \
--server-side-encryption-configuration \
'{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'Applying Bucket Policies for Zone Access
Each zone should have its own access policy so that different teams and services can only touch what they need. For example, data ingestion roles get s3:PutObject on the landing prefix, ETL roles get read access on landing and write on processing, and analytics roles get read-only on curated. This enforces least-privilege inside the data lake.
# Attach a policy that allows the ETL role to read landing/ and write processing/
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::123456789012:role/ETLRole"},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-data-lake-123/landing/*"
},
{
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::123456789012:role/ETLRole"},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my-data-lake-123/processing/*"
}
]
}Partitioning Data for Query Efficiency
Partitioning organises data in S3 by folder hierarchies that map to query predicates (e.g., year/month/day or region/service). When Athena or Glue reads partitioned data, it only scans the relevant partitions rather than the entire dataset, which dramatically reduces cost and query time. A good partition key is one that appears frequently in WHERE clauses.
# Hive-style partition naming for year/month/day
# s3://my-data-lake-123/curated/sales/
# year=2024/month=01/day=15/part-00000.parquet
# year=2024/month=01/day=16/part-00000.parquet
# year=2024/month=02/day=01/part-00000.parquet
# Athena recognises this naming automaticallyColumnar Formats: Parquet and ORC
Storing data in a columnar format such as Apache Parquet or ORC (Optimised Row Columnar) dramatically improves analytical query performance and lowers S3 data-scan costs. Columnar formats allow query engines to read only the columns needed, compress repetitive values efficiently, and support predicate pushdown. Always convert raw CSV or JSON to Parquet in the curated zone.
# Converting CSV to Parquet with AWS Glue (simplified PySpark)
import sys
from awsglue.context import GlueContext
from pyspark.context import SparkContext
sc = SparkContext()
glueContext = GlueContext(sc)
datasource = glueContext.create_dynamic_frame.from_catalog(
database='my_db', table_name='raw_sales')
glueContext.write_dynamic_frame.from_options(
frame=datasource,
connection_type='s3',
connection_options={'path': 's3://my-data-lake-123/curated/sales/'},
format='parquet')S3 Lifecycle Policies for Cost Management
Data in the landing zone grows continuously but older raw files are rarely re-accessed. Use S3 Lifecycle Policies to automatically transition raw data to cheaper storage classes over time. For example, move objects in landing/ to S3 Glacier Instant Retrieval after 30 days and Glacier Deep Archive after 90 days. This alone can cut storage costs by 70–90% for historical data.
aws s3api put-bucket-lifecycle-configuration \
--bucket my-data-lake-123 \
--lifecycle-configuration '{
"Rules": [{
"ID": "ArchiveLanding",
"Filter": {"Prefix": "landing/"},
"Status": "Enabled",
"Transitions": [
{"Days": 30, "StorageClass": "GLACIER_IR"},
{"Days": 90, "StorageClass": "DEEP_ARCHIVE"}
]
}]
}'Lake Formation for Fine-Grained Access
AWS Lake Formation sits on top of S3 and the Glue Data Catalogue to provide table-level, column-level, and row-level access control without writing complex bucket policies. Lake Formation integrates with Athena, Redshift Spectrum, and EMR. It is the preferred approach when multiple teams query the same data lake and need different visibility into sensitive columns like PII or financial data.
# Grant Lake Formation table access via CLI
aws lakeformation grant-permissions \
--principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/AnalystRole \
--permissions SELECT \
--resource '{
"Table": {
"DatabaseName": "my_db",
"Name": "curated_sales"
}
}'S3 Event Notifications for Ingestion Triggers
When a new file lands in the S3 landing zone, you need to automatically trigger processing. Use S3 Event Notifications to publish an event to SQS, SNS, or Lambda whenever an object is created. A Lambda function or Glue workflow then picks up the new file, validates it, and moves it through the pipeline. This creates a fully automated, event-driven data lake ingestion process.
# S3 event notification to trigger Lambda on new object
aws s3api put-bucket-notification-configuration \
--bucket my-data-lake-123 \
--notification-configuration '{
"LambdaFunctionConfigurations": [{
"LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:ProcessNewFile",
"Events": ["s3:ObjectCreated:*"],
"Filter": {
"Key": {"FilterRules": [{"Name": "prefix", "Value": "landing/"}]}
}
}]
}'Encryption and Compliance in the Data Lake
A production data lake must enforce encryption everywhere. Use AWS KMS Customer Managed Keys (CMK) for SSE-KMS on sensitive data, requiring explicit key grants for each consumer. Enable S3 Object Lock in Compliance mode for regulatory data that must not be deleted or overwritten. Use Macie to automatically discover and alert on PII stored in the lake.
# Enforce KMS encryption on all PUT operations via bucket policy
{
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my-data-lake-123/curated/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": "aws:kms"
}
}
}Cross-Account Data Lake Access
In large organisations, the data lake S3 bucket lives in a central data platform account while consumer teams operate in separate AWS accounts. Grant access using a combination of bucket policies (listing the consumer account principal) and IAM roles in the consumer account that assume cross-account permissions. Resource Access Manager (RAM) is an alternative for Lake Formation–based sharing.
# Bucket policy in central account allows consumer account to read curated/
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::999988887777:root"
},
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::my-data-lake-123",
"arn:aws:s3:::my-data-lake-123/curated/*"
]
}Quick Check
Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.
Lesson Recap
In this lesson you learned: data lakes use S3 with landing, processing, and curated zones, partitioning and Parquet format reduce Athena query costs, and Lake Formation provides fine-grained column and row-level access control. Next up we explore AWS Glue for serverless ETL and the Glue Data Catalogue.
Frequently asked questions
Is the “Building a Data Lake on S3” lesson free?
Yes — the full text of “Building a Data Lake on S3” is free to read here on the web, and the AWS Solutions Architect course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AWS Solutions Architect course, upgrade to CoddyKit PRO.
What will I learn in “Building a Data Lake on S3”?
Design an S3-based data lake with a landing, processing, and curated zone, apply bucket policies, and organise data by partition for query efficiency. You practise AWS Solutions Architect with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AWS Solutions Architect?
No prior experience is required. AWS Solutions Architect on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Building a Data Lake on S3” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AWS Solutions Architect lesson?
Yes. Every AWS Solutions Architect lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.