AWS Glue: ETL and Data Catalogue
Run serverless ETL jobs with AWS Glue, register table schemas in the Glue Data Catalogue, and crawl new data automatically.
AWS Glue: ETL and Data Catalogue is a free AWS Solutions Architect lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AWS Solutions Architect learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is AWS Glue?
AWS Glue is a fully managed, serverless ETL (Extract, Transform, Load) service. You do not provision or manage any servers — Glue allocates Spark or Python workers automatically when a job runs. Glue consists of two main components: the ETL engine for data transformation jobs and the Data Catalogue for storing metadata about your data sources and targets.
Glue Data Catalogue Explained
The Glue Data Catalogue is a centralised metadata repository compatible with the Apache Hive Metastore. It stores databases, tables, column definitions, data types, and partition information. Services such as Athena, Redshift Spectrum, and EMR all use the same Data Catalogue, making it a single source of truth for what data exists and where it lives in S3.
# List databases in the Glue Data Catalogue
aws glue get-databases --query 'DatabaseList[*].Name'
# List tables in a database
aws glue get-tables --database-name my_db --query 'TableList[*].Name'Glue Crawlers: Auto-Discovering Schema
A Glue Crawler connects to a data store (S3, JDBC, DynamoDB), samples the data, and automatically infers the schema and partition structure. It then writes or updates table definitions in the Data Catalogue. Crawlers can be run on a schedule or triggered on demand. They support incremental crawling so they only process new partitions on subsequent runs.
# Create and start a Glue Crawler via CLI
aws glue create-crawler \
--name sales-crawler \
--role arn:aws:iam::123456789012:role/GlueRole \
--database-name my_db \
--targets '{"S3Targets": [{"Path": "s3://my-data-lake-123/landing/sales/"}]}'
aws glue start-crawler --name sales-crawlerWriting a Glue ETL Job
A Glue ETL job is a PySpark or Scala Spark script that reads from a source, applies transformations using the DynamicFrame API, and writes to a target. DynamicFrames extend Spark DataFrames with schema flexibility: they handle inconsistent data types and nested JSON automatically. You can generate a boilerplate job script from the Glue Studio visual editor.
# Minimal Glue PySpark ETL job
from awsglue.context import GlueContext
from awsglue.transforms import *
from pyspark.context import SparkContext
sc = SparkContext()
glueContext = GlueContext(sc)
# Read from Data Catalogue table
source = glueContext.create_dynamic_frame.from_catalog(
database='my_db', table_name='raw_sales')
# Drop null order_id rows
cleaned = Filter.apply(frame=source, f=lambda r: r['order_id'] is not None)
# Write Parquet to curated zone
glueContext.write_dynamic_frame.from_options(
frame=cleaned,
connection_type='s3',
connection_options={'path': 's3://my-data-lake-123/curated/sales/'},
format='parquet')Glue Job Workers and DPUs
Glue allocates capacity in Data Processing Units (DPUs). One DPU equals 4 vCPUs and 16 GB of memory. You choose a worker type (G.1X, G.2X, or G.025X for flex/spark streaming) and a number of workers. For small jobs, use the G.025X Python Shell job type which uses a single DPU-quarter and is very cost-effective for simple transformations.
# Create a Glue job with G.1X workers
aws glue create-job \
--name clean-sales \
--role arn:aws:iam::123456789012:role/GlueRole \
--command '{"Name": "glueetl", "ScriptLocation": "s3://scripts/clean_sales.py", "PythonVersion": "3"}' \
--worker-type G.1X \
--number-of-workers 5 \
--glue-version '4.0'Glue Workflows and Triggers
A Glue Workflow chains crawlers and jobs into a dependency graph. A crawler discovers new data, then on-completion triggers a job, then another job, and so on. Triggers can be scheduled (cron), event-based (on-demand), or conditional (fire when a job succeeds or fails). Workflows give you a visual DAG for your ETL pipeline without a separate orchestration service.
# Create a schedule trigger that starts a job every night at 2 AM UTC
aws glue create-trigger \
--name nightly-clean \
--type SCHEDULED \
--schedule 'cron(0 2 * * ? *)' \
--actions '[{"JobName": "clean-sales"}]' \
--start-on-creationGlue Studio: Visual ETL Builder
Glue Studio provides a drag-and-drop interface to design ETL jobs graphically without writing PySpark code manually. You connect source nodes (S3, Catalogue tables, JDBC) through transform nodes (filter, join, aggregate, custom Python) to target nodes. Glue Studio generates the Spark script automatically. It also provides a job run dashboard to monitor execution status and data quality metrics.
Glue Data Quality
AWS Glue Data Quality (DQDL — Data Quality Definition Language) lets you write rules to validate data as it flows through an ETL job. Rules can check for null rates, value ranges, referential integrity, and uniqueness. If data fails quality rules, Glue can halt the job or route bad records to a quarantine path in S3 for manual review rather than silently passing corrupt data to the curated zone.
# DQDL rule example for Glue Data Quality
# Reject rows where order_total is negative or null
Rules = [
IsComplete 'order_total',
ColumnValues 'order_total' >= 0
]Glue vs Other ETL Options
For the SAA-C03 exam, know when to recommend Glue versus alternatives. Use Glue for serverless Spark ETL and Data Catalogue integration. Use AWS Data Pipeline for orchestrating older data movement tasks (mostly legacy). Use Amazon EMR when you need custom Hadoop/Spark cluster configuration or long-running workloads. Use Lambda for lightweight, event-driven micro-transformations on small payloads.
JDBC and Non-S3 Data Sources
Glue can connect to relational databases via JDBC connections — RDS, Aurora, Redshift, or on-premises databases through a Glue VPC Connection. You define a connection with a JDBC URL, credentials from Secrets Manager, and a VPC security group. Glue then uses a dedicated network interface placed inside your VPC subnet to reach private database endpoints without traversing the public internet.
# Create a Glue JDBC connection to RDS Aurora
aws glue create-connection \
--connection-input '{
"Name": "aurora-prod",
"ConnectionType": "JDBC",
"ConnectionProperties": {
"JDBC_CONNECTION_URL": "jdbc:mysql://aurora-cluster.cluster-xyz.us-east-1.rds.amazonaws.com:3306/sales",
"SECRET_ID": "arn:aws:secretsmanager:us-east-1:123:secret:rds-creds"
},
"PhysicalConnectionRequirements": {
"SubnetId": "subnet-abc123",
"SecurityGroupIdList": ["sg-xyz456"],
"AvailabilityZone": "us-east-1a"
}
}'Bookmarks: Incremental Processing
By default, a Glue job would reprocess every record on every run. Glue Job Bookmarks track which input files have already been processed, so subsequent runs only pick up new data. Bookmarks are essential for append-only S3 sources where the same prefix receives new files daily. Enable bookmarks in the job parameters with --job-bookmark-option job-bookmark-enable.
# Start Glue job with bookmark enabled
aws glue start-job-run \
--job-name clean-sales \
--arguments '{"--job-bookmark-option": "job-bookmark-enable"}'Quick Check
Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.
Lesson Recap
In this lesson you learned: AWS Glue provides serverless ETL via PySpark and the DynamicFrame API, Glue Crawlers auto-discover schemas and populate the Data Catalogue, and Glue Bookmarks enable incremental processing of new S3 data. Next up we explore Amazon Athena for serverless SQL queries directly on S3.
Frequently asked questions
Is the “AWS Glue: ETL and Data Catalogue” lesson free?
Yes — the full text of “AWS Glue: ETL and Data Catalogue” is free to read here on the web, and the AWS Solutions Architect course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AWS Solutions Architect course, upgrade to CoddyKit PRO.
What will I learn in “AWS Glue: ETL and Data Catalogue”?
Run serverless ETL jobs with AWS Glue, register table schemas in the Glue Data Catalogue, and crawl new data automatically. You practise AWS Solutions Architect with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AWS Solutions Architect?
No prior experience is required. AWS Solutions Architect on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “AWS Glue: ETL and Data Catalogue” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AWS Solutions Architect lesson?
Yes. Every AWS Solutions Architect lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Building a Data Lake on S3
- AWS Glue: ETL and Data Catalogue
- Amazon Athena: Serverless SQL on S3
- Kinesis Streams, Firehose, and Real-Time Analytics