AWS Glue：ETL 与数据目录

什么是 AWS Glue？

Glue Data Catalogue 详解

# List databases in the Glue Data Catalogue
aws glue get-databases --query 'DatabaseList[*].Name'

# List tables in a database
aws glue get-tables --database-name my_db --query 'TableList[*].Name'

Glue Crawler：自动发现架构

# Create and start a Glue Crawler via CLI
aws glue create-crawler \
  --name sales-crawler \
  --role arn:aws:iam::123456789012:role/GlueRole \
  --database-name my_db \
  --targets '{"S3Targets": [{"Path": "s3://my-data-lake-123/landing/sales/"}]}'

aws glue start-crawler --name sales-crawler

编写 Glue ETL 作业

# Minimal Glue PySpark ETL job
from awsglue.context import GlueContext
from awsglue.transforms import *
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)

# Read from Data Catalogue table
source = glueContext.create_dynamic_frame.from_catalog(
    database='my_db', table_name='raw_sales')

# Drop null order_id rows
cleaned = Filter.apply(frame=source, f=lambda r: r['order_id'] is not None)

# Write Parquet to curated zone
glueContext.write_dynamic_frame.from_options(
    frame=cleaned,
    connection_type='s3',
    connection_options={'path': 's3://my-data-lake-123/curated/sales/'},
    format='parquet')

Glue 作业工作进程与 DPU

# Create a Glue job with G.1X workers
aws glue create-job \
  --name clean-sales \
  --role arn:aws:iam::123456789012:role/GlueRole \
  --command '{"Name": "glueetl", "ScriptLocation": "s3://scripts/clean_sales.py", "PythonVersion": "3"}' \
  --worker-type G.1X \
  --number-of-workers 5 \
  --glue-version '4.0'

Glue 工作流与触发器

# Create a schedule trigger that starts a job every night at 2 AM UTC
aws glue create-trigger \
  --name nightly-clean \
  --type SCHEDULED \
  --schedule 'cron(0 2 * * ? *)' \
  --actions '[{"JobName": "clean-sales"}]' \
  --start-on-creation

Glue Studio：可视化 ETL 构建器

Glue Data Quality

# DQDL rule example for Glue Data Quality
# Reject rows where order_total is negative or null
Rules = [
  IsComplete 'order_total',
  ColumnValues 'order_total' >= 0
]

Glue 与其他 ETL 选项的比较

JDBC 与非 S3 数据源

# Create a Glue JDBC connection to RDS Aurora
aws glue create-connection \
  --connection-input '{
    "Name": "aurora-prod",
    "ConnectionType": "JDBC",
    "ConnectionProperties": {
      "JDBC_CONNECTION_URL": "jdbc:mysql://aurora-cluster.cluster-xyz.us-east-1.rds.amazonaws.com:3306/sales",
      "SECRET_ID": "arn:aws:secretsmanager:us-east-1:123:secret:rds-creds"
    },
    "PhysicalConnectionRequirements": {
      "SubnetId": "subnet-abc123",
      "SecurityGroupIdList": ["sg-xyz456"],
      "AvailabilityZone": "us-east-1a"
    }
  }'

书签：增量处理

# Start Glue job with bookmark enabled
aws glue start-job-run \
  --job-name clean-sales \
  --arguments '{"--job-bookmark-option": "job-bookmark-enable"}'

快速检查

课程回顾