S3에서 데이터 레이크 구축

데이터 레이크란 무엇입니까

데이터 레이크 영역 아키텍처

# Example zone structure inside one S3 bucket
# s3://my-data-lake/
#   landing/    <- raw ingest from source systems
#   processing/ <- cleansed, validated data
#   curated/    <- aggregated, analytics-ready

S3 버킷 구조 생성

aws s3api create-bucket \
  --bucket my-data-lake-123 \
  --region us-east-1

aws s3api put-bucket-versioning \
  --bucket my-data-lake-123 \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-encryption \
  --bucket my-data-lake-123 \
  --server-side-encryption-configuration \
    '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

영역 액세스를 위한 버킷 정책 적용

# Attach a policy that allows the ETL role to read landing/ and write processing/
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::123456789012:role/ETLRole"},
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-data-lake-123/landing/*"
    },
    {
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::123456789012:role/ETLRole"},
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-data-lake-123/processing/*"
    }
  ]
}

쿼리 효율을 위한 데이터 파티셔닝

# Hive-style partition naming for year/month/day
# s3://my-data-lake-123/curated/sales/
#   year=2024/month=01/day=15/part-00000.parquet
#   year=2024/month=01/day=16/part-00000.parquet
#   year=2024/month=02/day=01/part-00000.parquet

# Athena recognises this naming automatically

열 기반 형식: Parquet 및 ORC

# Converting CSV to Parquet with AWS Glue (simplified PySpark)
import sys
from awsglue.context import GlueContext
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)

datasource = glueContext.create_dynamic_frame.from_catalog(
    database='my_db', table_name='raw_sales')

glueContext.write_dynamic_frame.from_options(
    frame=datasource,
    connection_type='s3',
    connection_options={'path': 's3://my-data-lake-123/curated/sales/'},
    format='parquet')

비용 관리를 위한 S3 수명 주기 정책

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-data-lake-123 \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "ArchiveLanding",
      "Filter": {"Prefix": "landing/"},
      "Status": "Enabled",
      "Transitions": [
        {"Days": 30, "StorageClass": "GLACIER_IR"},
        {"Days": 90, "StorageClass": "DEEP_ARCHIVE"}
      ]
    }]
  }'

세분화된 액세스를 위한 Lake Formation

# Grant Lake Formation table access via CLI
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/AnalystRole \
  --permissions SELECT \
  --resource '{
    "Table": {
      "DatabaseName": "my_db",
      "Name": "curated_sales"
    }
  }'

수집 트리거를 위한 S3 이벤트 알림

# S3 event notification to trigger Lambda on new object
aws s3api put-bucket-notification-configuration \
  --bucket my-data-lake-123 \
  --notification-configuration '{
    "LambdaFunctionConfigurations": [{
      "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:ProcessNewFile",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {"FilterRules": [{"Name": "prefix", "Value": "landing/"}]}
      }
    }]
  }'

데이터 레이크의 암호화 및 규정 준수

# Enforce KMS encryption on all PUT operations via bucket policy
{
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::my-data-lake-123/curated/*",
  "Condition": {
    "StringNotEquals": {
      "s3:x-amz-server-side-encryption": "aws:kms"
    }
  }
}

계정 간 데이터 레이크 액세스

# Bucket policy in central account allows consumer account to read curated/
{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::999988887777:root"
  },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::my-data-lake-123",
    "arn:aws:s3:::my-data-lake-123/curated/*"
  ]
}

빠른 확인

레슨 요약