S3 and Data Transfer Cost Optimisation

Amazon S3 storage costs vary significantly by storage class and access pattern. S3 Standard charges approximately $0.023 per GB per month — at petabyte scale, this becomes significant. The good news is that S3 offers seven storage classes optimised for different access frequencies, with prices ranging from $0.023/GB (Standard) to $0.00099/GB (Glacier Deep Archive). The key to S3 cost optimisation is matching your data to the right storage class and automating transitions with lifecycle policies.

S3 Intelligent-Tiering automatically moves objects between access tiers based on actual access patterns — no retrieval fees, no minimum duration for the frequent tier. It monitors object access at the object level: objects not accessed for 30 days move to the infrequent tier (40% cheaper), and after 90 days to the archive instant tier (68% cheaper). There is a small monthly monitoring fee ($0.0025 per 1,000 objects). Use Intelligent-Tiering when you cannot predict access patterns or when you have mixed hot and cold data in the same bucket.

S3 Lifecycle rules automate object transitions and expirations based on age. You define rules by prefix or tag, then specify transitions at specific day thresholds. A typical archival lifecycle: Standard → Standard-IA (30 days) → Glacier Instant Retrieval (90 days) → Deep Archive (365 days) → Delete (2555 days). This can reduce per-GB storage costs from $0.023 to $0.00099 over time — a 95% reduction for archival data. Lifecycle transitions have a minimum object size of 128KB for IA and Glacier (smaller objects are not cost-effective to transition).

S3 Select allows you to retrieve only the data you need from an S3 object using SQL expressions, rather than downloading the entire object. For a 10 GB CSV file where you only need 100 rows, S3 Select can retrieve just those rows — reducing data transfer costs and application processing time dramatically. Glacier Select does the same for objects in Glacier. S3 Select supports CSV, JSON, and Parquet formats. This is especially valuable in data analytics pipelines where many queries only access a subset of each data file.

For large object uploads, S3 Multipart Upload splits objects into parts (minimum 5 MB per part) that upload in parallel, improving throughput and allowing resumable uploads. Failed parts can be retried without restarting the entire upload. Clean up incomplete multipart uploads with lifecycle rules to avoid being charged for incomplete parts. S3 Transfer Acceleration routes uploads through AWS CloudFront edge locations for faster uploads from distant locations — useful for users uploading large files globally, but adds cost ($0.04-$0.08/GB extra). Use only when acceleration provides measurable improvement.

Data transfer costs are one of the most surprising AWS bills. Key rules: Inbound data transfer (into AWS) is free. Outbound data transfer to the internet costs approximately $0.09/GB (first 10 TB/month). Cross-region data transfer costs approximately $0.02/GB. Cross-AZ data transfer within the same region costs $0.01/GB each way. Within the same AZ is free. Design architectures to minimise cross-region and internet-bound data transfer — this is why regional deployments and CloudFront are critical for cost-efficient content delivery.

Delivering content through Amazon CloudFront dramatically reduces data transfer costs in two ways. First, CloudFront charges lower egress rates than EC2 or S3 direct internet access (~$0.0085/GB vs $0.09/GB). Second, CloudFront caches content at edge locations, reducing the number of times data is served from your origin — potentially serving the same content millions of times from cache with only one origin fetch. S3 to CloudFront data transfer is also free. This makes CloudFront a cost optimisation tool, not just a performance tool.

NAT Gateway data processing charges ($0.045/GB) can become a significant cost when EC2 instances in private subnets frequently access S3 or DynamoDB. Instead, use VPC Gateway Endpoints for S3 and DynamoDB — they route traffic through the AWS network backbone at zero cost and do not require a NAT Gateway for these services. Additionally, VPC Interface Endpoints (PrivateLink) for other AWS services eliminate NAT Gateway processing fees for that service traffic, though Interface Endpoints themselves have a small hourly cost.

S3 charges per API request as well as per GB stored. PUT, COPY, POST, LIST requests cost $0.005 per 1,000; GET, SELECT requests cost $0.0004 per 1,000. For applications that make millions of requests, these costs add up. Optimise by: batching small objects into larger ones (fewer requests), using CloudFront to serve cached responses (reduces GET requests to origin), avoiding list operations where possible (LIST is expensive), and using S3 Batch Operations for bulk operations instead of millions of individual API calls.

For large-scale data transfers between on-premises and AWS, or between AWS services, AWS DataSync is more cost-effective than writing custom transfer code. DataSync uses a purpose-built protocol that achieves up to 10x the throughput of open-source tools and includes built-in data validation, scheduling, and monitoring. Data transferred via DataSync to S3, EFS, or FSx is charged at standard AWS data transfer rates — but DataSync's efficiency means you complete transfers faster, reducing the EC2 compute time (and cost) needed for large migrations.

Amazon S3 Storage Lens provides organisation-wide visibility into S3 storage usage and activity metrics across accounts and regions. It surfaces cost optimisation opportunities like identifying buckets with no lifecycle rules, buckets with incomplete multipart uploads, or storage class distribution. S3 Storage Lens default dashboard is free; the advanced metrics dashboard ($0.20/million objects) adds per-prefix drill-down and data age metrics. Use Storage Lens as your starting point for any S3 cost optimisation exercise to identify the highest-impact buckets.

Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.

In this lesson you learned: S3 Intelligent-Tiering and lifecycle rules reduce storage costs by moving data to cheaper storage classes automatically, CloudFront dramatically reduces data transfer costs compared to serving directly from S3 or EC2, and VPC Gateway Endpoints eliminate NAT Gateway charges for S3 and DynamoDB traffic. S3 Storage Lens provides visibility into optimisation opportunities. Next up we explore KMS, ACM, and encryption patterns.