Offloading Storage to S3-Compatible Buckets
Stream uploads directly to S3/MinIO with presigned URLs to keep the API stateless and scalable.
Why Offload Storage to S3?
Storing uploaded files on your API server's local disk does not scale. Every replica would need its own copy, disks fill up, and containers are ephemeral — restart the pod and the files vanish.
The fix is to push media to an external object store and keep your API stateless. Any S3-compatible service works:
- Amazon S3 — the original, fully managed.
- MinIO — self-hosted, S3 API-compatible, great for dev and on-prem.
- Cloudflare R2, Backblaze B2, DigitalOcean Spaces — cheaper egress, same API.
Because they all speak the S3 protocol, the same boto3 client code targets any of them just by changing the endpoint URL.
Configuring a boto3 S3 Client
The boto3 library is the standard AWS SDK for Python. To target a non-AWS provider like MinIO or R2, pass an explicit endpoint_url.
Keep credentials and the endpoint in settings, never hard-coded. Use config=Config(signature_version="s3v4") so presigned URLs are generated with the modern SigV4 algorithm that all providers accept.
import boto3
from botocore.config import Config
def build_s3_client():
return boto3.client(
"s3",
endpoint_url="https://s3.eu-central-1.amazonaws.com",
aws_access_key_id="AKIA...",
aws_secret_access_key="secret...",
region_name="eu-central-1",
config=Config(signature_version="s3v4"),
)
client = build_s3_client()
print(type(client).__name__)All lessons in this course
- Multipart Uploads and Content Validation
- Streaming Responses and Range Requests
- Offloading Storage to S3-Compatible Buckets
- Async Image and Document Transformation