0Pricing
FastAPI Backend Development Bootcamp · Lesson

Offloading Storage to S3-Compatible Buckets

Stream uploads directly to S3/MinIO with presigned URLs to keep the API stateless and scalable.

Offloading Storage to S3-Compatible Buckets is a free FastAPI Backend Development Bootcamp lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the FastAPI Backend Development Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Offload Storage to S3?

Storing uploaded files on your API server's local disk does not scale. Every replica would need its own copy, disks fill up, and containers are ephemeral — restart the pod and the files vanish.

The fix is to push media to an external object store and keep your API stateless. Any S3-compatible service works:

  • Amazon S3 — the original, fully managed.
  • MinIO — self-hosted, S3 API-compatible, great for dev and on-prem.
  • Cloudflare R2, Backblaze B2, DigitalOcean Spaces — cheaper egress, same API.

Because they all speak the S3 protocol, the same boto3 client code targets any of them just by changing the endpoint URL.

Configuring a boto3 S3 Client

The boto3 library is the standard AWS SDK for Python. To target a non-AWS provider like MinIO or R2, pass an explicit endpoint_url.

Keep credentials and the endpoint in settings, never hard-coded. Use config=Config(signature_version="s3v4") so presigned URLs are generated with the modern SigV4 algorithm that all providers accept.

import boto3
from botocore.config import Config

def build_s3_client():
    return boto3.client(
        "s3",
        endpoint_url="https://s3.eu-central-1.amazonaws.com",
        aws_access_key_id="AKIA...",
        aws_secret_access_key="secret...",
        region_name="eu-central-1",
        config=Config(signature_version="s3v4"),
    )

client = build_s3_client()
print(type(client).__name__)

The Naive Approach (and Why It Hurts)

The obvious first attempt is to read the whole upload into memory, then hand it to S3:

  • data = await file.read() loads the entire file into RAM.
  • A 2 GB video upload becomes 2 GB of process memory — multiply by concurrent requests and your workers OOM-crash.

This works for tiny avatars but is dangerous for media. We want to stream bytes through the API (or skip the API entirely with presigned URLs). The next scenes build up both techniques.

from fastapi import FastAPI, UploadFile

app = FastAPI()

@app.post("/upload-naive")
async def upload_naive(file: UploadFile):
    data = await file.read()  # whole file in RAM - avoid for large media!
    return {"size": len(data)}

Streaming Uploads with upload_fileobj

FastAPI's UploadFile wraps a SpooledTemporaryFile: small uploads stay in memory, large ones spill to disk automatically. Its .file attribute is a standard file-like object.

boto3's upload_fileobj reads that stream in chunks and performs a multipart upload under the hood — so memory stays bounded regardless of file size.

from fastapi import FastAPI, UploadFile

app = FastAPI()
BUCKET = "user-media"

@app.post("/upload")
async def upload(file: UploadFile):
    client.upload_fileobj(
        Fileobj=file.file,          # streams in chunks, no full read
        Bucket=BUCKET,
        Key=f"uploads/{file.filename}",
        ExtraArgs={"ContentType": file.content_type},
    )
    return {"key": f"uploads/{file.filename}"}

Don't Block the Event Loop

boto3 is synchronous. Calling upload_fileobj directly inside an async def endpoint blocks the event loop while bytes travel to S3, stalling every other request on that worker.

Offload the blocking call to a thread pool with run_in_threadpool (Starlette) or asyncio.to_thread. Now the event loop stays free to serve other connections.

from fastapi import FastAPI, UploadFile
from fastapi.concurrency import run_in_threadpool

app = FastAPI()
BUCKET = "user-media"

@app.post("/upload")
async def upload(file: UploadFile):
    key = f"uploads/{file.filename}"
    await run_in_threadpool(
        client.upload_fileobj, file.file, BUCKET, key,
        {"ContentType": file.content_type},
    )
    return {"key": key}

Presigned URLs: Let Clients Talk to S3 Directly

Streaming through the API still spends your bandwidth and CPU twice (client→API, API→S3). The most scalable pattern removes the API from the data path entirely using a presigned URL.

A presigned URL is a temporary, signed link that grants permission for one specific operation (PUT or GET) on one object, expiring after N seconds. The client uploads directly to S3; your API only signs the request.

  • API stays stateless and tiny — it never touches the bytes.
  • Credentials never leave the server; the signature encodes the grant.

Generating a Presigned PUT URL

Use generate_presigned_url with the put_object client method to mint an upload link. The endpoint returns the URL plus the final object key; the browser then issues a plain HTTP PUT to that URL with the file body.

Set a short ExpiresIn (e.g. 300–900 seconds) — just long enough to start the upload.

import uuid
from fastapi import FastAPI

app = FastAPI()
BUCKET = "user-media"

@app.post("/uploads/presign")
def presign_put(filename: str, content_type: str):
    key = f"uploads/{uuid.uuid4()}-{filename}"
    url = client.generate_presigned_url(
        ClientMethod="put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
        ExpiresIn=600,
    )
    return {"upload_url": url, "key": key}

The Client-Side Upload Flow

With a presigned PUT URL, the browser uploads with a single request — no multipart form, just the raw body. The flow is:

  • 1. Client asks your API for a presigned URL (sends filename + content type).
  • 2. API returns upload_url and the final key.
  • 3. Client does PUT upload_url with the file bytes and the matching Content-Type header.
  • 4. Client notifies your API of the key so you can persist it in the DB.

The Content-Type on the PUT must match the one you signed, or S3 returns 403.

// Browser-side (illustrative)
const { upload_url, key } = await api.presign(file.name, file.type);
await fetch(upload_url, {
  method: "PUT",
  headers: { "Content-Type": file.type },
  body: file,
});
await api.confirm(key);

Serving Private Files with Presigned GET URLs

Make buckets private by default. To let a user download or view a file, generate a short-lived presigned get_object URL on demand instead of making the object public.

This keeps access controlled by your API's auth: check the user owns the file, then sign a URL valid for a few minutes. Embed it in an <img> src or return it as a redirect.

from fastapi import FastAPI
from fastapi.responses import RedirectResponse

app = FastAPI()
BUCKET = "user-media"

@app.get("/files/{key:path}")
def download(key: str):
    url = client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": BUCKET, "Key": key},
        ExpiresIn=300,
    )
    return RedirectResponse(url)

Constraining Uploads with Presigned POST

A presigned PUT URL cannot limit file size — a malicious client could upload a 50 GB file. When you need server-enforced limits, use generate_presigned_post instead.

It returns a URL plus form fields, and lets you attach conditions like content-length-range and an exact content type. S3 rejects the upload server-side if the bytes violate the policy.

from fastapi import FastAPI

app = FastAPI()
BUCKET = "user-media"

@app.post("/uploads/presign-post")
def presign_post(key: str, content_type: str):
    return client.generate_presigned_post(
        Bucket=BUCKET,
        Key=key,
        Fields={"Content-Type": content_type},
        Conditions=[
            {"Content-Type": content_type},
            ["content-length-range", 1, 10 * 1024 * 1024],  # max 10 MB
        ],
        ExpiresIn=600,
    )

A Reusable Key Builder

Object keys should be collision-proof, organized, and never trust the client's raw filename (which can contain ../ or odd characters). A small helper centralizes this logic and is pure Python — easy to unit test.

A good key includes a logical prefix (owner, category), a UUID for uniqueness, and a sanitized extension.

import re
import uuid

def build_key(user_id: int, filename: str) -> str:
    ext = filename.rsplit(".", 1)[-1].lower() if "." in filename else "bin"
    ext = re.sub(r"[^a-z0-9]", "", ext)[:8] or "bin"
    return f"users/{user_id}/{uuid.uuid4().hex}.{ext}"

print(build_key(42, "My Vacation.JPG"))
print(build_key(7, "../../etc/passwd"))
print(build_key(1, "noext"))

Quick Check: Choosing the Scalable Pattern

You are building an endpoint that lets users upload large videos (up to 2 GB). You want the FastAPI service to stay stateless and to avoid routing the file bytes through the API server at all. Which approach best fits?

Recap: Stateless Media at Scale

You now have a full toolkit for offloading storage to S3-compatible buckets:

  • One client, many providers — boto3 with an endpoint_url targets S3, MinIO, R2, Spaces.
  • Never buffer whole files — if bytes must pass through the API, use upload_fileobj and offload it with run_in_threadpool so the event loop stays free.
  • Prefer presigned URLs — clients PUT/GET directly against S3; the API only signs, staying stateless.
  • Enforce limits with generate_presigned_post and a content-length-range condition.
  • Keep buckets private and serve files via short-lived presigned GET links gated by your auth.
  • Sanitize keys — UUID-based, prefixed, never trusting raw filenames.

The result is an API that handles 2 GB or 2 KB uploads with the same bounded footprint.

Frequently asked questions

Is the “Offloading Storage to S3-Compatible Buckets” lesson free?

Yes — the full text of “Offloading Storage to S3-Compatible Buckets” is free to read here on the web, and the FastAPI Backend Development Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the FastAPI Backend Development Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Offloading Storage to S3-Compatible Buckets”?

Stream uploads directly to S3/MinIO with presigned URLs to keep the API stateless and scalable. You practise FastAPI Backend Development Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start FastAPI Backend Development Bootcamp?

No prior experience is required. FastAPI Backend Development Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Offloading Storage to S3-Compatible Buckets” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this FastAPI Backend Development Bootcamp lesson?

Yes. Every FastAPI Backend Development Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Multipart Uploads and Content Validation
  2. Streaming Responses and Range Requests
  3. Offloading Storage to S3-Compatible Buckets
  4. Async Image and Document Transformation
← Back to FastAPI Backend Development Bootcamp