0Pricing
FastAPI Backend Development Bootcamp · レッスン

S3互換バケットへのストレージオフロード

署名付きURLを使ってアップロードをS3／MinIOへ直接ストリーミングし、APIをステートレスかつスケーラブルに保ちます。

「S3互換バケットへのストレージオフロード」はCoddyKit上の無料FastAPI Backend Development Bootcampレッスンです。 これはレッスン3/4です。 下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。 これはFastAPI Backend Development Bootcamp学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 FastAPI Backend Development Bootcampコースには全4レッスンが含まれています。

このレッスンの一部はまだ翻訳されておらず、英語で表示されています。

Why Offload Storage to S3?

Storing uploaded files on your API server's local disk does not scale. Every replica would need its own copy, disks fill up, and containers are ephemeral — restart the pod and the files vanish.

The fix is to push media to an external object store and keep your API stateless. Any S3-compatible service works:

  • Amazon S3 — the original, fully managed.
  • MinIO — self-hosted, S3 API-compatible, great for dev and on-prem.
  • Cloudflare R2, Backblaze B2, DigitalOcean Spaces — cheaper egress, same API.

Because they all speak the S3 protocol, the same boto3 client code targets any of them just by changing the endpoint URL.

Configuring a boto3 S3 Client

The boto3 library is the standard AWS SDK for Python. To target a non-AWS provider like MinIO or R2, pass an explicit endpoint_url.

Keep credentials and the endpoint in settings, never hard-coded. Use config=Config(signature_version="s3v4") so presigned URLs are generated with the modern SigV4 algorithm that all providers accept.

import boto3
from botocore.config import Config

def build_s3_client():
    return boto3.client(
        "s3",
        endpoint_url="https://s3.eu-central-1.amazonaws.com",
        aws_access_key_id="AKIA...",
        aws_secret_access_key="secret...",
        region_name="eu-central-1",
        config=Config(signature_version="s3v4"),
    )

client = build_s3_client()
print(type(client).__name__)

The Naive Approach (and Why It Hurts)

The obvious first attempt is to read the whole upload into memory, then hand it to S3:

  • data = await file.read() loads the entire file into RAM.
  • A 2 GB video upload becomes 2 GB of process memory — multiply by concurrent requests and your workers OOM-crash.

This works for tiny avatars but is dangerous for media. We want to stream bytes through the API (or skip the API entirely with presigned URLs). The next scenes build up both techniques.

from fastapi import FastAPI, UploadFile

app = FastAPI()

@app.post("/upload-naive")
async def upload_naive(file: UploadFile):
    data = await file.read()  # whole file in RAM - avoid for large media!
    return {"size": len(data)}

Streaming Uploads with upload_fileobj

FastAPI's UploadFile wraps a SpooledTemporaryFile: small uploads stay in memory, large ones spill to disk automatically. Its .file attribute is a standard file-like object.

boto3's upload_fileobj reads that stream in chunks and performs a multipart upload under the hood — so memory stays bounded regardless of file size.

from fastapi import FastAPI, UploadFile

app = FastAPI()
BUCKET = "user-media"

@app.post("/upload")
async def upload(file: UploadFile):
    client.upload_fileobj(
        Fileobj=file.file,          # streams in chunks, no full read
        Bucket=BUCKET,
        Key=f"uploads/{file.filename}",
        ExtraArgs={"ContentType": file.content_type},
    )
    return {"key": f"uploads/{file.filename}"}

Don't Block the Event Loop

boto3 is synchronous. Calling upload_fileobj directly inside an async def endpoint blocks the event loop while bytes travel to S3, stalling every other request on that worker.

Offload the blocking call to a thread pool with run_in_threadpool (Starlette) or asyncio.to_thread. Now the event loop stays free to serve other connections.

from fastapi import FastAPI, UploadFile
from fastapi.concurrency import run_in_threadpool

app = FastAPI()
BUCKET = "user-media"

@app.post("/upload")
async def upload(file: UploadFile):
    key = f"uploads/{file.filename}"
    await run_in_threadpool(
        client.upload_fileobj, file.file, BUCKET, key,
        {"ContentType": file.content_type},
    )
    return {"key": key}

Presigned URLs: Let Clients Talk to S3 Directly

Streaming through the API still spends your bandwidth and CPU twice (client→API, API→S3). The most scalable pattern removes the API from the data path entirely using a presigned URL.

A presigned URL is a temporary, signed link that grants permission for one specific operation (PUT or GET) on one object, expiring after N seconds. The client uploads directly to S3; your API only signs the request.

  • API stays stateless and tiny — it never touches the bytes.
  • Credentials never leave the server; the signature encodes the grant.

Generating a Presigned PUT URL

Use generate_presigned_url with the put_object client method to mint an upload link. The endpoint returns the URL plus the final object key; the browser then issues a plain HTTP PUT to that URL with the file body.

Set a short ExpiresIn (e.g. 300–900 seconds) — just long enough to start the upload.

import uuid
from fastapi import FastAPI

app = FastAPI()
BUCKET = "user-media"

@app.post("/uploads/presign")
def presign_put(filename: str, content_type: str):
    key = f"uploads/{uuid.uuid4()}-{filename}"
    url = client.generate_presigned_url(
        ClientMethod="put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
        ExpiresIn=600,
    )
    return {"upload_url": url, "key": key}

The Client-Side Upload Flow

With a presigned PUT URL, the browser uploads with a single request — no multipart form, just the raw body. The flow is:

  • 1. Client asks your API for a presigned URL (sends filename + content type).
  • 2. API returns upload_url and the final key.
  • 3. Client does PUT upload_url with the file bytes and the matching Content-Type header.
  • 4. Client notifies your API of the key so you can persist it in the DB.

The Content-Type on the PUT must match the one you signed, or S3 returns 403.

// Browser-side (illustrative)
const { upload_url, key } = await api.presign(file.name, file.type);
await fetch(upload_url, {
  method: "PUT",
  headers: { "Content-Type": file.type },
  body: file,
});
await api.confirm(key);

Serving Private Files with Presigned GET URLs

Make buckets private by default. To let a user download or view a file, generate a short-lived presigned get_object URL on demand instead of making the object public.

This keeps access controlled by your API's auth: check the user owns the file, then sign a URL valid for a few minutes. Embed it in an <img> src or return it as a redirect.

from fastapi import FastAPI
from fastapi.responses import RedirectResponse

app = FastAPI()
BUCKET = "user-media"

@app.get("/files/{key:path}")
def download(key: str):
    url = client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": BUCKET, "Key": key},
        ExpiresIn=300,
    )
    return RedirectResponse(url)

Constraining Uploads with Presigned POST

A presigned PUT URL cannot limit file size — a malicious client could upload a 50 GB file. When you need server-enforced limits, use generate_presigned_post instead.

It returns a URL plus form fields, and lets you attach conditions like content-length-range and an exact content type. S3 rejects the upload server-side if the bytes violate the policy.

from fastapi import FastAPI

app = FastAPI()
BUCKET = "user-media"

@app.post("/uploads/presign-post")
def presign_post(key: str, content_type: str):
    return client.generate_presigned_post(
        Bucket=BUCKET,
        Key=key,
        Fields={"Content-Type": content_type},
        Conditions=[
            {"Content-Type": content_type},
            ["content-length-range", 1, 10 * 1024 * 1024],  # max 10 MB
        ],
        ExpiresIn=600,
    )

A Reusable Key Builder

Object keys should be collision-proof, organized, and never trust the client's raw filename (which can contain ../ or odd characters). A small helper centralizes this logic and is pure Python — easy to unit test.

A good key includes a logical prefix (owner, category), a UUID for uniqueness, and a sanitized extension.

import re
import uuid

def build_key(user_id: int, filename: str) -> str:
    ext = filename.rsplit(".", 1)[-1].lower() if "." in filename else "bin"
    ext = re.sub(r"[^a-z0-9]", "", ext)[:8] or "bin"
    return f"users/{user_id}/{uuid.uuid4().hex}.{ext}"

print(build_key(42, "My Vacation.JPG"))
print(build_key(7, "../../etc/passwd"))
print(build_key(1, "noext"))

Quick Check: Choosing the Scalable Pattern

You are building an endpoint that lets users upload large videos (up to 2 GB). You want the FastAPI service to stay stateless and to avoid routing the file bytes through the API server at all. Which approach best fits?

Recap: Stateless Media at Scale

You now have a full toolkit for offloading storage to S3-compatible buckets:

  • One client, many providers — boto3 with an endpoint_url targets S3, MinIO, R2, Spaces.
  • Never buffer whole files — if bytes must pass through the API, use upload_fileobj and offload it with run_in_threadpool so the event loop stays free.
  • Prefer presigned URLs — clients PUT/GET directly against S3; the API only signs, staying stateless.
  • Enforce limits with generate_presigned_post and a content-length-range condition.
  • Keep buckets private and serve files via short-lived presigned GET links gated by your auth.
  • Sanitize keys — UUID-based, prefixed, never trusting raw filenames.

The result is an API that handles 2 GB or 2 KB uploads with the same bounded footprint.

よくある質問

「S3互換バケットへのストレージオフロード」レッスンは無料ですか？

はい。「S3互換バケットへのストレージオフロード」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、FastAPI Backend Development Bootcampコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 FastAPI Backend Development Bootcampコースには全4レッスンが含まれています。

「S3互換バケットへのストレージオフロード」で何を学びますか？

署名付きURLを使ってアップロードをS3／MinIOへ直接ストリーミングし、APIをステートレスかつスケーラブルに保ちます。 ブラウザで直接実行するハンズオンコードでFastAPI Backend Development Bootcampを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。

FastAPI Backend Development Bootcampを始めるのに経験は必要ですか？

事前経験は必要ありません。CoddyKitのFastAPI Backend Development Bootcampは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。 これはレッスン3/4です。

「S3互換バケットへのストレージオフロード」レッスンにはどのくらい時間がかかりますか？

ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。

このFastAPI Backend Development Bootcampレッスンでコードを書いて実行できますか？

はい。すべてのFastAPI Backend Development Bootcampレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。

このコースのすべてのレッスン

  1. マルチパートアップロードとコンテンツ検証
  2. ストリーミングレスポンスとRangeリクエスト
  3. S3互換バケットへのストレージオフロード
  4. 非同期の画像・ドキュメント変換
← FastAPI Backend Development Bootcampに戻る