FastAPI Backend Development Bootcamp · 강의

S3 호환 버킷으로 스토리지 오프로딩

사전 서명 URL을 사용해 업로드를 S3/MinIO로 직접 스트리밍하여 API를 무상태로 유지하고 확장성을 높입니다.

S3 호환 버킷으로 스토리지 오프로딩은(는) CoddyKit의 무료 FastAPI Backend Development Bootcamp 강의입니다. 이것은 4개 중 3번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 FastAPI Backend Development Bootcamp 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. FastAPI Backend Development Bootcamp 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

Why Offload Storage to S3?

Storing uploaded files on your API server's local disk does not scale. Every replica would need its own copy, disks fill up, and containers are ephemeral — restart the pod and the files vanish.

The fix is to push media to an external object store and keep your API stateless. Any S3-compatible service works:

Amazon S3 — the original, fully managed.
MinIO — self-hosted, S3 API-compatible, great for dev and on-prem.
Cloudflare R2, Backblaze B2, DigitalOcean Spaces — cheaper egress, same API.

Because they all speak the S3 protocol, the same boto3 client code targets any of them just by changing the endpoint URL.

Configuring a boto3 S3 Client

The boto3 library is the standard AWS SDK for Python. To target a non-AWS provider like MinIO or R2, pass an explicit endpoint_url.

Keep credentials and the endpoint in settings, never hard-coded. Use config=Config(signature_version="s3v4") so presigned URLs are generated with the modern SigV4 algorithm that all providers accept.

import boto3
from botocore.config import Config

def build_s3_client():
    return boto3.client(
        "s3",
        endpoint_url="https://s3.eu-central-1.amazonaws.com",
        aws_access_key_id="AKIA...",
        aws_secret_access_key="secret...",
        region_name="eu-central-1",
        config=Config(signature_version="s3v4"),
    )

client = build_s3_client()
print(type(client).__name__)

The Naive Approach (and Why It Hurts)

The obvious first attempt is to read the whole upload into memory, then hand it to S3:

data = await file.read() loads the entire file into RAM.
A 2 GB video upload becomes 2 GB of process memory — multiply by concurrent requests and your workers OOM-crash.

This works for tiny avatars but is dangerous for media. We want to stream bytes through the API (or skip the API entirely with presigned URLs). The next scenes build up both techniques.

from fastapi import FastAPI, UploadFile

app = FastAPI()

@app.post("/upload-naive")
async def upload_naive(file: UploadFile):
    data = await file.read()  # whole file in RAM - avoid for large media!
    return {"size": len(data)}

Streaming Uploads with upload_fileobj

FastAPI's UploadFile wraps a SpooledTemporaryFile: small uploads stay in memory, large ones spill to disk automatically. Its .file attribute is a standard file-like object.

boto3's upload_fileobj reads that stream in chunks and performs a multipart upload under the hood — so memory stays bounded regardless of file size.

from fastapi import FastAPI, UploadFile

app = FastAPI()
BUCKET = "user-media"

@app.post("/upload")
async def upload(file: UploadFile):
    client.upload_fileobj(
        Fileobj=file.file,          # streams in chunks, no full read
        Bucket=BUCKET,
        Key=f"uploads/{file.filename}",
        ExtraArgs={"ContentType": file.content_type},
    )
    return {"key": f"uploads/{file.filename}"}

Don't Block the Event Loop

boto3 is synchronous. Calling upload_fileobj directly inside an async def endpoint blocks the event loop while bytes travel to S3, stalling every other request on that worker.

Offload the blocking call to a thread pool with run_in_threadpool (Starlette) or asyncio.to_thread. Now the event loop stays free to serve other connections.

from fastapi import FastAPI, UploadFile
from fastapi.concurrency import run_in_threadpool

app = FastAPI()
BUCKET = "user-media"

@app.post("/upload")
async def upload(file: UploadFile):
    key = f"uploads/{file.filename}"
    await run_in_threadpool(
        client.upload_fileobj, file.file, BUCKET, key,
        {"ContentType": file.content_type},
    )
    return {"key": key}

Presigned URLs: Let Clients Talk to S3 Directly

Streaming through the API still spends your bandwidth and CPU twice (client→API, API→S3). The most scalable pattern removes the API from the data path entirely using a presigned URL.

A presigned URL is a temporary, signed link that grants permission for one specific operation (PUT or GET) on one object, expiring after N seconds. The client uploads directly to S3; your API only signs the request.

API stays stateless and tiny — it never touches the bytes.
Credentials never leave the server; the signature encodes the grant.

Generating a Presigned PUT URL

Use generate_presigned_url with the put_object client method to mint an upload link. The endpoint returns the URL plus the final object key; the browser then issues a plain HTTP PUT to that URL with the file body.

Set a short ExpiresIn (e.g. 300–900 seconds) — just long enough to start the upload.

import uuid
from fastapi import FastAPI

app = FastAPI()
BUCKET = "user-media"

@app.post("/uploads/presign")
def presign_put(filename: str, content_type: str):
    key = f"uploads/{uuid.uuid4()}-{filename}"
    url = client.generate_presigned_url(
        ClientMethod="put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
        ExpiresIn=600,
    )
    return {"upload_url": url, "key": key}

The Client-Side Upload Flow

With a presigned PUT URL, the browser uploads with a single request — no multipart form, just the raw body. The flow is:

1. Client asks your API for a presigned URL (sends filename + content type).
2. API returns upload_url and the final key.
3. Client does PUT upload_url with the file bytes and the matching Content-Type header.
4. Client notifies your API of the key so you can persist it in the DB.

The Content-Type on the PUT must match the one you signed, or S3 returns 403.

// Browser-side (illustrative)
const { upload_url, key } = await api.presign(file.name, file.type);
await fetch(upload_url, {
  method: "PUT",
  headers: { "Content-Type": file.type },
  body: file,
});
await api.confirm(key);

Serving Private Files with Presigned GET URLs

Make buckets private by default. To let a user download or view a file, generate a short-lived presigned get_object URL on demand instead of making the object public.

This keeps access controlled by your API's auth: check the user owns the file, then sign a URL valid for a few minutes. Embed it in an <img> src or return it as a redirect.

from fastapi import FastAPI
from fastapi.responses import RedirectResponse

app = FastAPI()
BUCKET = "user-media"

@app.get("/files/{key:path}")
def download(key: str):
    url = client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": BUCKET, "Key": key},
        ExpiresIn=300,
    )
    return RedirectResponse(url)

Constraining Uploads with Presigned POST

A presigned PUT URL cannot limit file size — a malicious client could upload a 50 GB file. When you need server-enforced limits, use generate_presigned_post instead.

It returns a URL plus form fields, and lets you attach conditions like content-length-range and an exact content type. S3 rejects the upload server-side if the bytes violate the policy.

from fastapi import FastAPI

app = FastAPI()
BUCKET = "user-media"

@app.post("/uploads/presign-post")
def presign_post(key: str, content_type: str):
    return client.generate_presigned_post(
        Bucket=BUCKET,
        Key=key,
        Fields={"Content-Type": content_type},
        Conditions=[
            {"Content-Type": content_type},
            ["content-length-range", 1, 10 * 1024 * 1024],  # max 10 MB
        ],
        ExpiresIn=600,
    )

A Reusable Key Builder

Object keys should be collision-proof, organized, and never trust the client's raw filename (which can contain ../ or odd characters). A small helper centralizes this logic and is pure Python — easy to unit test.

A good key includes a logical prefix (owner, category), a UUID for uniqueness, and a sanitized extension.

import re
import uuid

def build_key(user_id: int, filename: str) -> str:
    ext = filename.rsplit(".", 1)[-1].lower() if "." in filename else "bin"
    ext = re.sub(r"[^a-z0-9]", "", ext)[:8] or "bin"
    return f"users/{user_id}/{uuid.uuid4().hex}.{ext}"

print(build_key(42, "My Vacation.JPG"))
print(build_key(7, "../../etc/passwd"))
print(build_key(1, "noext"))

Quick Check: Choosing the Scalable Pattern

You are building an endpoint that lets users upload large videos (up to 2 GB). You want the FastAPI service to stay stateless and to avoid routing the file bytes through the API server at all. Which approach best fits?

Recap: Stateless Media at Scale

You now have a full toolkit for offloading storage to S3-compatible buckets:

One client, many providers — boto3 with an endpoint_url targets S3, MinIO, R2, Spaces.
Never buffer whole files — if bytes must pass through the API, use upload_fileobj and offload it with run_in_threadpool so the event loop stays free.
Prefer presigned URLs — clients PUT/GET directly against S3; the API only signs, staying stateless.
Enforce limits with generate_presigned_post and a content-length-range condition.
Keep buckets private and serve files via short-lived presigned GET links gated by your auth.
Sanitize keys — UUID-based, prefixed, never trusting raw filenames.

The result is an API that handles 2 GB or 2 KB uploads with the same bounded footprint.

자주 묻는 질문

“S3 호환 버킷으로 스토리지 오프로딩” 강의는 무료인가요?

네 — “S3 호환 버킷으로 스토리지 오프로딩” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 FastAPI Backend Development Bootcamp 강의 전체를 잠금 해제할 수 있습니다. FastAPI Backend Development Bootcamp 강의에는 총 4개의 강의가 포함되어 있습니다.

“S3 호환 버킷으로 스토리지 오프로딩”에서 뭘 배우나요?

사전 서명 URL을 사용해 업로드를 S3/MinIO로 직접 스트리밍하여 API를 무상태로 유지하고 확장성을 높입니다. 브라우저에서 직접 실행하는 실습 코드로 FastAPI Backend Development Bootcamp을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

FastAPI Backend Development Bootcamp을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 FastAPI Backend Development Bootcamp은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 3번째 강의입니다.

“S3 호환 버킷으로 스토리지 오프로딩” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 FastAPI Backend Development Bootcamp 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 FastAPI Backend Development Bootcamp 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← FastAPI Backend Development Bootcamp(으)로 돌아가기