FastAPI Backend Development Bootcamp · 课时

将存储卸载到兼容 S3 的存储桶

使用预签名 URL 将上传内容直接传输到 S3/MinIO，保持 API 无状态且可扩展。

第 3 / 4 课13 个步骤

将存储卸载到兼容 S3 的存储桶 是 CoddyKit 上的免费 FastAPI Backend Development Bootcamp 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 FastAPI Backend Development Bootcamp 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 FastAPI Backend Development Bootcamp 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Offload Storage to S3?

Storing uploaded files on your API server's local disk does not scale. Every replica would need its own copy, disks fill up, and containers are ephemeral — restart the pod and the files vanish.

The fix is to push media to an external object store and keep your API stateless. Any S3-compatible service works:

Amazon S3 — the original, fully managed.
MinIO — self-hosted, S3 API-compatible, great for dev and on-prem.
Cloudflare R2, Backblaze B2, DigitalOcean Spaces — cheaper egress, same API.

Because they all speak the S3 protocol, the same boto3 client code targets any of them just by changing the endpoint URL.

Configuring a boto3 S3 Client

The boto3 library is the standard AWS SDK for Python. To target a non-AWS provider like MinIO or R2, pass an explicit endpoint_url.

Keep credentials and the endpoint in settings, never hard-coded. Use config=Config(signature_version="s3v4") so presigned URLs are generated with the modern SigV4 algorithm that all providers accept.

import boto3
from botocore.config import Config

def build_s3_client():
    return boto3.client(
        "s3",
        endpoint_url="https://s3.eu-central-1.amazonaws.com",
        aws_access_key_id="AKIA...",
        aws_secret_access_key="secret...",
        region_name="eu-central-1",
        config=Config(signature_version="s3v4"),
    )

client = build_s3_client()
print(type(client).__name__)

The Naive Approach (and Why It Hurts)

The obvious first attempt is to read the whole upload into memory, then hand it to S3:

data = await file.read() loads the entire file into RAM.
A 2 GB video upload becomes 2 GB of process memory — multiply by concurrent requests and your workers OOM-crash.

This works for tiny avatars but is dangerous for media. We want to stream bytes through the API (or skip the API entirely with presigned URLs). The next scenes build up both techniques.

from fastapi import FastAPI, UploadFile

app = FastAPI()

@app.post("/upload-naive")
async def upload_naive(file: UploadFile):
    data = await file.read()  # whole file in RAM - avoid for large media!
    return {"size": len(data)}

Streaming Uploads with upload_fileobj

FastAPI's UploadFile wraps a SpooledTemporaryFile: small uploads stay in memory, large ones spill to disk automatically. Its .file attribute is a standard file-like object.

boto3's upload_fileobj reads that stream in chunks and performs a multipart upload under the hood — so memory stays bounded regardless of file size.

from fastapi import FastAPI, UploadFile

app = FastAPI()
BUCKET = "user-media"

@app.post("/upload")
async def upload(file: UploadFile):
    client.upload_fileobj(
        Fileobj=file.file,          # streams in chunks, no full read
        Bucket=BUCKET,
        Key=f"uploads/{file.filename}",
        ExtraArgs={"ContentType": file.content_type},
    )
    return {"key": f"uploads/{file.filename}"}

Don't Block the Event Loop

boto3 is synchronous. Calling upload_fileobj directly inside an async def endpoint blocks the event loop while bytes travel to S3, stalling every other request on that worker.

Offload the blocking call to a thread pool with run_in_threadpool (Starlette) or asyncio.to_thread. Now the event loop stays free to serve other connections.

from fastapi import FastAPI, UploadFile
from fastapi.concurrency import run_in_threadpool

app = FastAPI()
BUCKET = "user-media"

@app.post("/upload")
async def upload(file: UploadFile):
    key = f"uploads/{file.filename}"
    await run_in_threadpool(
        client.upload_fileobj, file.file, BUCKET, key,
        {"ContentType": file.content_type},
    )
    return {"key": key}

Presigned URLs: Let Clients Talk to S3 Directly

Streaming through the API still spends your bandwidth and CPU twice (client→API, API→S3). The most scalable pattern removes the API from the data path entirely using a presigned URL.

A presigned URL is a temporary, signed link that grants permission for one specific operation (PUT or GET) on one object, expiring after N seconds. The client uploads directly to S3; your API only signs the request.

API stays stateless and tiny — it never touches the bytes.
Credentials never leave the server; the signature encodes the grant.

Generating a Presigned PUT URL

Use generate_presigned_url with the put_object client method to mint an upload link. The endpoint returns the URL plus the final object key; the browser then issues a plain HTTP PUT to that URL with the file body.

Set a short ExpiresIn (e.g. 300–900 seconds) — just long enough to start the upload.

import uuid
from fastapi import FastAPI

app = FastAPI()
BUCKET = "user-media"

@app.post("/uploads/presign")
def presign_put(filename: str, content_type: str):
    key = f"uploads/{uuid.uuid4()}-{filename}"
    url = client.generate_presigned_url(
        ClientMethod="put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
        ExpiresIn=600,
    )
    return {"upload_url": url, "key": key}

The Client-Side Upload Flow

With a presigned PUT URL, the browser uploads with a single request — no multipart form, just the raw body. The flow is:

1. Client asks your API for a presigned URL (sends filename + content type).
2. API returns upload_url and the final key.
3. Client does PUT upload_url with the file bytes and the matching Content-Type header.
4. Client notifies your API of the key so you can persist it in the DB.

The Content-Type on the PUT must match the one you signed, or S3 returns 403.

// Browser-side (illustrative)
const { upload_url, key } = await api.presign(file.name, file.type);
await fetch(upload_url, {
  method: "PUT",
  headers: { "Content-Type": file.type },
  body: file,
});
await api.confirm(key);

Serving Private Files with Presigned GET URLs

Make buckets private by default. To let a user download or view a file, generate a short-lived presigned get_object URL on demand instead of making the object public.

This keeps access controlled by your API's auth: check the user owns the file, then sign a URL valid for a few minutes. Embed it in an <img> src or return it as a redirect.

from fastapi import FastAPI
from fastapi.responses import RedirectResponse

app = FastAPI()
BUCKET = "user-media"

@app.get("/files/{key:path}")
def download(key: str):
    url = client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": BUCKET, "Key": key},
        ExpiresIn=300,
    )
    return RedirectResponse(url)

Constraining Uploads with Presigned POST

A presigned PUT URL cannot limit file size — a malicious client could upload a 50 GB file. When you need server-enforced limits, use generate_presigned_post instead.

It returns a URL plus form fields, and lets you attach conditions like content-length-range and an exact content type. S3 rejects the upload server-side if the bytes violate the policy.

from fastapi import FastAPI

app = FastAPI()
BUCKET = "user-media"

@app.post("/uploads/presign-post")
def presign_post(key: str, content_type: str):
    return client.generate_presigned_post(
        Bucket=BUCKET,
        Key=key,
        Fields={"Content-Type": content_type},
        Conditions=[
            {"Content-Type": content_type},
            ["content-length-range", 1, 10 * 1024 * 1024],  # max 10 MB
        ],
        ExpiresIn=600,
    )

A Reusable Key Builder

Object keys should be collision-proof, organized, and never trust the client's raw filename (which can contain ../ or odd characters). A small helper centralizes this logic and is pure Python — easy to unit test.

A good key includes a logical prefix (owner, category), a UUID for uniqueness, and a sanitized extension.

import re
import uuid

def build_key(user_id: int, filename: str) -> str:
    ext = filename.rsplit(".", 1)[-1].lower() if "." in filename else "bin"
    ext = re.sub(r"[^a-z0-9]", "", ext)[:8] or "bin"
    return f"users/{user_id}/{uuid.uuid4().hex}.{ext}"

print(build_key(42, "My Vacation.JPG"))
print(build_key(7, "../../etc/passwd"))
print(build_key(1, "noext"))

Quick Check: Choosing the Scalable Pattern

You are building an endpoint that lets users upload large videos (up to 2 GB). You want the FastAPI service to stay stateless and to avoid routing the file bytes through the API server at all. Which approach best fits?

在交互式课程中回答

Recap: Stateless Media at Scale

You now have a full toolkit for offloading storage to S3-compatible buckets:

One client, many providers — boto3 with an endpoint_url targets S3, MinIO, R2, Spaces.
Never buffer whole files — if bytes must pass through the API, use upload_fileobj and offload it with run_in_threadpool so the event loop stays free.
Prefer presigned URLs — clients PUT/GET directly against S3; the API only signs, staying stateless.
Enforce limits with generate_presigned_post and a content-length-range condition.
Keep buckets private and serve files via short-lived presigned GET links gated by your auth.
Sanitize keys — UUID-based, prefixed, never trusting raw filenames.

The result is an API that handles 2 GB or 2 KB uploads with the same bounded footprint.

免费开始

用 AI 导师学习 FastAPI Backend Development Bootcamp — 免费

在浏览器中编写并运行真实代码，获得全天候 AI 导师的即时帮助，并在网页或应用中继续学习。

课程: 21
课程: 84

免费开始学习查看完整课程

常见问题解答

「将存储卸载到兼容 S3 的存储桶」课时是免费的吗？

是的 — 「将存储卸载到兼容 S3 的存储桶」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 FastAPI Backend Development Bootcamp 课程的其余内容，请升级到 CoddyKit PRO。 FastAPI Backend Development Bootcamp 课程共包含 4 节课。

「将存储卸载到兼容 S3 的存储桶」这节课中我会学到什么？

使用预签名 URL 将上传内容直接传输到 S3/MinIO，保持 API 无状态且可扩展。你通过在浏览器中直接运行的动手代码来练习 FastAPI Backend Development Bootcamp，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 FastAPI Backend Development Bootcamp 需要有经验吗？

无需任何先前经验。CoddyKit 上的 FastAPI Backend Development Bootcamp 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「将存储卸载到兼容 S3 的存储桶」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 FastAPI Backend Development Bootcamp 课中编写并运行代码吗？

能。每节 FastAPI Backend Development Bootcamp 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 FastAPI Backend Development Bootcamp