نقل التخزين إلى حاويات متوافقة مع S3

Storing uploaded files on your API server's local disk does not scale. Every replica would need its own copy, disks fill up, and containers are ephemeral — restart the pod and the files vanish.

The fix is to push media to an external object store and keep your API stateless. Any S3-compatible service works:

• Amazon S3 — the original, fully managed.
• MinIO — self-hosted, S3 API-compatible, great for dev and on-prem.
• Cloudflare R2, Backblaze B2, DigitalOcean Spaces — cheaper egress, same API.

Because they all speak the S3 protocol, the same boto3 client code targets any of them just by changing the endpoint URL.

The boto3 library is the standard AWS SDK for Python. To target a non-AWS provider like MinIO or R2, pass an explicit endpoint_url.

Keep credentials and the endpoint in settings, never hard-coded. Use config=Config(signature_version="s3v4") so presigned URLs are generated with the modern SigV4 algorithm that all providers accept.

The obvious first attempt is to read the whole upload into memory, then hand it to S3:

• data = await file.read() loads the entire file into RAM.
• A 2 GB video upload becomes 2 GB of process memory — multiply by concurrent requests and your workers OOM-crash.

This works for tiny avatars but is dangerous for media. We want to stream bytes through the API (or skip the API entirely with presigned URLs). The next scenes build up both techniques.

FastAPI's UploadFile wraps a SpooledTemporaryFile: small uploads stay in memory, large ones spill to disk automatically. Its .file attribute is a standard file-like object.

boto3's upload_fileobj reads that stream in chunks and performs a multipart upload under the hood — so memory stays bounded regardless of file size.

boto3 is synchronous. Calling upload_fileobj directly inside an async def endpoint blocks the event loop while bytes travel to S3, stalling every other request on that worker.

Offload the blocking call to a thread pool with run_in_threadpool (Starlette) or asyncio.to_thread. Now the event loop stays free to serve other connections.

Streaming through the API still spends your bandwidth and CPU twice (client→API, API→S3). The most scalable pattern removes the API from the data path entirely using a presigned URL.

A presigned URL is a temporary, signed link that grants permission for one specific operation (PUT or GET) on one object, expiring after N seconds. The client uploads directly to S3; your API only signs the request.

• API stays stateless and tiny — it never touches the bytes.
• Credentials never leave the server; the signature encodes the grant.

Use generate_presigned_url with the put_object client method to mint an upload link. The endpoint returns the URL plus the final object key; the browser then issues a plain HTTP PUT to that URL with the file body.

Set a short ExpiresIn (e.g. 300–900 seconds) — just long enough to start the upload.

With a presigned PUT URL, the browser uploads with a single request — no multipart form, just the raw body. The flow is:

• 1. Client asks your API for a presigned URL (sends filename + content type).
• 2. API returns upload_url and the final key.
• 3. Client does PUT upload_url with the file bytes and the matching Content-Type header.
• 4. Client notifies your API of the key so you can persist it in the DB.

The Content-Type on the PUT must match the one you signed, or S3 returns 403.

Make buckets private by default. To let a user download or view a file, generate a short-lived presigned get_object URL on demand instead of making the object public.

This keeps access controlled by your API's auth: check the user owns the file, then sign a URL valid for a few minutes. Embed it in an <img> src or return it as a redirect.

A presigned PUT URL cannot limit file size — a malicious client could upload a 50 GB file. When you need server-enforced limits, use generate_presigned_post instead.

It returns a URL plus form fields, and lets you attach conditions like content-length-range and an exact content type. S3 rejects the upload server-side if the bytes violate the policy.

Object keys should be collision-proof, organized, and never trust the client's raw filename (which can contain ../ or odd characters). A small helper centralizes this logic and is pure Python — easy to unit test.

A good key includes a logical prefix (owner, category), a UUID for uniqueness, and a sanitized extension.

You are building an endpoint that lets users upload large videos (up to 2 GB). You want the FastAPI service to stay stateless and to avoid routing the file bytes through the API server at all. Which approach best fits?

You now have a full toolkit for offloading storage to S3-compatible buckets:

• One client, many providers — boto3 with an endpoint_url targets S3, MinIO, R2, Spaces.
• Never buffer whole files — if bytes must pass through the API, use upload_fileobj and offload it with run_in_threadpool so the event loop stays free.
• Prefer presigned URLs — clients PUT/GET directly against S3; the API only signs, staying stateless.
• Enforce limits with generate_presigned_post and a content-length-range condition.
• Keep buckets private and serve files via short-lived presigned GET links gated by your auth.
• Sanitize keys — UUID-based, prefixed, never trusting raw filenames.

The result is an API that handles 2 GB or 2 KB uploads with the same bounded footprint.