Retries, Idempotency and Dead-Letter Handling
Make tasks resilient with exponential backoff, idempotency keys, and dead-letter routing for poisoned messages.
Why Tasks Need Resilience
In a FastAPI backend, you push slow work (sending emails, charging cards, calling third-party APIs) to a Celery worker so the HTTP request returns fast. But background work runs in a hostile world: networks blip, APIs rate-limit you, and workers crash mid-task.
A resilient task must survive three failure modes:
- Transient failures — retry with exponential backoff so you do not hammer a struggling service.
- Duplicate delivery — the same message may be processed twice, so tasks must be idempotent.
- Poisoned messages — a task that fails forever must be routed to a dead-letter queue instead of looping endlessly.
This lesson wires all three together.
At-Least-Once Delivery
Celery brokers (RabbitMQ, Redis) give you at-least-once delivery, not exactly-once. A message is acknowledged (ack) only after the task finishes. If a worker dies after doing work but before acking, the broker redelivers the message and the task runs again.
With acks_late=True the ack happens after execution — safer against crashes, but it guarantees that some tasks will run twice. That is the core reason idempotency is not optional.
from celery import Celery
app = Celery("jobs", broker="redis://localhost:6379/0")
# Recommended resilience defaults
app.conf.update(
task_acks_late=True, # ack only after the task body returns
task_reject_on_worker_lost=True, # requeue if the worker is killed
worker_prefetch_multiplier=1, # don't hoard messages on one worker
)All lessons in this course
- Lightweight Offloading with BackgroundTasks
- Wiring Celery Workers to a FastAPI App
- Retries, Idempotency and Dead-Letter Handling
- Scheduled and Periodic Jobs with Celery Beat