Retries, Idempotenz und Dead-Letter-Verarbeitung

In a FastAPI backend, you push slow work (sending emails, charging cards, calling third-party APIs) to a Celery worker so the HTTP request returns fast. But background work runs in a hostile world: networks blip, APIs rate-limit you, and workers crash mid-task.

A resilient task must survive three failure modes:

• Transient failures — retry with exponential backoff so you do not hammer a struggling service.
• Duplicate delivery — the same message may be processed twice, so tasks must be idempotent.
• Poisoned messages — a task that fails forever must be routed to a dead-letter queue instead of looping endlessly.

This lesson wires all three together.

Celery brokers (RabbitMQ, Redis) give you at-least-once delivery, not exactly-once. A message is acknowledged (ack) only after the task finishes. If a worker dies after doing work but before acking, the broker redelivers the message and the task runs again.

With acks_late=True the ack happens after execution — safer against crashes, but it guarantees that some tasks will run twice. That is the core reason idempotency is not optional.

The simplest way to retry is to declare which exceptions are retryable. Celery catches them and re-schedules the task automatically.

• autoretry_for — the exception classes that trigger a retry.
• max_retries — the cap before the task is marked failed.
• retry_backoff — turns on exponential backoff (delays double each attempt).

Only retry on transient errors (timeouts, 5xx, connection resets). Never blindly retry a ValueError from bad input — it will fail identically every time.

Exponential backoff means the wait between attempts grows geometrically: the delay for attempt n is roughly base * 2 ** n, capped at a maximum. This gives a struggling downstream service time to recover instead of being retried into the ground.

Jitter adds randomness so that thousands of tasks that failed at the same instant do not all retry at the same instant (the “thundering herd”). Here is the math Celery applies, as a plain standalone program.

When you need custom logic — deciding the delay from a response header, or only retrying on certain status codes — bind the task and call self.retry() explicitly.

Use bind=True to get self, read self.request.retries to know the attempt number, and pass countdown for the delay. Raising the result of self.retry() stops the current execution cleanly.

An operation is idempotent if running it twice has the same effect as running it once. Because Celery delivers at-least-once, every task that mutates state (charging a card, creating a record, decrementing stock) must be idempotent or you will double-charge customers.

The standard tool is an idempotency key: a unique identifier for the business intent, not for the message. You record “I have already processed key X” in durable storage and short-circuit on the next delivery.

• Pass the key in from the API request (clients can supply it too).
• Store it in a table or Redis with a UNIQUE constraint.
• The constraint — not application logic — is what makes it race-safe.

Here is the pattern in isolation: a guard that remembers completed keys and refuses to run the effect twice, even under concurrent calls. In real code the seen set becomes a Redis SET NX or a DB row with a unique key, but the logic is identical.

In practice you wrap the effect in a database transaction and let a UNIQUE constraint be the source of truth. Insert the idempotency key first; if the insert raises a unique-violation, a previous (or concurrent) delivery already handled it, so you return early.

This keeps the check and the side effect atomic — there is no window where one delivery sees “not done” while another is mid-charge.

Some messages can never succeed: malformed payloads, references to deleted rows, a bug that always throws. Retrying them forever wastes workers and floods your logs. These are poisoned messages.

A dead-letter queue (DLQ) is a separate queue where exhausted or rejected messages are parked for inspection, alerting, or manual replay. With RabbitMQ you declare a queue with a x-dead-letter-exchange argument; messages that are rejected (nack with requeue=False) or that exceed a TTL get routed there automatically by the broker.

The broker dead-letters on reject, but Celery's retry machinery does not auto-reject when max_retries is hit — it just marks the task FAILED. To send exhausted tasks to a DLQ you catch MaxRetriesExceededError (or detect the final attempt) and explicitly forward the payload to your dead-letter task or queue.

The dead-letter handler should never reprocess — it records the failure, emits an alert, and stores the payload so an operator can replay it after fixing the root cause.

A production-grade resilient task combines every piece:

• acks_late so crashes redeliver instead of losing work.
• autoretry_for on transient errors only, with exponential backoff + jitter.
• An idempotency key guarded by a unique constraint so redelivery is harmless.
• A dead-letter path for permanent errors and exhausted retries.

The mental model: retry the transient, deduplicate the duplicate, dead-letter the doomed. Each mechanism covers a different failure mode — together they let a worker fail safely instead of silently corrupting data.

Your Celery task charges a credit card and runs with acks_late=True. Because the broker delivers at-least-once, the same message is occasionally processed twice. What is the correct primary defense against double-charging?

You learned how to make Celery tasks survive real-world failure:

• Celery delivers at-least-once; acks_late=True protects against worker crashes but guarantees occasional duplicate runs.
• Exponential backoff with jitter (via retry_backoff / autoretry_for or manual self.retry) handles transient failures without overwhelming downstream services — retry only transient errors.
• Idempotency keys backed by a unique constraint make duplicate deliveries harmless and keep the check atomic with the side effect.
• Dead-letter queues park poisoned messages and exhausted retries for alerting and manual replay, instead of looping forever.

Remember the rule: retry the transient, deduplicate the duplicate, dead-letter the doomed.