FastAPI Backend Development Bootcamp · Lesson

Retries, Idempotency and Dead-Letter Handling

Make tasks resilient with exponential backoff, idempotency keys, and dead-letter routing for poisoned messages.

Retries, Idempotency and Dead-Letter Handling is a free FastAPI Backend Development Bootcamp lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the FastAPI Backend Development Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Tasks Need Resilience

In a FastAPI backend, you push slow work (sending emails, charging cards, calling third-party APIs) to a Celery worker so the HTTP request returns fast. But background work runs in a hostile world: networks blip, APIs rate-limit you, and workers crash mid-task.

A resilient task must survive three failure modes:

Transient failures — retry with exponential backoff so you do not hammer a struggling service.
Duplicate delivery — the same message may be processed twice, so tasks must be idempotent.
Poisoned messages — a task that fails forever must be routed to a dead-letter queue instead of looping endlessly.

This lesson wires all three together.

At-Least-Once Delivery

Celery brokers (RabbitMQ, Redis) give you at-least-once delivery, not exactly-once. A message is acknowledged (ack) only after the task finishes. If a worker dies after doing work but before acking, the broker redelivers the message and the task runs again.

With acks_late=True the ack happens after execution — safer against crashes, but it guarantees that some tasks will run twice. That is the core reason idempotency is not optional.

from celery import Celery

app = Celery("jobs", broker="redis://localhost:6379/0")

# Recommended resilience defaults
app.conf.update(
    task_acks_late=True,          # ack only after the task body returns
    task_reject_on_worker_lost=True,  # requeue if the worker is killed
    worker_prefetch_multiplier=1, # don't hoard messages on one worker
)

Automatic Retries with autoretry_for

The simplest way to retry is to declare which exceptions are retryable. Celery catches them and re-schedules the task automatically.

autoretry_for — the exception classes that trigger a retry.
max_retries — the cap before the task is marked failed.
retry_backoff — turns on exponential backoff (delays double each attempt).

Only retry on transient errors (timeouts, 5xx, connection resets). Never blindly retry a ValueError from bad input — it will fail identically every time.

import requests
from celery import Celery

app = Celery("jobs", broker="redis://localhost:6379/0")

@app.task(
    autoretry_for=(requests.exceptions.RequestException,),
    max_retries=5,
    retry_backoff=True,        # 1s, 2s, 4s, 8s, ...
    retry_backoff_max=600,     # cap the delay at 10 minutes
    retry_jitter=True,         # randomize to avoid thundering herd
)
def call_payment_api(charge_id: str):
    resp = requests.post("https://api.example.com/charge", json={"id": charge_id}, timeout=10)
    resp.raise_for_status()
    return resp.json()

Exponential Backoff, Explained

Exponential backoff means the wait between attempts grows geometrically: the delay for attempt n is roughly base * 2 ** n, capped at a maximum. This gives a struggling downstream service time to recover instead of being retried into the ground.

Jitter adds randomness so that thousands of tasks that failed at the same instant do not all retry at the same instant (the “thundering herd”). Here is the math Celery applies, as a plain standalone program.

import random

def backoff_delay(attempt, base=1, cap=600, jitter=True):
    delay = min(cap, base * (2 ** attempt))
    if jitter:
        delay = random.uniform(0, delay)  # full jitter
    return delay

for attempt in range(8):
    raw = min(600, 1 * (2 ** attempt))
    print(f"attempt {attempt}: raw={raw:>3}s  jittered~={backoff_delay(attempt):6.1f}s")

Manual Retry with self.retry

When you need custom logic — deciding the delay from a response header, or only retrying on certain status codes — bind the task and call self.retry() explicitly.

Use bind=True to get self, read self.request.retries to know the attempt number, and pass countdown for the delay. Raising the result of self.retry() stops the current execution cleanly.

import requests
from celery import Celery

app = Celery("jobs", broker="redis://localhost:6379/0")

@app.task(bind=True, max_retries=5)
def sync_inventory(self, sku: str):
    resp = requests.get(f"https://api.example.com/stock/{sku}", timeout=5)
    if resp.status_code == 429:  # rate limited
        wait = int(resp.headers.get("Retry-After", 2 ** self.request.retries))
        raise self.retry(countdown=wait)
    resp.raise_for_status()
    return resp.json()["qty"]

What Idempotency Really Means

An operation is idempotent if running it twice has the same effect as running it once. Because Celery delivers at-least-once, every task that mutates state (charging a card, creating a record, decrementing stock) must be idempotent or you will double-charge customers.

The standard tool is an idempotency key: a unique identifier for the business intent, not for the message. You record “I have already processed key X” in durable storage and short-circuit on the next delivery.

Pass the key in from the API request (clients can supply it too).
Store it in a table or Redis with a UNIQUE constraint.
The constraint — not application logic — is what makes it race-safe.

An Idempotency Guard

Here is the pattern in isolation: a guard that remembers completed keys and refuses to run the effect twice, even under concurrent calls. In real code the seen set becomes a Redis SET NX or a DB row with a unique key, but the logic is identical.

import threading

class IdempotencyGuard:
    def __init__(self):
        self._seen = set()
        self._lock = threading.Lock()

    def run_once(self, key, effect):
        with self._lock:            # the unique-constraint stand-in
            if key in self._seen:
                return "skipped (duplicate)"
            self._seen.add(key)
        return effect()

guard = IdempotencyGuard()
charges = []

def charge():
    charges.append(99)
    return "charged 99"

print(guard.run_once("order-123", charge))
print(guard.run_once("order-123", charge))  # duplicate delivery
print("total charges applied:", len(charges))

Idempotency Inside a Celery Task

In practice you wrap the effect in a database transaction and let a UNIQUE constraint be the source of truth. Insert the idempotency key first; if the insert raises a unique-violation, a previous (or concurrent) delivery already handled it, so you return early.

This keeps the check and the side effect atomic — there is no window where one delivery sees “not done” while another is mid-charge.

from sqlalchemy.exc import IntegrityError
from celery import Celery

app = Celery("jobs", broker="redis://localhost:6379/0")

@app.task(bind=True, acks_late=True, max_retries=3, retry_backoff=True)
def charge_order(self, order_id: str, idem_key: str):
    with db_session() as s:
        try:
            s.add(ProcessedKey(key=idem_key))  # UNIQUE column
            s.flush()                            # raises on duplicate
        except IntegrityError:
            s.rollback()
            return {"status": "already_processed", "order_id": order_id}
        amount = payment_gateway.charge(order_id, idempotency_key=idem_key)
        s.commit()
        return {"status": "charged", "amount": amount}

Poisoned Messages and the Dead-Letter Queue

Some messages can never succeed: malformed payloads, references to deleted rows, a bug that always throws. Retrying them forever wastes workers and floods your logs. These are poisoned messages.

A dead-letter queue (DLQ) is a separate queue where exhausted or rejected messages are parked for inspection, alerting, or manual replay. With RabbitMQ you declare a queue with a x-dead-letter-exchange argument; messages that are rejected (nack with requeue=False) or that exceed a TTL get routed there automatically by the broker.

from kombu import Exchange, Queue

dead_exchange = Exchange("dlx", type="direct")

task_queues = (
    Queue(
        "payments",
        Exchange("payments"),
        routing_key="payments",
        queue_arguments={
            "x-dead-letter-exchange": "dlx",
            "x-dead-letter-routing-key": "payments.dead",
        },
    ),
    Queue("payments_dead", dead_exchange, routing_key="payments.dead"),
)

Routing Exhausted Tasks to the DLQ

The broker dead-letters on reject, but Celery's retry machinery does not auto-reject when max_retries is hit — it just marks the task FAILED. To send exhausted tasks to a DLQ you catch MaxRetriesExceededError (or detect the final attempt) and explicitly forward the payload to your dead-letter task or queue.

The dead-letter handler should never reprocess — it records the failure, emits an alert, and stores the payload so an operator can replay it after fixing the root cause.

from celery import Celery
from celery.exceptions import MaxRetriesExceededError

app = Celery("jobs", broker="redis://localhost:6379/0")

@app.task(bind=True, max_retries=5, retry_backoff=True)
def process_event(self, payload: dict):
    try:
        do_work(payload)
    except TransientError as exc:
        try:
            raise self.retry(exc=exc)
        except MaxRetriesExceededError:
            dead_letter.delay(payload, reason=str(exc))  # park it
    except PermanentError as exc:
        dead_letter.delay(payload, reason=str(exc))      # never retry

@app.task
def dead_letter(payload: dict, reason: str):
    store_failed_message(payload, reason)
    alert_oncall(reason)

Putting It All Together

A production-grade resilient task combines every piece:

acks_late so crashes redeliver instead of losing work.
autoretry_for on transient errors only, with exponential backoff + jitter.
An idempotency key guarded by a unique constraint so redelivery is harmless.
A dead-letter path for permanent errors and exhausted retries.

The mental model: retry the transient, deduplicate the duplicate, dead-letter the doomed. Each mechanism covers a different failure mode — together they let a worker fail safely instead of silently corrupting data.

@app.task(
    bind=True, acks_late=True,
    autoretry_for=(TransientError,),
    max_retries=5, retry_backoff=True, retry_jitter=True,
)
def handle_webhook(self, payload: dict, idem_key: str):
    if already_processed(idem_key):       # unique-constraint check
        return "duplicate-ignored"
    try:
        result = apply_effect(payload, idem_key)
    except PermanentError as exc:
        dead_letter.delay(payload, reason=str(exc))
        return "dead-lettered"
    mark_processed(idem_key)
    return result

Quick Check

Your Celery task charges a credit card and runs with acks_late=True. Because the broker delivers at-least-once, the same message is occasionally processed twice. What is the correct primary defense against double-charging?

Recap

You learned how to make Celery tasks survive real-world failure:

Celery delivers at-least-once; acks_late=True protects against worker crashes but guarantees occasional duplicate runs.
Exponential backoff with jitter (via retry_backoff / autoretry_for or manual self.retry) handles transient failures without overwhelming downstream services — retry only transient errors.
Idempotency keys backed by a unique constraint make duplicate deliveries harmless and keep the check atomic with the side effect.
Dead-letter queues park poisoned messages and exhausted retries for alerting and manual replay, instead of looping forever.

Remember the rule: retry the transient, deduplicate the duplicate, dead-letter the doomed.

Frequently asked questions

Is the “Retries, Idempotency and Dead-Letter Handling” lesson free?

Yes — the full text of “Retries, Idempotency and Dead-Letter Handling” is free to read here on the web, and the FastAPI Backend Development Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the FastAPI Backend Development Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Retries, Idempotency and Dead-Letter Handling”?

Make tasks resilient with exponential backoff, idempotency keys, and dead-letter routing for poisoned messages. You practise FastAPI Backend Development Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start FastAPI Backend Development Bootcamp?

No prior experience is required. FastAPI Backend Development Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Retries, Idempotency and Dead-Letter Handling” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this FastAPI Backend Development Bootcamp lesson?

Yes. Every FastAPI Backend Development Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Lightweight Offloading with BackgroundTasks
Wiring Celery Workers to a FastAPI App
Retries, Idempotency and Dead-Letter Handling
Scheduled and Periodic Jobs with Celery Beat

← Back to FastAPI Backend Development Bootcamp