FastAPI Backend Development Bootcamp · Lección

Consumidores idempotentes y semántica exactly-once

Deduplicate y cree checkpoints del procesamiento de mensajes para lograr una entrega downstream efectiva exactly-once.

Lección 4 de 413 pasos

Consumidores idempotentes y semántica exactly-once es una lección gratuita de FastAPI Backend Development Bootcamp en CoddyKit. Esta es la lección 4 de 4. Puedes leer la lección completa abajo gratuitamente — luego la practicas en el navegador con un editor de código integrado y un tutor de IA 24/7. Forma parte de la ruta de aprendizaje de FastAPI Backend Development Bootcamp, y tu progreso se sincroniza en la web y la app de CoddyKit. El curso de FastAPI Backend Development Bootcamp incluye 4 lecciones en total.

Partes de esta lección aún no han sido traducidas y se muestran en inglés.

The Delivery Problem

Message brokers like Kafka and Pulsar default to at-least-once delivery. If a consumer crashes after processing a message but before acknowledging it, the broker redelivers that message on restart.

For a FastAPI backend draining an order-events topic, that means the same OrderPlaced event can hit your handler twice. Without protection, you charge the customer twice or send two confirmation emails.

At-most-once: ack before processing — fast, but you lose messages on crash.
At-least-once: ack after processing — no loss, but duplicates.
Exactly-once (effective): at-least-once delivery plus idempotent handling.

Idempotency Is the Real Goal

True exactly-once delivery across a network is impossible in the general case. What you can build is effective exactly-once: the broker may deliver a message many times, but your consumer applies its effect exactly once.

A handler is idempotent when running it twice with the same input yields the same final state as running it once. The strategy: give every message a stable unique key, record keys you have already processed, and skip anything you have seen before.

Choosing a Deduplication Key

The dedup key must be stable across redeliveries. The broker offset is NOT stable — a rebalanced partition can change offsets, and Pulsar message IDs differ from Kafka offsets.

Prefer a business-level identifier carried in the message itself, or a producer-assigned event id:

event_id set by the producer (a UUID) — best general choice.
A natural key like order_id when each order produces one event of a kind.
(topic, partition, offset) only as a last resort.

The pure dedup function below just remembers keys it has seen.

def make_deduper():
    seen = set()

    def process(event_id, payload):
        if event_id in seen:
            return "skipped (duplicate)"
        seen.add(event_id)
        return f"processed {payload}"

    return process


if __name__ == "__main__":
    handle = make_deduper()
    events = [
        ("evt-1", "order#100"),
        ("evt-2", "order#101"),
        ("evt-1", "order#100"),  # redelivery
    ]
    for eid, payload in events:
        print(eid, "->", handle(eid, payload))

In-Memory Dedup Is Not Enough

The set-based deduper resets every time the FastAPI process restarts — and crashes are exactly when redeliveries happen. You need a durable dedup store that survives restarts and is shared across worker replicas.

Two common backends:

Database table of processed ids — strong consistency, joins with your business data, supports transactions.
Redis with TTL — fast, great when redeliveries only happen within a bounded window.

The right choice depends on how long a duplicate can realistically arrive after the original.

The Inbox Table Pattern

The inbox pattern stores each processed message id in a table with a UNIQUE constraint. Before applying effects, you try to insert the id; a unique-violation means it is a duplicate, so you skip.

Crucially, you insert the dedup row and apply the business change in the same database transaction. Either both commit or neither does — there is no window where the effect happened but the id was not recorded.

-- Inbox table for the consumer (Postgres)
CREATE TABLE processed_messages (
    event_id   TEXT PRIMARY KEY,
    consumer   TEXT NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Same transaction: dedup insert + business write
BEGIN;
INSERT INTO processed_messages (event_id, consumer)
VALUES ('evt-1', 'order-service');  -- fails if already present

UPDATE accounts SET balance = balance - 50
WHERE id = 'acct-7';
COMMIT;

Idempotent Consumer in FastAPI

Here is the dedup-in-transaction idea wired into an async consumer using SQLAlchemy. The INSERT of the event id and the business mutation share one transaction. If the insert raises IntegrityError, we already processed this event and roll back.

Because this depends on a running database and SQLAlchemy session, it is framework/infrastructure code — not something an online judge can execute standalone.

from sqlalchemy.exc import IntegrityError
from sqlalchemy import text

async def handle_event(session, event_id: str, order_id: str, amount: int):
    try:
        async with session.begin():
            await session.execute(
                text("INSERT INTO processed_messages (event_id, consumer) "
                     "VALUES (:eid, 'order-service')"),
                {"eid": event_id},
            )
            await session.execute(
                text("UPDATE orders SET total = total + :amt WHERE id = :oid"),
                {"amt": amount, "oid": order_id},
            )
    except IntegrityError:
        # Duplicate delivery: effect already applied, safe to ignore
        await session.rollback()
        return "duplicate"
    return "processed"

Checkpointing: Commit Offsets After Effects

Idempotency handles duplicates; checkpointing controls when the broker advances past a message. The golden rule: commit the offset only after the effect is durably persisted.

Disable auto-commit and commit manually after your transaction succeeds. This guarantees at-least-once: a crash before the commit replays the message, and idempotency absorbs the replay.

Kafka: enable.auto.commit=false, then consumer.commit().
Pulsar: turn off automatic ack, call consumer.acknowledge(msg) after success.

# Kafka consumer loop with manual commit AFTER processing
from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-service",
    "enable.auto.commit": False,   # we commit ourselves
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["orders"])

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    process_with_dedup(msg)          # durable, idempotent
    consumer.commit(message=msg)     # advance offset only now

Why Ordering of Ack and Effect Matters

The sequence of operations is the whole ballgame. Compare two orderings after the effect is applied but before a crash:

Commit-then-process: offset advances first. A crash loses the message — at-most-once. Bad for money.
Process-then-commit: effect persists first. A crash replays the message — at-least-once. Dedup makes the replay harmless.

Always choose process-then-commit and lean on idempotency. The simulation below shows a redelivery being absorbed.

def consume(messages, crash_after=None):
    seen = set()
    balance = 0
    committed = 0
    for i, (eid, amount) in enumerate(messages):
        if eid not in seen:        # idempotent effect
            seen.add(eid)
            balance += amount
        committed = i + 1          # commit AFTER effect
        if crash_after is not None and i == crash_after:
            break
    return balance, committed


if __name__ == "__main__":
    stream = [("e1", 50), ("e2", 30)]
    # Crash before committing e2, then e2 is redelivered
    bal, _ = consume(stream, crash_after=0)
    replay = stream[1:] + [("e2", 30)]  # redelivery duplicate
    seen = {"e1"}
    for eid, amount in replay:
        if eid not in seen:
            seen.add(eid)
            bal += amount
    print("final balance:", bal)  # 80, not 110

Redis Dedup With a TTL Window

When duplicates can only arrive within a bounded window (say, broker retry plus rebalance time), Redis is a lightweight dedup store. Use SET key value NX EX ttl: it sets the key only if absent and auto-expires it.

The atomic NX flag makes the check-and-claim a single operation, so two concurrent workers cannot both win the same event id. Below is a pure simulation of that logic with an expiry clock.

class FakeRedisNX:
    def __init__(self):
        self.store = {}

    def set_nx_ex(self, key, now, ttl):
        exp = self.store.get(key)
        if exp is not None and exp > now:
            return False          # still claimed -> duplicate
        self.store[key] = now + ttl
        return True               # claimed -> first time


if __name__ == "__main__":
    r = FakeRedisNX()
    print(r.set_nx_ex("evt:1", now=0, ttl=60))   # True
    print(r.set_nx_ex("evt:1", now=10, ttl=60))  # False (dup)
    print(r.set_nx_ex("evt:1", now=70, ttl=60))  # True (expired)

Naturally Idempotent Operations

Sometimes you can sidestep an explicit dedup store by making the operation itself idempotent, so applying it twice equals applying it once.

Upserts: INSERT ... ON CONFLICT DO UPDATE keyed by a business id.
Set assignments: status = 'PAID' instead of a relative toggle.
Conditional writes: apply only if a version or state guard matches.

Relative mutations like balance = balance - 50 are NOT idempotent — running them twice double-counts. Convert such effects to absolute state or guard them with a processed-id check.

-- Idempotent upsert keyed by business id
INSERT INTO order_totals (order_id, total)
VALUES ('order-100', 250)
ON CONFLICT (order_id)
DO UPDATE SET total = EXCLUDED.total;

-- Idempotent state transition (absolute, not relative)
UPDATE orders SET status = 'PAID'
WHERE id = 'order-100' AND status = 'PENDING';

End-to-End Exactly-Once Pipeline

Putting it together for a FastAPI consumer:

Producer stamps every event with a stable event_id (UUID).
Consumer reads with auto-commit disabled.
In one DB transaction: insert the event_id into the inbox (dedup) and apply the business effect.
On IntegrityError, skip — the event was already handled.
Only after the transaction commits, commit/ack the offset to the broker.

This yields effective exactly-once: at-least-once delivery from the broker, exactly-once effect from idempotency, and no lost messages from checkpoint ordering.

Quick Check

You run an at-least-once Kafka consumer that decrements account balances. To achieve effective exactly-once, what is the correct combination?

Responde en la lección interactiva

Recap

You learned how to turn at-least-once delivery into effective exactly-once processing:

Idempotency is the goal — make double-processing produce the same state as single processing.
Pick a stable dedup key, preferably a producer-assigned event_id, never a raw broker offset.
Use a durable inbox (DB unique constraint or Redis NX+TTL) so dedup survives restarts and replicas.
One transaction for the dedup insert and the business effect — all or nothing.
Checkpoint last: commit/ack the offset only after the effect is persisted (process-then-commit).
Prefer naturally idempotent operations (upserts, absolute state) over relative mutations.

Gratis para empezar

Aprende FastAPI Backend Development Bootcamp con un tutor de IA — gratis

Escribe y ejecuta código real en tu navegador, obtén ayuda instantánea de un tutor de IA disponible 24/7 y continúa donde lo dejaste en la web o en la aplicación.

Cursos: 21
Lecciones: 84

Empezar a aprender gratis Ver el curso completo

Preguntas frecuentes

¿La lección «Consumidores idempotentes y semántica exactly-once» es gratis?

Sí — el texto completo de «Consumidores idempotentes y semántica exactly-once» es gratis para leer aquí en la web. Para practicarla de forma interactiva (editor de código integrado y tutor de IA 24/7) y desbloquear el resto del curso de FastAPI Backend Development Bootcamp, actualiza a CoddyKit PRO. El curso de FastAPI Backend Development Bootcamp incluye 4 lecciones en total.

¿Qué aprenderé en «Consumidores idempotentes y semántica exactly-once»?

Deduplicate y cree checkpoints del procesamiento de mensajes para lograr una entrega downstream efectiva exactly-once. Practicas FastAPI Backend Development Bootcamp con código real que ejecutas directamente en el navegador, y un tutor de IA 24/7 responde tus preguntas mientras trabajas en la lección.

¿Necesito experiencia previa para empezar FastAPI Backend Development Bootcamp?

No se requiere experiencia previa. FastAPI Backend Development Bootcamp en CoddyKit está estructurado para principiantes hasta estudiantes avanzados, así que puedes empezar aquí o desde el inicio y avanzar a tu ritmo. Esta es la lección 4 de 4.

¿Cuánto tiempo toma la lección «Consumidores idempotentes y semántica exactly-once»?

La mayoría de las lecciones de CoddyKit toman alrededor de 5–10 minutos. Cada una es compacta e interactiva, así que avanzas constantemente y retomas exactamente por donde dejaste en la web y la app.

¿Puedo escribir y ejecutar código en esta lección de FastAPI Backend Development Bootcamp?

Sí. Cada lección de FastAPI Backend Development Bootcamp incluye un editor de código integrado, así que escribes y ejecutas código real directamente en tu navegador y obtienes retroalimentación instantánea de IA — sin configuración local necesaria.

Todas las lecciones de este curso

Producción y consumo asíncronos de eventos de Kafka
Registro de esquemas y evolución de contratos Avro
Patrón transactional outbox
Consumidores idempotentes y semántica exactly-once

← Volver a FastAPI Backend Development Bootcamp