FastAPI Backend Development Bootcamp · Lesson

Idempotent Consumers and Exactly-Once Semantics

Deduplicate and checkpoint message processing to achieve effective exactly-once delivery downstream.

Idempotent Consumers and Exactly-Once Semantics is a free FastAPI Backend Development Bootcamp lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the FastAPI Backend Development Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Delivery Problem

Message brokers like Kafka and Pulsar default to at-least-once delivery. If a consumer crashes after processing a message but before acknowledging it, the broker redelivers that message on restart.

For a FastAPI backend draining an order-events topic, that means the same OrderPlaced event can hit your handler twice. Without protection, you charge the customer twice or send two confirmation emails.

At-most-once: ack before processing — fast, but you lose messages on crash.
At-least-once: ack after processing — no loss, but duplicates.
Exactly-once (effective): at-least-once delivery plus idempotent handling.

Idempotency Is the Real Goal

True exactly-once delivery across a network is impossible in the general case. What you can build is effective exactly-once: the broker may deliver a message many times, but your consumer applies its effect exactly once.

A handler is idempotent when running it twice with the same input yields the same final state as running it once. The strategy: give every message a stable unique key, record keys you have already processed, and skip anything you have seen before.

Choosing a Deduplication Key

The dedup key must be stable across redeliveries. The broker offset is NOT stable — a rebalanced partition can change offsets, and Pulsar message IDs differ from Kafka offsets.

Prefer a business-level identifier carried in the message itself, or a producer-assigned event id:

event_id set by the producer (a UUID) — best general choice.
A natural key like order_id when each order produces one event of a kind.
(topic, partition, offset) only as a last resort.

The pure dedup function below just remembers keys it has seen.

def make_deduper():
    seen = set()

    def process(event_id, payload):
        if event_id in seen:
            return "skipped (duplicate)"
        seen.add(event_id)
        return f"processed {payload}"

    return process


if __name__ == "__main__":
    handle = make_deduper()
    events = [
        ("evt-1", "order#100"),
        ("evt-2", "order#101"),
        ("evt-1", "order#100"),  # redelivery
    ]
    for eid, payload in events:
        print(eid, "->", handle(eid, payload))

In-Memory Dedup Is Not Enough

The set-based deduper resets every time the FastAPI process restarts — and crashes are exactly when redeliveries happen. You need a durable dedup store that survives restarts and is shared across worker replicas.

Two common backends:

Database table of processed ids — strong consistency, joins with your business data, supports transactions.
Redis with TTL — fast, great when redeliveries only happen within a bounded window.

The right choice depends on how long a duplicate can realistically arrive after the original.

The Inbox Table Pattern

The inbox pattern stores each processed message id in a table with a UNIQUE constraint. Before applying effects, you try to insert the id; a unique-violation means it is a duplicate, so you skip.

Crucially, you insert the dedup row and apply the business change in the same database transaction. Either both commit or neither does — there is no window where the effect happened but the id was not recorded.

-- Inbox table for the consumer (Postgres)
CREATE TABLE processed_messages (
    event_id   TEXT PRIMARY KEY,
    consumer   TEXT NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Same transaction: dedup insert + business write
BEGIN;
INSERT INTO processed_messages (event_id, consumer)
VALUES ('evt-1', 'order-service');  -- fails if already present

UPDATE accounts SET balance = balance - 50
WHERE id = 'acct-7';
COMMIT;

Idempotent Consumer in FastAPI

Here is the dedup-in-transaction idea wired into an async consumer using SQLAlchemy. The INSERT of the event id and the business mutation share one transaction. If the insert raises IntegrityError, we already processed this event and roll back.

Because this depends on a running database and SQLAlchemy session, it is framework/infrastructure code — not something an online judge can execute standalone.

from sqlalchemy.exc import IntegrityError
from sqlalchemy import text

async def handle_event(session, event_id: str, order_id: str, amount: int):
    try:
        async with session.begin():
            await session.execute(
                text("INSERT INTO processed_messages (event_id, consumer) "
                     "VALUES (:eid, 'order-service')"),
                {"eid": event_id},
            )
            await session.execute(
                text("UPDATE orders SET total = total + :amt WHERE id = :oid"),
                {"amt": amount, "oid": order_id},
            )
    except IntegrityError:
        # Duplicate delivery: effect already applied, safe to ignore
        await session.rollback()
        return "duplicate"
    return "processed"

Checkpointing: Commit Offsets After Effects

Idempotency handles duplicates; checkpointing controls when the broker advances past a message. The golden rule: commit the offset only after the effect is durably persisted.

Disable auto-commit and commit manually after your transaction succeeds. This guarantees at-least-once: a crash before the commit replays the message, and idempotency absorbs the replay.

Kafka: enable.auto.commit=false, then consumer.commit().
Pulsar: turn off automatic ack, call consumer.acknowledge(msg) after success.

# Kafka consumer loop with manual commit AFTER processing
from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-service",
    "enable.auto.commit": False,   # we commit ourselves
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["orders"])

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    process_with_dedup(msg)          # durable, idempotent
    consumer.commit(message=msg)     # advance offset only now

Why Ordering of Ack and Effect Matters

The sequence of operations is the whole ballgame. Compare two orderings after the effect is applied but before a crash:

Commit-then-process: offset advances first. A crash loses the message — at-most-once. Bad for money.
Process-then-commit: effect persists first. A crash replays the message — at-least-once. Dedup makes the replay harmless.

Always choose process-then-commit and lean on idempotency. The simulation below shows a redelivery being absorbed.

def consume(messages, crash_after=None):
    seen = set()
    balance = 0
    committed = 0
    for i, (eid, amount) in enumerate(messages):
        if eid not in seen:        # idempotent effect
            seen.add(eid)
            balance += amount
        committed = i + 1          # commit AFTER effect
        if crash_after is not None and i == crash_after:
            break
    return balance, committed


if __name__ == "__main__":
    stream = [("e1", 50), ("e2", 30)]
    # Crash before committing e2, then e2 is redelivered
    bal, _ = consume(stream, crash_after=0)
    replay = stream[1:] + [("e2", 30)]  # redelivery duplicate
    seen = {"e1"}
    for eid, amount in replay:
        if eid not in seen:
            seen.add(eid)
            bal += amount
    print("final balance:", bal)  # 80, not 110

Redis Dedup With a TTL Window

When duplicates can only arrive within a bounded window (say, broker retry plus rebalance time), Redis is a lightweight dedup store. Use SET key value NX EX ttl: it sets the key only if absent and auto-expires it.

The atomic NX flag makes the check-and-claim a single operation, so two concurrent workers cannot both win the same event id. Below is a pure simulation of that logic with an expiry clock.

class FakeRedisNX:
    def __init__(self):
        self.store = {}

    def set_nx_ex(self, key, now, ttl):
        exp = self.store.get(key)
        if exp is not None and exp > now:
            return False          # still claimed -> duplicate
        self.store[key] = now + ttl
        return True               # claimed -> first time


if __name__ == "__main__":
    r = FakeRedisNX()
    print(r.set_nx_ex("evt:1", now=0, ttl=60))   # True
    print(r.set_nx_ex("evt:1", now=10, ttl=60))  # False (dup)
    print(r.set_nx_ex("evt:1", now=70, ttl=60))  # True (expired)

Naturally Idempotent Operations

Sometimes you can sidestep an explicit dedup store by making the operation itself idempotent, so applying it twice equals applying it once.

Upserts: INSERT ... ON CONFLICT DO UPDATE keyed by a business id.
Set assignments: status = 'PAID' instead of a relative toggle.
Conditional writes: apply only if a version or state guard matches.

Relative mutations like balance = balance - 50 are NOT idempotent — running them twice double-counts. Convert such effects to absolute state or guard them with a processed-id check.

-- Idempotent upsert keyed by business id
INSERT INTO order_totals (order_id, total)
VALUES ('order-100', 250)
ON CONFLICT (order_id)
DO UPDATE SET total = EXCLUDED.total;

-- Idempotent state transition (absolute, not relative)
UPDATE orders SET status = 'PAID'
WHERE id = 'order-100' AND status = 'PENDING';

End-to-End Exactly-Once Pipeline

Putting it together for a FastAPI consumer:

Producer stamps every event with a stable event_id (UUID).
Consumer reads with auto-commit disabled.
In one DB transaction: insert the event_id into the inbox (dedup) and apply the business effect.
On IntegrityError, skip — the event was already handled.
Only after the transaction commits, commit/ack the offset to the broker.

This yields effective exactly-once: at-least-once delivery from the broker, exactly-once effect from idempotency, and no lost messages from checkpoint ordering.

Quick Check

You run an at-least-once Kafka consumer that decrements account balances. To achieve effective exactly-once, what is the correct combination?

Recap

You learned how to turn at-least-once delivery into effective exactly-once processing:

Idempotency is the goal — make double-processing produce the same state as single processing.
Pick a stable dedup key, preferably a producer-assigned event_id, never a raw broker offset.
Use a durable inbox (DB unique constraint or Redis NX+TTL) so dedup survives restarts and replicas.
One transaction for the dedup insert and the business effect — all or nothing.
Checkpoint last: commit/ack the offset only after the effect is persisted (process-then-commit).
Prefer naturally idempotent operations (upserts, absolute state) over relative mutations.

Frequently asked questions

Is the “Idempotent Consumers and Exactly-Once Semantics” lesson free?

Yes — the full text of “Idempotent Consumers and Exactly-Once Semantics” is free to read here on the web, and the FastAPI Backend Development Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the FastAPI Backend Development Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Idempotent Consumers and Exactly-Once Semantics”?

Deduplicate and checkpoint message processing to achieve effective exactly-once delivery downstream. You practise FastAPI Backend Development Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start FastAPI Backend Development Bootcamp?

No prior experience is required. FastAPI Backend Development Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Idempotent Consumers and Exactly-Once Semantics” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this FastAPI Backend Development Bootcamp lesson?

Yes. Every FastAPI Backend Development Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Producing and Consuming Kafka Events Asynchronously
Schema Registry and Avro Contract Evolution
The Transactional Outbox Pattern
Idempotent Consumers and Exactly-Once Semantics

← Back to FastAPI Backend Development Bootcamp