Идемпотентные потребители и семантика однократной доставки

Message brokers like Kafka and Pulsar default to at-least-once delivery. If a consumer crashes after processing a message but before acknowledging it, the broker redelivers that message on restart.

For a FastAPI backend draining an order-events topic, that means the same OrderPlaced event can hit your handler twice. Without protection, you charge the customer twice or send two confirmation emails.

• At-most-once: ack before processing — fast, but you lose messages on crash.
• At-least-once: ack after processing — no loss, but duplicates.
• Exactly-once (effective): at-least-once delivery plus idempotent handling.

True exactly-once delivery across a network is impossible in the general case. What you can build is effective exactly-once: the broker may deliver a message many times, but your consumer applies its effect exactly once.

A handler is idempotent when running it twice with the same input yields the same final state as running it once. The strategy: give every message a stable unique key, record keys you have already processed, and skip anything you have seen before.

The dedup key must be stable across redeliveries. The broker offset is NOT stable — a rebalanced partition can change offsets, and Pulsar message IDs differ from Kafka offsets.

Prefer a business-level identifier carried in the message itself, or a producer-assigned event id:

• event_id set by the producer (a UUID) — best general choice.
• A natural key like order_id when each order produces one event of a kind.
• (topic, partition, offset) only as a last resort.

The pure dedup function below just remembers keys it has seen.

The set-based deduper resets every time the FastAPI process restarts — and crashes are exactly when redeliveries happen. You need a durable dedup store that survives restarts and is shared across worker replicas.

Two common backends:

• Database table of processed ids — strong consistency, joins with your business data, supports transactions.
• Redis with TTL — fast, great when redeliveries only happen within a bounded window.

The right choice depends on how long a duplicate can realistically arrive after the original.

The inbox pattern stores each processed message id in a table with a UNIQUE constraint. Before applying effects, you try to insert the id; a unique-violation means it is a duplicate, so you skip.

Crucially, you insert the dedup row and apply the business change in the same database transaction. Either both commit or neither does — there is no window where the effect happened but the id was not recorded.

Here is the dedup-in-transaction idea wired into an async consumer using SQLAlchemy. The INSERT of the event id and the business mutation share one transaction. If the insert raises IntegrityError, we already processed this event and roll back.

Because this depends on a running database and SQLAlchemy session, it is framework/infrastructure code — not something an online judge can execute standalone.

Idempotency handles duplicates; checkpointing controls when the broker advances past a message. The golden rule: commit the offset only after the effect is durably persisted.

Disable auto-commit and commit manually after your transaction succeeds. This guarantees at-least-once: a crash before the commit replays the message, and idempotency absorbs the replay.

• Kafka: enable.auto.commit=false, then consumer.commit().
• Pulsar: turn off automatic ack, call consumer.acknowledge(msg) after success.

The sequence of operations is the whole ballgame. Compare two orderings after the effect is applied but before a crash:

• Commit-then-process: offset advances first. A crash loses the message — at-most-once. Bad for money.
• Process-then-commit: effect persists first. A crash replays the message — at-least-once. Dedup makes the replay harmless.

Always choose process-then-commit and lean on idempotency. The simulation below shows a redelivery being absorbed.

When duplicates can only arrive within a bounded window (say, broker retry plus rebalance time), Redis is a lightweight dedup store. Use SET key value NX EX ttl: it sets the key only if absent and auto-expires it.

The atomic NX flag makes the check-and-claim a single operation, so two concurrent workers cannot both win the same event id. Below is a pure simulation of that logic with an expiry clock.

Sometimes you can sidestep an explicit dedup store by making the operation itself idempotent, so applying it twice equals applying it once.

• Upserts: INSERT ... ON CONFLICT DO UPDATE keyed by a business id.
• Set assignments: status = 'PAID' instead of a relative toggle.
• Conditional writes: apply only if a version or state guard matches.

Relative mutations like balance = balance - 50 are NOT idempotent — running them twice double-counts. Convert such effects to absolute state or guard them with a processed-id check.

Putting it together for a FastAPI consumer:

• Producer stamps every event with a stable event_id (UUID).
• Consumer reads with auto-commit disabled.
• In one DB transaction: insert the event_id into the inbox (dedup) and apply the business effect.
• On IntegrityError, skip — the event was already handled.
• Only after the transaction commits, commit/ack the offset to the broker.

This yields effective exactly-once: at-least-once delivery from the broker, exactly-once effect from idempotency, and no lost messages from checkpoint ordering.

You learned how to turn at-least-once delivery into effective exactly-once processing:

• Idempotency is the goal — make double-processing produce the same state as single processing.
• Pick a stable dedup key, preferably a producer-assigned event_id, never a raw broker offset.
• Use a durable inbox (DB unique constraint or Redis NX+TTL) so dedup survives restarts and replicas.
• One transaction for the dedup insert and the business effect — all or nothing.
• Checkpoint last: commit/ack the offset only after the effect is persisted (process-then-commit).
• Prefer naturally idempotent operations (upserts, absolute state) over relative mutations.