Idempotent Consumers and Exactly-Once Semantics

Message brokers like Kafka and Pulsar default to at-least-once delivery. If a consumer crashes after processing a message but before acknowledging it, the broker redelivers that message on restart.

For a FastAPI backend draining an order-events topic, that means the same OrderPlaced event can hit your handler twice. Without protection, you charge the customer twice or send two confirmation emails.

• At-most-once: ack before processing — fast, but you lose messages on crash.
• At-least-once: ack after processing — no loss, but duplicates.
• Exactly-once (effective): at-least-once delivery plus idempotent handling.

True exactly-once delivery across a network is impossible in the general case. What you can build is effective exactly-once: the broker may deliver a message many times, but your consumer applies its effect exactly once.

A handler is idempotent when running it twice with the same input yields the same final state as running it once. The strategy: give every message a stable unique key, record keys you have already processed, and skip anything you have seen before.