The Transactional Outbox Pattern
Guarantee atomic state changes and event publishing using an outbox table and relay process.
The Dual-Write Problem
In an event-driven FastAPI service, a single request often needs to do two things: persist state to your database and publish an event to Kafka or Pulsar.
The trap is that these are two separate systems with no shared transaction. If you commit the DB row then crash before publishing, downstream services never hear about it. If you publish first then the DB commit fails, you emit an event for state that does not exist.
db.commit()succeeds,producer.send()fails → lost eventproducer.send()succeeds,db.commit()fails → phantom event
This is the dual-write problem, and no amount of try/except fully solves it.
async def create_order(db, producer, payload):
order = Order(**payload)
db.add(order)
await db.commit() # write #1: database
await producer.send(
"orders", order.as_event()
) # write #2: broker (may fail!)
return orderWhy You Cannot Just Retry
A common first instinct is: "I'll just wrap the publish in a retry loop." But retries do not close the gap.
- The process can be killed (OOM, deploy, k8s eviction) between commit and publish — no code runs to retry.
- Retrying after a broker timeout can produce duplicates if the first send actually landed.
- You cannot roll back a Kafka write once partially acknowledged.
The core issue is that the commit and the intent to publish are not atomic. We need to make the publish-intent part of the same database transaction as the state change. That is exactly what the Transactional Outbox pattern does.