0Pricing
FastAPI Backend Development Bootcamp · 강의

트랜잭션 아웃박스 패턴

아웃박스 테이블과 릴레이 프로세스를 사용해 상태 변경과 이벤트 발행의 원자성을 보장합니다.

트랜잭션 아웃박스 패턴은(는) CoddyKit의 무료 FastAPI Backend Development Bootcamp 강의입니다. 이것은 4개 중 3번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 FastAPI Backend Development Bootcamp 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. FastAPI Backend Development Bootcamp 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

The Dual-Write Problem

In an event-driven FastAPI service, a single request often needs to do two things: persist state to your database and publish an event to Kafka or Pulsar.

The trap is that these are two separate systems with no shared transaction. If you commit the DB row then crash before publishing, downstream services never hear about it. If you publish first then the DB commit fails, you emit an event for state that does not exist.

  • db.commit() succeeds, producer.send() fails → lost event
  • producer.send() succeeds, db.commit() fails → phantom event

This is the dual-write problem, and no amount of try/except fully solves it.

async def create_order(db, producer, payload):
    order = Order(**payload)
    db.add(order)
    await db.commit()          # write #1: database
    await producer.send(
        "orders", order.as_event()
    )                          # write #2: broker (may fail!)
    return order

Why You Cannot Just Retry

A common first instinct is: "I'll just wrap the publish in a retry loop." But retries do not close the gap.

  • The process can be killed (OOM, deploy, k8s eviction) between commit and publish — no code runs to retry.
  • Retrying after a broker timeout can produce duplicates if the first send actually landed.
  • You cannot roll back a Kafka write once partially acknowledged.

The core issue is that the commit and the intent to publish are not atomic. We need to make the publish-intent part of the same database transaction as the state change. That is exactly what the Transactional Outbox pattern does.

The Outbox Idea

The Transactional Outbox pattern adds an outbox table in the same database as your business data. Instead of publishing to the broker inline, you INSERT a row describing the event into the outbox table — inside the same transaction that changes your state.

Because the order row and the outbox row are written in one transaction, they either both commit or both roll back. There is no window where one exists without the other.

A separate relay process later reads unpublished outbox rows and pushes them to Kafka/Pulsar, marking them sent. The broker write is decoupled from the request path.

  • Atomicity comes from the DB transaction, not from a distributed transaction.
  • Delivery becomes at-least-once — consumers must be idempotent.

Designing the Outbox Table

The outbox table needs enough metadata for the relay to publish correctly and for you to debug. A typical schema:

  • id — UUID primary key, also reused as the event id for idempotency
  • aggregate_type / aggregate_id — what the event is about (e.g. order / order id), often used as the partition key
  • event_type — e.g. OrderCreated
  • payload — JSON body of the event
  • created_at — ordering
  • published_at — NULL until the relay sends it

Keeping aggregate_id lets the relay set the Kafka partition key so all events for one order stay ordered.

from sqlalchemy import Column, String, DateTime, JSON, func
from sqlalchemy.dialects.postgresql import UUID
from .db import Base
import uuid

class Outbox(Base):
    __tablename__ = "outbox"
    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    aggregate_type = Column(String, nullable=False)
    aggregate_id = Column(String, nullable=False)
    event_type = Column(String, nullable=False)
    payload = Column(JSON, nullable=False)
    created_at = Column(DateTime(timezone=True), server_default=func.now())
    published_at = Column(DateTime(timezone=True), nullable=True)

Writing State and Event Atomically

Here is the heart of the pattern. The order and the outbox row are added to the same session and committed together. Notice there is no broker call in the request handler at all.

If commit() fails, neither row exists. If it succeeds, both exist. The event will be delivered by the relay, guaranteed.

import uuid

async def create_order(db, payload):
    order = Order(**payload)
    db.add(order)

    event = Outbox(
        id=uuid.uuid4(),
        aggregate_type="order",
        aggregate_id=str(order.id),
        event_type="OrderCreated",
        payload={"order_id": str(order.id), **payload},
    )
    db.add(event)

    await db.commit()   # ONE transaction: state + event intent
    return order

The Relay: Polling Publisher

The simplest relay is a polling publisher: a background loop that repeatedly selects unpublished outbox rows, sends them to the broker, and marks them published.

  • Select rows where published_at IS NULL, ordered by created_at.
  • Publish each to Kafka/Pulsar using aggregate_id as the key.
  • Set published_at = now() and commit.

If the process crashes after publishing but before marking, the row gets re-sent on the next poll — hence at-least-once and the need for idempotent consumers. The event id is the deduplication key.

async def relay_once(db, producer, batch=100):
    rows = await db.fetch(
        "SELECT id, aggregate_id, event_type, payload "
        "FROM outbox WHERE published_at IS NULL "
        "ORDER BY created_at LIMIT $1",
        batch,
    )
    for r in rows:
        await producer.send(
            topic=r["event_type"],
            key=r["aggregate_id"].encode(),
            value=r["payload"],
            headers=[("event_id", str(r["id"]).encode())],
        )
        await db.execute(
            "UPDATE outbox SET published_at = now() WHERE id = $1",
            r["id"],
        )

Avoiding Double Processing with FOR UPDATE SKIP LOCKED

If you run more than one relay replica for throughput, two workers might grab the same outbox rows and double-publish. Postgres gives you a clean fix: SELECT ... FOR UPDATE SKIP LOCKED.

  • FOR UPDATE locks the selected rows for the transaction.
  • SKIP LOCKED makes other workers skip already-locked rows instead of blocking.

Each worker claims a disjoint batch, publishes, marks them, and commits — releasing the locks. This turns the outbox into a safe concurrent work queue without an external queue system.

SELECT id, aggregate_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;

Idempotent Consumers Are Mandatory

Because the outbox guarantees at-least-once delivery, every consumer must tolerate seeing the same event more than once. The standard technique is a processed-events table keyed by the event id.

Before applying an event, attempt to insert its id. If it already exists, you have seen it — skip. Do the dedup insert and the business effect in the same transaction so a crash cannot leave them out of sync.

def handle_event(conn, event_id, body):
    cur = conn.cursor()
    try:
        cur.execute(
            "INSERT INTO processed_events(event_id) VALUES (%s)",
            (event_id,),
        )
    except UniqueViolation:
        conn.rollback()      # duplicate -> already handled
        return
    apply_business_change(cur, body)
    conn.commit()            # dedup + effect: one transaction

Change Data Capture as a Relay

Polling is simple but adds latency and DB load. A higher-performance alternative reads Postgres's write-ahead log (WAL) directly using Change Data Capture — typically Debezium.

  • Debezium tails the WAL and emits a Kafka message for every INSERT into the outbox table.
  • No published_at column or polling loop needed — the log is the queue.
  • Its Outbox Event Router SMT maps outbox columns to topic, key, and payload.

Trade-off: CDC is operationally heavier (connector, replication slot) but gives near-real-time delivery and offloads the relay entirely from your FastAPI app.

Ordering and Partition Keys

Events for the same aggregate usually must arrive in order — OrderCreated before OrderShipped. Both the outbox and the broker preserve this if you are careful:

  • Order the relay select by created_at (or a monotonic sequence).
  • Use aggregate_id as the Kafka partition key so all of one order's events land on the same partition, which Kafka delivers in order.

Events for different aggregates can interleave freely — you only need per-aggregate ordering, which is what partition-by-key buys you. A pure global ordering across all orders is rarely required and kills parallelism.

# Stable hash -> partition keeps one aggregate on one partition
def partition_for(aggregate_id: str, num_partitions: int) -> int:
    h = 0
    for ch in aggregate_id:
        h = (h * 31 + ord(ch)) & 0xFFFFFFFF
    return h % num_partitions

if __name__ == "__main__":
    ids = ["order-1", "order-1", "order-2", "order-3"]
    for a in ids:
        print(a, "->", partition_for(a, 6))

Operating the Outbox in Production

A few practices keep the outbox healthy at scale:

  • Prune published rows — periodically delete rows where published_at is older than a retention window, or move them to an archive, so the table stays small and the partial index stays fast.
  • Partial index on unpublished rows: CREATE INDEX ... ON outbox (created_at) WHERE published_at IS NULL; keeps the relay's hot query cheap.
  • Monitor lag — alert on the count or age of unpublished rows; a growing backlog means the relay is down or the broker is unreachable.
  • Idempotent producer — enable Kafka enable.idempotence=true to suppress relay-side resend duplicates at the broker level.

Quick Check: Why the Outbox Works

Test your understanding of why the Transactional Outbox pattern actually solves the dual-write problem.

Recap

You learned how to publish events reliably from a FastAPI service:

  • The dual-write problem: committing to the DB and publishing to Kafka/Pulsar are separate, non-atomic actions that can desync on a crash.
  • The Transactional Outbox writes an event row into an outbox table in the same transaction as the state change, making them atomic.
  • A relay (polling publisher or Debezium CDC) later sends outbox rows to the broker and marks them published.
  • Use FOR UPDATE SKIP LOCKED for safe concurrent relays, aggregate_id as the partition key for ordering, and a processed-events table for idempotent, at-least-once consumers.
  • Operate it with pruning, a partial index, and backlog monitoring.

자주 묻는 질문

“트랜잭션 아웃박스 패턴” 강의는 무료인가요?

네 — “트랜잭션 아웃박스 패턴” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 FastAPI Backend Development Bootcamp 강의 전체를 잠금 해제할 수 있습니다. FastAPI Backend Development Bootcamp 강의에는 총 4개의 강의가 포함되어 있습니다.

“트랜잭션 아웃박스 패턴”에서 뭘 배우나요?

아웃박스 테이블과 릴레이 프로세스를 사용해 상태 변경과 이벤트 발행의 원자성을 보장합니다. 브라우저에서 직접 실행하는 실습 코드로 FastAPI Backend Development Bootcamp을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

FastAPI Backend Development Bootcamp을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 FastAPI Backend Development Bootcamp은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 3번째 강의입니다.

“트랜잭션 아웃박스 패턴” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 FastAPI Backend Development Bootcamp 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 FastAPI Backend Development Bootcamp 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

  1. Kafka 이벤트의 비동기 생성과 소비
  2. 스키마 레지스트리와 Avro 계약 진화
  3. 트랜잭션 아웃박스 패턴
  4. 멱등 소비자와 정확히 한 번 의미론
← FastAPI Backend Development Bootcamp(으)로 돌아가기