0PricingLogin▼
FastAPI Backend Development Bootcamp · Lesson

Producing and Consuming Kafka Events Asynchronously

Integrate aiokafka with FastAPI to publish and consume events without blocking the event loop.

Why async Kafka in FastAPI

FastAPI runs on an asyncio event loop. If you publish or poll Kafka with a blocking client (like the standard kafka-python library), every network call freezes the entire loop, stalling all concurrent requests.

  • aiokafka is a native asyncio Kafka client that never blocks the loop.
  • Its send and getone operations are coroutines you await.
  • This lets one worker handle thousands of in-flight requests while Kafka I/O is pending.

In this lesson you will wire an AIOKafkaProducer and an AIOKafkaConsumer into a FastAPI app the right way.

Producer lifecycle with lifespan

A producer maintains TCP connections and a background sender task. You must start() it once at app boot and stop() it on shutdown — never per request. The modern FastAPI way is the lifespan context manager.

  • await producer.start() opens connections and the sender loop.
  • await producer.stop() flushes pending batches and closes cleanly.
  • Store the producer on app.state so routes can reach it.
from contextlib import asynccontextmanager
from fastapi import FastAPI
from aiokafka import AIOKafkaProducer


@asynccontextmanager
async def lifespan(app: FastAPI):
    producer = AIOKafkaProducer(
        bootstrap_servers="localhost:9092",
        enable_idempotence=True,
    )
    await producer.start()
    app.state.producer = producer
    try:
        yield
    finally:
        await producer.stop()


app = FastAPI(lifespan=lifespan)

All lessons in this course

  1. Producing and Consuming Kafka Events Asynchronously
  2. Schema Registry and Avro Contract Evolution
  3. The Transactional Outbox Pattern
  4. Idempotent Consumers and Exactly-Once Semantics
← Back to FastAPI Backend Development Bootcamp