Schema Registry und Avro-Vertragsentwicklung

In an event-driven FastAPI backend, your service publishes events to Kafka or Pulsar and many independent consumers read them. The event payload is a contract: producers and consumers must agree on field names, types, and structure.

• If the producer renames user_id to userId, every consumer breaks silently.
• Plain JSON has no enforced shape, so a typo ships straight to production.

A Schema Registry stores versioned schemas centrally and rejects messages that violate the agreed contract, decoupling teams while keeping data safe.

Apache Avro is the most common serialization format used with schema registries. Each record is described by a JSON schema, and the binary payload itself carries no field names, only values, making it compact.

• The schema defines name, type, fields, and optional default values.
• Readers need the schema to decode the bytes, which is exactly why the registry exists.

Below is a minimal Avro schema for an OrderCreated event.

The pure-Python fastavro library lets you encode and decode Avro records without any broker. This is the same byte layout your producer would push to Kafka.

• parse_schema validates the schema once.
• schemaless_writer writes the binary body; schemaless_reader decodes it.

Notice the encoded bytes contain values only, not field names.

When you publish through a registry, the value is not bare Avro. Confluent's serializer prepends a 5-byte header so consumers know which schema to fetch.

• Byte 0: a magic byte, always 0x00.
• Bytes 1-4: a big-endian 4-byte schema ID.
• Remaining bytes: the schemaless Avro body.

The consumer reads the ID, downloads that exact schema version from the registry, and decodes the body. This is how old and new payloads coexist on the same topic.

A clean pattern is to register your producer's schema once at application startup using the registry's REST API. The registry returns a stable schema ID you reuse for every message.

• Subjects follow the <topic>-value convention by default.
• Registering an identical schema is idempotent: you get the same ID back.

This snippet posts an Avro schema to a Confluent-compatible registry.

The registry enforces an evolution policy per subject. The mode you pick decides which schema changes are allowed and dictates your upgrade order.

• BACKWARD (default): new schema can read data written by the previous schema. Upgrade consumers first.
• FORWARD: previous schema can read data written by the new schema. Upgrade producers first.
• FULL: both directions hold. Order does not matter.
• *_TRANSITIVE: the check runs against all prior versions, not just the latest.

Most teams default to BACKWARD because consumers usually lag behind producers.

Under BACKWARD compatibility, you may add a field only if it has a default. A consumer using the new schema reading an old message simply fills in the default; a removed field also needs the old one to have had a default.

• Adding discount_cents with default: 0 is safe.
• Adding it without a default is rejected, because old records have no value to supply.

The new version below evolves the order event safely.

Avro resolves differences between the writer schema (used to encode) and the reader schema (used to decode). When the reader has a new field with a default, decoding old bytes injects that default automatically.

• Pass both schemas to schemaless_reader as reader and writer.
• The missing discount_cents appears as its default 0.

This is exactly what makes BACKWARD evolution non-breaking in production.

Some edits can never be compatible and the registry's compatibility check (a pre-flight POST .../compatibility/subjects/<s>/versions/latest) will return is_compatible: false.

• Renaming a field (it becomes an add + a remove without aliases).
• Changing a type incompatibly, e.g. string to long.
• Adding a required field with no default under BACKWARD.

To rename safely, use Avro aliases so the reader maps the old name onto the new one.

Catch breaking changes before they reach the broker by calling the registry's compatibility endpoint from your CI pipeline. If the proposed schema is incompatible, fail the build.

• This protects every consumer without running a single message through Kafka.
• Run it as a step in the same job that builds your FastAPI image.

The helper returns True only when the registry approves the new version.

In production you let the serializer handle the wire format and registry lookups. confluent-kafka's AvroSerializer registers the schema, prepends the ID, and encodes the body; AvroDeserializer reverses it.

• The serializer caches schema IDs, so the registry is hit rarely.
• Consumers transparently fetch whatever writer schema each message was encoded with.

This is the glue between your FastAPI event publisher and downstream services.

Your team needs to add a new optional field to a Kafka event, and you cannot redeploy every consumer at the same moment as the producer. Consumers typically lag behind producers in deployment.

You now know how to enforce and evolve event contracts in an event-driven FastAPI backend.

• A Schema Registry stores versioned schemas and hands out a schema ID embedded in the Confluent wire format (magic byte + 4-byte ID + Avro body).
• Avro separates writer and reader schemas, resolving differences via defaults and aliases.
• BACKWARD (the common default) means new schemas read old data: add fields only with defaults, upgrade consumers first.
• FORWARD upgrades producers first; FULL allows either order; TRANSITIVE variants check all prior versions.
• Run the registry's compatibility check in CI to block breaking changes before they reach Kafka or Pulsar.

Treat your schemas as code: version them, review them, and let the registry guard the contract.