Avro & Protobuf Schemas
Learn about popular serialization formats like Avro and Protobuf and how they are used with Schema Registry.
Avro & Protobuf Schemas is a free Apache Kafka & Stream Processing Fundamentals lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Apache Kafka & Stream Processing Fundamentals learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Data Formats for Kafka
When sending data through Kafka, it's just bytes. To make sense of these bytes, both the sender (producer) and receiver (consumer) need to agree on a common structure.
This is where data serialization formats and schemas come in, providing a blueprint for your data.
Meet Apache Avro
Apache Avro is a popular, language-agnostic data serialization system. It relies heavily on schemas to define the structure of data.
- Compactness: Data is serialized into a compact binary format.
- Schema Evolution: Avro handles schema changes gracefully, allowing producers and consumers with different schema versions to communicate.
- Language Agnostic: Tools exist for many programming languages.
Avro Schemas: JSON Power
Avro schemas are defined using JSON. This makes them human-readable and easy to manage. A schema describes the data's fields, their types, and any default values.
Basic Avro types include string, int, long, boolean, float, double, bytes, and null.
Building an Avro Schema
Let's define a simple Avro schema for a "User" record. Notice the type, name, and fields with their own name and type.
{
"type": "record",
"name": "User",
"namespace": "com.coddykit.avro",
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": ["int", "null"], "default": 0}
]
}How Avro Uses Schemas
With Avro, the schema travels with the data (or is known by the consumer via Schema Registry). This means the actual data payload is very small, as field names and types aren't repeated for every message.
The schema acts as a contract, ensuring data consistency and enabling efficient serialization and deserialization.
Meet Protocol Buffers (Protobuf)
Protocol Buffers (Protobuf) is Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. It's designed to be smaller and faster than XML.
- Efficiency: Very compact binary format.
- Code Generation: Compilers generate code for various languages based on
.protodefinitions. - Backward/Forward Compatibility: Supports schema evolution through careful field numbering.
Protobuf Schemas: .proto Files
Protobuf schemas are defined in .proto files using a special syntax. You define message types, which are like classes, and specify fields within them.
Each field requires a type, a name, and a unique field number. These numbers are crucial for backward and forward compatibility.
Building a Protobuf Schema
Here's a simple .proto definition for a "Product" message. Notice the syntax, message keyword, and the assigned field numbers (e.g., 1, 2).
syntax = "proto3";
package com.coddykit.protobuf;
message Product {
string id = 1;
string name = 2;
double price = 3;
}Avro vs. Protobuf: A Quick Comparison
Both Avro and Protobuf are excellent for data serialization, but they have different characteristics:
- Schema Format: Avro uses JSON; Protobuf uses its own
.protosyntax. - Code Generation: Avro is schema-first (can generate code); Protobuf is often code-first (generates code from
.proto). - Data Size: Both are very compact, often outperforming JSON/XML.
- Schema Evolution: Both support it, but with different strategies (Avro relies on Schema Registry, Protobuf on field numbers).
Quick Check
You've learned about Avro and Protobuf. Which statement correctly identifies the primary way Avro and Protobuf schemas are defined?
Recap: Structured Data for Kafka
Great job! In this lesson, we explored two powerful data serialization formats: Apache Avro and Protocol Buffers (Protobuf).
- Avro uses JSON for schema definitions and is excellent for schema evolution.
- Protobuf uses a
.protosyntax with field numbers for compact, efficient data.
Both help ensure data consistency and efficiency when used with Kafka and Schema Registry, which we'll explore further next!
Frequently asked questions
Is the “Avro & Protobuf Schemas” lesson free?
Yes — the full text of “Avro & Protobuf Schemas” is free to read here on the web, and the Apache Kafka & Stream Processing Fundamentals course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Apache Kafka & Stream Processing Fundamentals course, upgrade to CoddyKit PRO.
What will I learn in “Avro & Protobuf Schemas”?
Learn about popular serialization formats like Avro and Protobuf and how they are used with Schema Registry. You practise Apache Kafka & Stream Processing Fundamentals with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Apache Kafka & Stream Processing Fundamentals?
No prior experience is required. Apache Kafka & Stream Processing Fundamentals on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Avro & Protobuf Schemas” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Apache Kafka & Stream Processing Fundamentals lesson?
Yes. Every Apache Kafka & Stream Processing Fundamentals lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.