Skema Avro & Protobuf

When sending data through Kafka, it's just bytes. To make sense of these bytes, both the sender (producer) and receiver (consumer) need to agree on a common structure.

This is where data serialization formats and schemas come in, providing a blueprint for your data.

Apache Avro is a popular, language-agnostic data serialization system. It relies heavily on schemas to define the structure of data.

• Compactness: Data is serialized into a compact binary format.
• Schema Evolution: Avro handles schema changes gracefully, allowing producers and consumers with different schema versions to communicate.
• Language Agnostic: Tools exist for many programming languages.

Avro schemas are defined using JSON. This makes them human-readable and easy to manage. A schema describes the data's fields, their types, and any default values.

Basic Avro types include string, int, long, boolean, float, double, bytes, and null.

Let's define a simple Avro schema for a "User" record. Notice the type, name, and fields with their own name and type.

With Avro, the schema travels with the data (or is known by the consumer via Schema Registry). This means the actual data payload is very small, as field names and types aren't repeated for every message.

The schema acts as a contract, ensuring data consistency and enabling efficient serialization and deserialization.

Protocol Buffers (Protobuf) is Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. It's designed to be smaller and faster than XML.

• Efficiency: Very compact binary format.
• Code Generation: Compilers generate code for various languages based on .proto definitions.
• Backward/Forward Compatibility: Supports schema evolution through careful field numbering.

Protobuf schemas are defined in .proto files using a special syntax. You define message types, which are like classes, and specify fields within them.

Each field requires a type, a name, and a unique field number. These numbers are crucial for backward and forward compatibility.

Here's a simple .proto definition for a "Product" message. Notice the syntax, message keyword, and the assigned field numbers (e.g., 1, 2).

Both Avro and Protobuf are excellent for data serialization, but they have different characteristics:

• Schema Format: Avro uses JSON; Protobuf uses its own .proto syntax.
• Code Generation: Avro is schema-first (can generate code); Protobuf is often code-first (generates code from .proto).
• Data Size: Both are very compact, often outperforming JSON/XML.
• Schema Evolution: Both support it, but with different strategies (Avro relies on Schema Registry, Protobuf on field numbers).

You've learned about Avro and Protobuf. Which statement correctly identifies the primary way Avro and Protobuf schemas are defined?

Great job! In this lesson, we explored two powerful data serialization formats: Apache Avro and Protocol Buffers (Protobuf).

• Avro uses JSON for schema definitions and is excellent for schema evolution.
• Protobuf uses a .proto syntax with field numbers for compact, efficient data.

Both help ensure data consistency and efficiency when used with Kafka and Schema Registry, which we'll explore further next!