Kafka Streams의 Serdes와 데이터 직렬화

A Serde is a combined Serializer and Deserializer. Kafka Streams uses it to convert between your Java objects and the bytes stored in Kafka.

Every key and value flowing through a topology needs a Serde.

Unlike a plain consumer, Kafka Streams writes intermediate results back to Kafka (for repartitioning and state).

So it must know how to serialize data not just on input/output, but at every shuffle and store — hence Serdes appear throughout the API.

The Serdes factory class provides ready-made Serdes for common types.

You can set default key and value Serdes in the Streams config so you don't repeat them everywhere.

When a specific stream uses a different type, override the default with Consumed, Produced, or Materialized.

For domain objects, a common choice is a JSON Serde that maps a class to and from JSON bytes.

For schema-governed data, use the Confluent SpecificAvroSerde or GenericAvroSerde.

• They register and look up schemas in Schema Registry.
• They enforce compatibility as your data evolves.

Operations like selectKey or groupBy trigger a repartition through an internal topic.

Kafka Streams needs valid key/value Serdes at that point — a missing or wrong Serde here is a frequent source of runtime errors.

Stateful operations materialize results into a store backed by a changelog topic.

Watch for these pitfalls:

• ClassCastException — the default Serde doesn't match the actual type.
• SerializationException — bytes don't match the deserializer (wrong topic data).
• Forgetting to override the Serde after changing the value type.

Keep serialization sane:

• Set sensible defaults, override only when types change.
• Use Schema Registry Serdes for evolving data contracts.
• Make custom Serdes null-safe.
• Match Serde types exactly at repartition and store boundaries.

You learned how Serdes drive serialization in Kafka Streams.

• A Serde bundles a serializer and deserializer.
• Set defaults in config; override with Consumed/Produced/Materialized.
• Use JSON or Avro/Schema Registry Serdes for domain objects.
• Mismatched Serdes are a top cause of runtime errors.