Single Message Transforms (SMTs)
Learn how Kafka Connect Single Message Transforms reshape records inline between connectors and Kafka without writing a stream processor.
Single Message Transforms (SMTs) is a free Apache Kafka & Stream Processing Fundamentals lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Apache Kafka & Stream Processing Fundamentals learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Are SMTs?
Single Message Transforms (SMTs) are lightweight functions applied to each record as it flows through a Kafka Connect connector.
They let you tweak data — rename fields, add metadata, route topics — without a separate processing job.
Where SMTs Run
SMTs run inside the connector worker:
- For a source connector, after reading from the system and before writing to Kafka.
- For a sink connector, after reading from Kafka and before writing to the target.
Configuring a Transform
Transforms are configured declaratively in the connector's JSON or properties. You name the chain, then configure each transform.
{
"transforms": "addTs",
"transforms.addTs.type":
"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.addTs.timestamp.field": "ingestedAt"
}Common Built-in SMTs
Kafka Connect ships many useful transforms:
InsertField— add a static or metadata field.ReplaceField— rename or drop fields.MaskField— redact sensitive values.ExtractField— pull one field up as the whole value.
Routing with RegexRouter
RegexRouter rewrites the destination topic name, handy for adding prefixes when ingesting many tables.
{
"transforms": "route",
"transforms.route.type":
"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "(.*)",
"transforms.route.replacement": "db_$1"
}Chaining Transforms
List multiple transforms separated by commas; they run in order, each receiving the previous one's output.
{
"transforms": "mask,route",
"transforms.mask.type":
"org.apache.kafka.connect.transforms.MaskField$Value",
"transforms.mask.fields": "ssn,creditCard",
"transforms.route.type":
"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "(.*)",
"transforms.route.replacement": "secure_$1"
}Key vs. Value Variants
Many SMTs come in two forms, suffixed with $Key or $Value, so you can target the record's key or value independently.
For example InsertField$Key versus InsertField$Value.
Predicates
Predicates let a transform apply conditionally — for example, only to tombstone records or to a specific topic.
{
"predicates": "isTomb",
"predicates.isTomb.type":
"org.apache.kafka.connect.transforms.predicates.RecordIsTombstone",
"transforms.drop.predicate": "isTomb"
}When NOT to Use SMTs
SMTs operate on one record at a time. They cannot:
- Join across records or topics.
- Aggregate or window.
- Call external services efficiently.
For those, use Kafka Streams or ksqlDB instead.
Custom SMTs
You can implement your own by writing a class that implements the Transformation interface, packaging it as a JAR, and placing it on the worker plugin path.
public class UppercaseField
implements Transformation<SourceRecord> {
public SourceRecord apply(SourceRecord record) {
// reshape and return the record
return record;
}
}Best Practices
Use SMTs wisely:
- Keep chains short and readable.
- Mask sensitive data as early as possible.
- Prefer predicates over per-topic connectors when filtering.
- Move heavy logic to a stream processor.
Quick Check
Test your understanding of SMTs.
Recap
You learned about Single Message Transforms.
- SMTs reshape each record inline within a connector.
- Use built-ins like InsertField, ReplaceField, MaskField, RegexRouter.
- Chain them and gate with predicates.
- For joins, aggregations, or windows, use a stream processor instead.
Frequently asked questions
Is the “Single Message Transforms (SMTs)” lesson free?
Yes — the full text of “Single Message Transforms (SMTs)” is free to read here on the web, and the Apache Kafka & Stream Processing Fundamentals course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Apache Kafka & Stream Processing Fundamentals course, upgrade to CoddyKit PRO.
What will I learn in “Single Message Transforms (SMTs)”?
Learn how Kafka Connect Single Message Transforms reshape records inline between connectors and Kafka without writing a stream processor. You practise Apache Kafka & Stream Processing Fundamentals with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Apache Kafka & Stream Processing Fundamentals?
No prior experience is required. Apache Kafka & Stream Processing Fundamentals on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Single Message Transforms (SMTs)” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Apache Kafka & Stream Processing Fundamentals lesson?
Yes. Every Apache Kafka & Stream Processing Fundamentals lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Introduction to Kafka Connect
- Source Connectors for Ingestion
- Sink Connectors for Export
- Single Message Transforms (SMTs)