Apache Kafka with PHP
Stream high-throughput events using Kafka.
Apache Kafka with PHP is a free PHP Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the PHP Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Kafka with PHP
Kafka is not a task queue — it is a distributed, append-only commit log. Producers append records to topics; consumers read at their own offset and can replay history. This makes Kafka ideal for high-throughput event streams, event sourcing, and feeding multiple independent consumer groups from one stream.
In PHP you talk to Kafka through ext-rdkafka, a binding over the battle-tested C library librdkafka.
Log, Not Queue
The mental shift from RabbitMQ to Kafka:
- Messages are not deleted when consumed; they expire by retention policy (time or size).
- Each consumer tracks its own offset — its position in the log.
- A topic is split into partitions; ordering is guaranteed only within a partition.
- Multiple consumer groups read the same topic independently.
If you need replay, fan-out to many readers, or massive throughput, Kafka fits. If you need per-message routing and TTLs, RabbitMQ fits.
Installing ext-rdkafka
Install the native library first, then the PECL extension, then optionally a higher-level wrapper.
# Debian/Ubuntu
apt-get install -y librdkafka-dev
pecl install rdkafka
echo "extension=rdkafka.so" >> php.ini
# Optional ergonomic wrapper
composer require longlang/phpkafka # or arnaud-lb/php-rdkafka-stubs for IDEProducing Records
Create a RdKafka\Producer, get a topic handle, and produce(). The key controls which partition a record lands in — same key, same partition, preserved order. Always flush() before exit or you lose buffered records.
<?php
$conf = new RdKafka\Conf();
$conf->set('bootstrap.servers', 'localhost:9092');
$producer = new RdKafka\Producer($conf);
$topic = $producer->newTopic('orders');
// RD_KAFKA_PARTITION_UA = let the partitioner choose by key
$key = 'order-42';
$topic->produce(RD_KAFKA_PARTITION_UA, 0, json_encode(['id' => 42]), $key);
$producer->poll(0);
$result = $producer->flush(10000); // wait up to 10s
if ($result !== RD_KAFKA_RESP_ERR_NO_ERROR) {
throw new RuntimeException('Failed to flush');
}Partitioning by Key
Partitioning is the heart of Kafka's scalability and ordering. The default partitioner hashes the record key: partition = hash(key) % numPartitions. Choosing a good key matters:
- Key by
customerId→ all of a customer's events stay ordered and on one partition. - Null key → round-robin across partitions (max throughput, no ordering).
You can never reduce a topic's partition count, and adding partitions reshuffles the hash mapping — so size partitions for peak parallelism up front.
Consumer Groups & Offsets
Use the high-level KafkaConsumer with group.id. Kafka assigns partitions across the group's members and rebalances when members join or leave. Each member reads only its assigned partitions, giving you horizontal scaling for free.
<?php
$conf = new RdKafka\Conf();
$conf->set('bootstrap.servers', 'localhost:9092');
$conf->set('group.id', 'order-emailers');
$conf->set('auto.offset.reset', 'earliest'); // start of log if no offset
$conf->set('enable.auto.commit', 'false'); // we commit manually
$consumer = new RdKafka\KafkaConsumer($conf);
$consumer->subscribe(['orders']);
while (true) {
$msg = $consumer->consume(5000);
if ($msg->err === RD_KAFKA_RESP_ERR_NO_ERROR) {
handle($msg->payload);
$consumer->commit($msg); // commit offset AFTER work
}
}
function handle(string $p): void {}When to Commit
The offset commit defines your delivery semantics:
- Commit after processing → at-least-once (a crash before commit replays the record).
- Commit before processing → at-most-once (a crash loses it).
Auto-commit (enable.auto.commit=true) commits on a timer regardless of whether your work finished — convenient but it can silently drop records on crash. Disable it and commit manually when correctness matters.
Handling Consume Errors
Not every return from consume() is a message. You must branch on the error code — __PARTITION_EOF and __TIMED_OUT are normal control signals, not failures.
<?php
$msg = $consumer->consume(2000);
switch ($msg->err) {
case RD_KAFKA_RESP_ERR_NO_ERROR:
echo "Got: {$msg->payload} @ offset {$msg->offset}\n";
break;
case RD_KAFKA_RESP_ERR__PARTITION_EOF:
echo "Reached end of partition\n"; // caught up, keep polling
break;
case RD_KAFKA_RESP_ERR__TIMED_OUT:
echo "No message this poll\n";
break;
default:
throw new \Exception($msg->errstr(), $msg->err);
}Throughput Tuning
Kafka's throughput comes from batching. Key producer settings:
linger.ms— wait briefly to batch more records per request (e.g. 5–20ms).batch.size/queue.buffering.max.messages— bigger buffers, fewer round-trips.compression.type—lz4orzstdcuts network cost dramatically.acks—allfor durability,1for lower latency.
<?php
$conf = new RdKafka\Conf();
$conf->set('bootstrap.servers', 'localhost:9092');
$conf->set('compression.type', 'lz4');
$conf->set('linger.ms', '10');
$conf->set('batch.size', '65536');
$conf->set('acks', 'all');
$producer = new RdKafka\Producer($conf);Delivery Report Callbacks
Because produce() is asynchronous, a failed send won't throw on the spot. Register a delivery-report callback on the producer config to learn the fate of each record — this is the only reliable way to detect silent producer failures in PHP.
<?php
$conf = new RdKafka\Conf();
$conf->set('bootstrap.servers', 'localhost:9092');
$conf->setDrMsgCb(function ($producer, $msg) {
if ($msg->err) {
fwrite(STDERR, 'Delivery FAILED: ' . rd_kafka_err2str($msg->err) . "\n");
} else {
echo "Delivered to partition {$msg->partition} @ offset {$msg->offset}\n";
}
});
$producer = new RdKafka\Producer($conf);
// poll() services the callback queue; call it after producing
$producer->poll(0);PHP-Specific Gotchas
Kafka assumes long-lived clients; PHP's request lifecycle fights that:
- Producers buffer asynchronously — always
flush()before the script ends or records are lost. - Run consumers as persistent CLI workers under a supervisor, never inside a web request.
- Rebalances pause consumption; keep your per-message work short or use
max.poll.interval.msgenerously so you aren't kicked from the group. - Set
log_leveland register the producer's delivery-report callback to catch silent failures.
Quick Check
Ordering guarantees in Kafka.
Recap
Kafka, the PHP way:
- Kafka is a replayable log, not a queue; consumers track offsets.
- Partitioning by key gives per-key ordering and scaling.
- Consumer groups split partitions and rebalance automatically.
- Commit offsets after work for at-least-once; disable auto-commit for control.
- Tune
linger.ms,batch.size, andcompression.type; alwaysflush().
Next: wiring these primitives into reliable event-driven workflows.
Frequently asked questions
Is the “Apache Kafka with PHP” lesson free?
Yes — the full text of “Apache Kafka with PHP” is free to read here on the web, and the PHP Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the PHP Academy course, upgrade to CoddyKit PRO.
What will I learn in “Apache Kafka with PHP”?
Stream high-throughput events using Kafka. You practise PHP Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start PHP Academy?
No prior experience is required. PHP Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Apache Kafka with PHP” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this PHP Academy lesson?
Yes. Every PHP Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.