Command-Line Tools for Kafka

Welcome! Kafka command-line interface (CLI) tools are essential for managing your Kafka cluster. They allow you to interact directly with brokers, topics, and consumer groups.

Mastering these tools is a crucial skill for any Kafka administrator or developer, enabling quick setup, debugging, and monitoring.

Kafka's CLI tools are shell scripts located in the bin/ directory of your Kafka installation. You typically execute them directly from your terminal.

Most commands follow a similar structure: bin/kafka-tool.sh --bootstrap-server <broker_address> --<action> [options].

The kafka-topics.sh tool is your primary utility for managing topics. Let's start by listing all topics currently available in your Kafka cluster.

The --list option does exactly that, showing you an overview of your data streams.

To store data, you need to create a topic. When creating one, you must specify its name, the number of partitions, and the replication factor.

• Partitions: How many segments a topic is divided into. More partitions allow for greater parallelism.
• Replication Factor: The number of copies of each partition kept across different brokers for fault tolerance.

Once a topic is created, you might want to check its configuration and status. The --describe option provides detailed information, including partition assignments, leader, and replica status.

You can remove topics that are no longer needed. Be cautious, as deleting a topic will permanently remove all its data!

For topic deletion to work, ensure delete.topic.enable=true is set in your Kafka broker's configuration.

The kafka-console-producer.sh tool lets you send messages directly from your terminal. It's perfect for quick tests or manually injecting data into a topic.

After running the command, simply type your message and press Enter to send it.

To read messages from a topic, use kafka-console-consumer.sh. You can specify whether to read from the beginning of the topic or only new messages that arrive after the consumer starts.

Consumers typically operate within a "consumer group" to enable distributed message processing. The kafka-consumer-groups.sh tool is vital for inspecting and managing these groups.

You can use it to list all active consumer groups and check their offsets.

You need to view the detailed configuration, including partitions, replication factor, leader, and replica assignments, for a Kafka topic named app_events. Which command-line option for kafka-topics.sh would you use?

You've learned how Kafka's command-line tools are indispensable for daily administration and quick interactions with your cluster. We covered:

• Listing, creating, describing, and deleting topics using kafka-topics.sh.
• Sending messages with kafka-console-producer.sh.
• Receiving messages with kafka-console-consumer.sh.
• Inspecting consumer groups with kafka-consumer-groups.sh.

These powerful tools are your first line of interaction for managing and monitoring Kafka!