Setting up Local Kafka with Docker

When building applications that use Apache Kafka, you'll need a Kafka cluster to connect to. For development and testing, setting up a local cluster is ideal.

It provides a safe, isolated environment where you can experiment without affecting production systems. This lesson will guide you through setting it up easily with Docker.

Docker is a platform for developing, shipping, and running applications in containers. Containers are lightweight, portable, and self-sufficient units.

Docker Compose is a tool for defining and running multi-container Docker applications. You use a YAML file to configure your application's services, then create and start them all with a single command.

A minimal Kafka cluster requires two main components to function:

• Zookeeper: This service manages Kafka brokers, handling leader election, configuration, and coordination. Kafka relies on Zookeeper for its operational state.
• Kafka Broker: This is the core server that stores topics, handles message production and consumption, and replicates data. You'll typically run at least one broker.

We'll define our Kafka setup using a docker-compose.yml file. This file tells Docker Compose which services to run, which images to use, and how they should be configured and networked.

Each service in the file represents a container that will be launched. Let's start building it step-by-step.

First, let's define the Zookeeper service. We'll use a standard Zookeeper image and expose its default client port.

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

Here, ports: "2181:2181" maps the container's port 2181 to your host's port 2181.

Next, we add the Kafka broker service. It needs to know how to connect to Zookeeper and also exposes its own ports for clients.

  kafka:
    image: confluentinc/cp-kafka:7.5.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    depends_on:
      - zookeeper

depends_on: zookeeper ensures Zookeeper starts before Kafka.

Combining both services, your docker-compose.yml file looks like this:

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:7.5.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    depends_on:
      - zookeeper

Save this content into a file named docker-compose.yml in an empty directory.

Once your docker-compose.yml file is ready, navigate to its directory in your terminal. Then, use the following command to start your Kafka cluster:

docker-compose up -d

• up: Builds, (re)creates, starts, and attaches to containers for a service.
• -d: Runs containers in detached mode (in the background).

Docker will download images (if not present) and start Zookeeper, then Kafka.

After running docker-compose up -d, you can check if your services are running correctly:

• docker ps: Lists all running Docker containers. You should see zookeeper and kafka.
• docker logs kafka: Shows the logs for the Kafka container. Look for messages indicating it started successfully, like "started (kafka.server.KafkaServer)".

To stop and remove the containers, use docker-compose down.

You've just created a docker-compose.yml file and want to start your Kafka cluster in the background. Which command should you use?

Congratulations! You've learned how to set up a local Kafka cluster using Docker Compose. Here's a quick summary:

• Local Kafka provides an isolated dev environment.
• Docker Compose uses a docker-compose.yml file to define multi-container apps.
• A minimal Kafka setup requires Zookeeper and a Kafka Broker.
• The docker-compose.yml specifies images, ports, and environment variables.
• Use docker-compose up -d to start and docker-compose down to stop your cluster.

Now you have a sandbox to build your Spring Boot Kafka applications!