Redis Caching & Messaging (Pub/Sub, Streams) · Lesson

Redis Cluster for Sharding

Implement Redis Cluster to shard data across multiple nodes, achieving horizontal scaling and fault tolerance.

Redis Cluster for Sharding is a free Redis Caching & Messaging (Pub/Sub, Streams) lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Redis Caching & Messaging (Pub/Sub, Streams) learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Scaling Beyond a Single Node

So far, we've learned about Redis replication and Sentinel for high availability. These are great for redundancy and read scaling, but what about write scaling or handling datasets larger than a single server's memory?

This is where Redis Cluster comes in. It's designed to provide horizontal scaling and fault tolerance by distributing your data across multiple Redis instances.

What is Sharding?

At its core, Redis Cluster uses sharding (also known as data partitioning). Imagine you have a massive library of books.

Instead of one giant shelf, you split the books across many smaller shelves.
Each shelf holds a portion of the books, and a librarian manages that shelf.

In Redis Cluster, each Redis instance (or node) acts like a 'librarian' managing a 'shelf' of your data. This allows you to scale both memory and CPU.

Redis Cluster Architecture

A Redis Cluster is a collection of interconnected Redis instances. Each instance can be a master or a replica.

Master Nodes: These nodes hold and manage a portion of the dataset.
Replica Nodes: These are copies of master nodes, providing high availability. If a master fails, one of its replicas can be promoted to take its place.

The cluster ensures data is distributed, and it can continue operating even if some nodes fail.

Hash Slots: How Data is Distributed

How does Redis know which node stores which piece of data? It uses hash slots.

The entire dataset is divided into 16,384 logical slots (from 0 to 16383).
Each master node in the cluster is responsible for a subset of these hash slots.
When you store a key, Redis calculates a hash of the key to determine which slot it belongs to.
This slot then maps to a specific master node.

This system allows for flexible data distribution and easy rebalancing.

Creating a Redis Cluster

To create a cluster, you need at least three master nodes for a fault-tolerant setup (each master can have replicas). Here's a conceptual CLI command using redis-cli:

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 --cluster-replicas 1

This command creates a cluster with 3 masters (on ports 7000, 7001, 7002).
--cluster-replicas 1 means each master will have one replica node.

The redis-cli tool automatically assigns hash slots to the master nodes.

Client Interaction & Redirection

When a client wants to interact with Redis Cluster, it can connect to any node. If the key the client requests doesn't belong to the connected node's hash slots, the node will respond with a MOVED redirection error.

Modern Redis client libraries (like Jedis for Java) are cluster-aware. They automatically handle these redirections by:

Learning the cluster topology (which node owns which hash slots).
Directly connecting to the correct node for a given key.

This makes interacting with a cluster surprisingly seamless for developers.

Connecting with a Java Client

Let's see how a Java client (using the Jedis library) connects to a Redis Cluster. The client needs to know at least one node in the cluster to discover the full topology.

Try running this example:

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
import java.util.HashSet;
import java.util.Set;

public class RedisClusterClient {
    public static void main(String[] args) {
        // Provide at least one node to connect to the cluster
        Set<HostAndPort> jedisClusterNodes = new HashSet<>();
        jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7000)); // Example node

        try (JedisCluster jc = new JedisCluster(jedisClusterNodes)) {
            System.out.println("Connected to Redis Cluster!");

            String key = "user:100:name";
            String value = "Alice";
            jc.set(key, value);
            System.out.println("Set: " + key + " = " + value);

            String retrievedValue = jc.get(key);
            System.out.println("Get: " + key + " = " + retrievedValue);

            // Client automatically handles sharding for different keys
            jc.set("product:5:price", "99.99");
            System.out.println("Set: product:5:price = " + jc.get("product:5:price"));

        } catch (Exception e) {
            System.err.println("Error connecting to Redis Cluster: " + e.getMessage());
            System.err.println("Ensure a Redis Cluster is running on 127.0.0.1:7000");
        }
    }
}

Resharding & Rebalancing

One of the powerful features of Redis Cluster is its ability to reshard. This means you can dynamically add or remove nodes from the cluster while it's running.

When you add new master nodes, you can migrate hash slots from existing masters to the new ones.
When you remove nodes, their hash slots can be moved to other remaining masters.

This allows for flexible scaling up or down of your cluster resources without downtime.

Fault Tolerance in Cluster

Redis Cluster provides robust fault tolerance:

If a master node fails, its assigned replica is automatically promoted to become the new master.
The cluster continues to operate, as the hash slots previously managed by the failed master are now handled by its promoted replica.

However, if a master and all its replicas fail, the portion of the data managed by that master's hash slots becomes unavailable, and the cluster might cease to operate unless configured otherwise (cluster-require-full-coverage no).

Check Your Cluster Knowledge

Which of the following are primary benefits or characteristics of Redis Cluster?

Recap: Redis Cluster for Scale

In this lesson, we explored Redis Cluster, a powerful solution for scaling Redis horizontally. We learned:

Cluster uses sharding to distribute data across multiple master nodes.
Hash slots determine which node stores which key.
Client libraries are cluster-aware, handling redirections automatically.
It offers fault tolerance by promoting replicas upon master failure.
The ability to reshard allows for dynamic scaling.

Redis Cluster is your go-to when a single Redis instance can no longer meet your application's data size or throughput demands.

Frequently asked questions

Is the “Redis Cluster for Sharding” lesson free?

Yes — the full text of “Redis Cluster for Sharding” is free to read here on the web, and the Redis Caching & Messaging (Pub/Sub, Streams) course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Redis Caching & Messaging (Pub/Sub, Streams) course, upgrade to CoddyKit PRO.

What will I learn in “Redis Cluster for Sharding”?

Implement Redis Cluster to shard data across multiple nodes, achieving horizontal scaling and fault tolerance. You practise Redis Caching & Messaging (Pub/Sub, Streams) with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Redis Caching & Messaging (Pub/Sub, Streams)?

No prior experience is required. Redis Caching & Messaging (Pub/Sub, Streams) on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Redis Cluster for Sharding” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Redis Caching & Messaging (Pub/Sub, Streams) lesson?

Yes. Every Redis Caching & Messaging (Pub/Sub, Streams) lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Redis Caching & Messaging (Pub/Sub, Streams)