Pub/Sub, Streams, and Rate Limiting with Redis

Redis is more than a key/value cache. In a Node.js backend it doubles as a lightweight message broker and a coordination engine. In this lesson you will learn three production patterns that ship with every Redis install:

• Pub/Sub — fire-and-forget broadcast to all connected listeners.
• Streams — an append-only, durable log with consumer groups and replay.
• Rate limiting — atomic counters that throttle abusive clients.

The key idea that ties them together: a single round-trip to Redis can do work that would otherwise need a database, a queue server, and a lock service. We will use the ioredis client throughout, which is the de-facto standard for Node backends because it supports pipelining, Lua scripting, and cluster mode.

Redis Pub/Sub lets one process PUBLISH a message to a channel and have every SUBSCRIBEd client receive it instantly. It is perfect for cache-invalidation fan-out or pushing live updates to WebSocket servers.

The critical gotcha: a connection in subscribe mode cannot run normal commands. You must create a dedicated subscriber connection separate from the one you use for GET/SET. Below, sub only listens; a second client would be used to publish.

Beyond exact channels, Redis supports PSUBSCRIBE with glob patterns. Subscribing to order:* receives messages from order:created, order:shipped, and so on. The callback is pmessage and includes the matched pattern.

The crucial limitation to internalize: Pub/Sub has no persistence. If no subscriber is connected when you publish, the message is gone forever — there is no buffering, no acknowledgement, no replay. If a subscriber crashes and reconnects, it misses everything that happened while it was down.

• Use Pub/Sub for ephemeral events where a missed message is acceptable.
• For events that must not be lost, you need Streams (next section).

A Redis Stream is an append-only log. Each entry gets a monotonically increasing ID like 1718000000000-0 (milliseconds-sequence). Unlike Pub/Sub, entries are stored until you trim them, so late or restarted consumers can replay history.

You append with XADD. The special ID * tells Redis to auto-generate the next ID. After the ID you pass field/value pairs, exactly like a hash.

The real power of Streams is consumer groups. A group lets multiple worker processes share the load of one stream: each entry is delivered to exactly one consumer in the group, enabling horizontal scaling.

Create the group once with XGROUP CREATE. The MKSTREAM option creates the stream if it does not exist yet. The start ID $ means 'only deliver entries added after the group was created'; use 0 to consume from the very beginning.

Each worker reads with XREADGROUP, passing its group name and a unique consumer name. The special ID > means 'give me entries never delivered to any consumer in this group'.

An unacknowledged entry stays in the group's Pending Entries List (PEL). After you finish processing, you call XACK so Redis knows it is safely handled. If your worker crashes before XACK, the entry remains pending and can be reclaimed — this is what makes Streams at-least-once and durable.

What if worker-1 dies mid-processing? Its entries stay in the PEL forever unless reclaimed. Use XAUTOCLAIM (Redis 6.2+) to transfer entries idle longer than a threshold to a healthy consumer:

• XAUTOCLAIM stream group consumer min-idle-time start — grabs entries idle past min-idle-time ms.
• Inspect outstanding work with XPENDING before reclaiming.

Streams grow forever, so cap memory with capped XADD: XADD key MAXLEN ~ 10000 * .... The ~ means 'approximately', which lets Redis trim efficiently in whole macro-nodes instead of exact counts.

Switching gears to throttling. A first instinct is to GET a counter, check it in Node, then SET it back. This is a classic race condition: between your GET and SET, another concurrent request reads the same stale value, and both think they are under the limit. Under load you let through far more requests than allowed.

The fix is atomicity. Redis runs each command (and each Lua script) single-threaded and indivisibly. A simple fixed-window limiter uses INCR plus EXPIRE so the read-modify-write happens server-side with no gap. The pure-JS demo below shows the race conceptually before we move it into Redis.

The simplest correct limiter: key the counter by client + time window, INCR it, and set a TTL the first time the window opens. Because INCR returns the new value atomically, there is no race.

The weakness of fixed windows is boundary bursting: a client can send the full quota at 0:59 and again at 1:00, briefly doubling the rate. Acceptable for many APIs, but token buckets (next) smooth this out.

A token bucket gives each client a bucket of capacity C that refills at R tokens/second. Each request costs one token; if the bucket is empty, the request is rejected. It allows controlled bursts while enforcing a steady average rate.

Refill and consume must be one atomic step, so we ship them as a Lua script via EVAL. Redis runs the whole script without interleaving other commands. We store two fields in a hash — current tokens and last refill ts — and lazily compute how many tokens regenerated since the last call.

In a real Node backend the limiter lives in middleware so every route is protected. On rejection you return HTTP 429 and, as a courtesy, a Retry-After header telling the client when to try again.

Best practices to remember:

• Key by a stable identity (API key or authenticated user id), not just IP — IPs are shared behind NAT.
• Fail open or closed deliberately: if Redis is unreachable, decide whether to allow traffic (availability) or block it (protection). Make it an explicit choice, not an accident.
• Return rate-limit headers so well-behaved clients can self-throttle.

You are building an order-processing pipeline. Worker processes can crash and restart, and no order event may ever be lost; each event must be processed by exactly one of several workers, with crashed work automatically retried. Which Redis feature fits?

You now have three Redis messaging and control patterns and, more importantly, the judgment to choose between them:

• Pub/Sub — instant fan-out, but ephemeral. Use a dedicated subscriber connection. Great for cache invalidation and live notifications where a missed message is harmless.
• Streams — durable, replayable log. Consumer groups give exactly-one-of-N delivery; XACK plus the Pending Entries List and XAUTOCLAIM give at-least-once processing with crash recovery. Cap growth with MAXLEN ~.
• Rate limiting — never check-then-set in app code; that races. Use atomic INCR+EXPIRE for fixed windows, or an EVAL Lua token bucket for smooth bursts. Wrap it in middleware, return 429 with Retry-After, and decide consciously whether to fail open or closed.

The unifying principle: let Redis do the atomic work in a single round-trip, and choose the primitive by the durability guarantee your use case actually requires.