Node.js Backend Development Bootcamp · 课时

使用 Redis 实现发布/订阅、流与速率限制

广播事件、构建持久化 Redis Streams，并原子化实现令牌桶速率限制器。

使用 Redis 实现发布/订阅、流与速率限制 是 CoddyKit 上的免费 Node.js Backend Development Bootcamp 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Node.js Backend Development Bootcamp 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Node.js Backend Development Bootcamp 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Three Messaging Primitives in One Server

Redis is more than a key/value cache. In a Node.js backend it doubles as a lightweight message broker and a coordination engine. In this lesson you will learn three production patterns that ship with every Redis install:

Pub/Sub — fire-and-forget broadcast to all connected listeners.
Streams — an append-only, durable log with consumer groups and replay.
Rate limiting — atomic counters that throttle abusive clients.

The key idea that ties them together: a single round-trip to Redis can do work that would otherwise need a database, a queue server, and a lock service. We will use the ioredis client throughout, which is the de-facto standard for Node backends because it supports pipelining, Lua scripting, and cluster mode.

Pub/Sub: Broadcast to Every Subscriber

Redis Pub/Sub lets one process PUBLISH a message to a channel and have every SUBSCRIBEd client receive it instantly. It is perfect for cache-invalidation fan-out or pushing live updates to WebSocket servers.

The critical gotcha: a connection in subscribe mode cannot run normal commands. You must create a dedicated subscriber connection separate from the one you use for GET/SET. Below, sub only listens; a second client would be used to publish.

const Redis = require('ioredis');

const sub = new Redis();   // dedicated subscriber connection
const pub = new Redis();   // separate connection for publishing

sub.subscribe('cache:invalidate', (err, count) => {
  if (err) throw err;
  console.log('Subscribed to ' + count + ' channel(s)');
});

sub.on('message', (channel, message) => {
  console.log('[' + channel + '] ' + message);
});

// Another part of the app publishes an event
pub.publish('cache:invalidate', JSON.stringify({ key: 'user:42' }));

Pattern Subscriptions and the Fire-and-Forget Caveat

Beyond exact channels, Redis supports PSUBSCRIBE with glob patterns. Subscribing to order:* receives messages from order:created, order:shipped, and so on. The callback is pmessage and includes the matched pattern.

The crucial limitation to internalize: Pub/Sub has no persistence. If no subscriber is connected when you publish, the message is gone forever — there is no buffering, no acknowledgement, no replay. If a subscriber crashes and reconnects, it misses everything that happened while it was down.

Use Pub/Sub for ephemeral events where a missed message is acceptable.
For events that must not be lost, you need Streams (next section).

const Redis = require('ioredis');
const sub = new Redis();

sub.psubscribe('order:*', (err, count) => {
  console.log('Pattern subscriptions: ' + count);
});

sub.on('pmessage', (pattern, channel, message) => {
  console.log('matched ' + pattern + ' on ' + channel + ': ' + message);
});

Streams: A Durable, Replayable Log

A Redis Stream is an append-only log. Each entry gets a monotonically increasing ID like 1718000000000-0 (milliseconds-sequence). Unlike Pub/Sub, entries are stored until you trim them, so late or restarted consumers can replay history.

You append with XADD. The special ID * tells Redis to auto-generate the next ID. After the ID you pass field/value pairs, exactly like a hash.

const Redis = require('ioredis');
const redis = new Redis();

async function publishOrder() {
  // XADD key * field value field value ...
  const id = await redis.xadd(
    'stream:orders', '*',
    'orderId', '42',
    'amount', '99.90',
    'status', 'created'
  );
  console.log('Appended entry with ID ' + id);

  // Read the latest 5 entries (newest last)
  const entries = await redis.xrange('stream:orders', '-', '+', 'COUNT', 5);
  console.log(JSON.stringify(entries, null, 2));
}

publishOrder();

Consumer Groups: Scale Out Without Double-Processing

The real power of Streams is consumer groups. A group lets multiple worker processes share the load of one stream: each entry is delivered to exactly one consumer in the group, enabling horizontal scaling.

Create the group once with XGROUP CREATE. The MKSTREAM option creates the stream if it does not exist yet. The start ID $ means 'only deliver entries added after the group was created'; use 0 to consume from the very beginning.

const Redis = require('ioredis');
const redis = new Redis();

async function setup() {
  try {
    await redis.xgroup(
      'CREATE', 'stream:orders', 'order-workers', '$', 'MKSTREAM'
    );
    console.log('Group created');
  } catch (e) {
    if (e.message.includes('BUSYGROUP')) {
      console.log('Group already exists, continuing');
    } else {
      throw e;
    }
  }
}

setup();

Reading and Acknowledging Stream Entries

Each worker reads with XREADGROUP, passing its group name and a unique consumer name. The special ID > means 'give me entries never delivered to any consumer in this group'.

An unacknowledged entry stays in the group's Pending Entries List (PEL). After you finish processing, you call XACK so Redis knows it is safely handled. If your worker crashes before XACK, the entry remains pending and can be reclaimed — this is what makes Streams at-least-once and durable.

const Redis = require('ioredis');
const redis = new Redis();

async function consume() {
  const res = await redis.xreadgroup(
    'GROUP', 'order-workers', 'worker-1',
    'COUNT', 10, 'BLOCK', 5000,
    'STREAMS', 'stream:orders', '>'
  );
  if (!res) return; // BLOCK timed out with no new entries

  for (const [, entries] of res) {
    for (const [id, fields] of entries) {
      console.log('processing ' + id, fields);
      // ... do real work here ...
      await redis.xack('stream:orders', 'order-workers', id);
    }
  }
}

consume();

Reclaiming Stuck Messages and Trimming

What if worker-1 dies mid-processing? Its entries stay in the PEL forever unless reclaimed. Use XAUTOCLAIM (Redis 6.2+) to transfer entries idle longer than a threshold to a healthy consumer:

XAUTOCLAIM stream group consumer min-idle-time start — grabs entries idle past min-idle-time ms.
Inspect outstanding work with XPENDING before reclaiming.

Streams grow forever, so cap memory with capped XADD: XADD key MAXLEN ~ 10000 * .... The ~ means 'approximately', which lets Redis trim efficiently in whole macro-nodes instead of exact counts.

const Redis = require('ioredis');
const redis = new Redis();

async function recover() {
  // Reclaim entries idle > 30s, hand them to worker-2
  const [cursor, claimed] = await redis.xautoclaim(
    'stream:orders', 'order-workers', 'worker-2',
    30000, '0', 'COUNT', 25
  );
  console.log('Reclaimed ' + claimed.length + ' entries; next cursor ' + cursor);

  // Append with an approximate cap to bound memory
  await redis.xadd('stream:orders', 'MAXLEN', '~', 10000, '*', 'orderId', '99');
}

recover();

Why Naive Rate Limiting Breaks

Switching gears to throttling. A first instinct is to GET a counter, check it in Node, then SET it back. This is a classic race condition: between your GET and SET, another concurrent request reads the same stale value, and both think they are under the limit. Under load you let through far more requests than allowed.

The fix is atomicity. Redis runs each command (and each Lua script) single-threaded and indivisibly. A simple fixed-window limiter uses INCR plus EXPIRE so the read-modify-write happens server-side with no gap. The pure-JS demo below shows the race conceptually before we move it into Redis.

// Demonstrates WHY check-then-set races. Two 'requests' interleave.
let counter = 0;
const LIMIT = 3;

function tryRequest(name) {
  const current = counter;        // read
  if (current < LIMIT) {
    // imagine an await here: another request runs before we write
    counter = current + 1;        // write (stale!)
    return name + ': allowed (' + counter + ')';
  }
  return name + ': blocked';
}

// Both read 0 before either writes -> over-admission
const a = tryRequest('reqA');
const b = tryRequest('reqB');
console.log(a);
console.log(b);
console.log('Final counter: ' + counter);

Fixed-Window Limiter with INCR + EXPIRE

The simplest correct limiter: key the counter by client + time window, INCR it, and set a TTL the first time the window opens. Because INCR returns the new value atomically, there is no race.

The weakness of fixed windows is boundary bursting: a client can send the full quota at 0:59 and again at 1:00, briefly doubling the rate. Acceptable for many APIs, but token buckets (next) smooth this out.

const Redis = require('ioredis');
const redis = new Redis();

async function allow(userId, limit = 100, windowSec = 60) {
  const key = 'rl:' + userId + ':' + Math.floor(Date.now() / 1000 / windowSec);
  const count = await redis.incr(key);
  if (count === 1) {
    await redis.expire(key, windowSec); // set TTL only on first hit
  }
  return count <= limit;
}

allow('user:42').then((ok) => {
  console.log(ok ? 'request allowed' : '429 Too Many Requests');
});

Atomic Token Bucket with a Lua Script

A token bucket gives each client a bucket of capacity C that refills at R tokens/second. Each request costs one token; if the bucket is empty, the request is rejected. It allows controlled bursts while enforcing a steady average rate.

Refill and consume must be one atomic step, so we ship them as a Lua script via EVAL. Redis runs the whole script without interleaving other commands. We store two fields in a hash — current tokens and last refill ts — and lazily compute how many tokens regenerated since the last call.

const Redis = require('ioredis');
const redis = new Redis();

const LUA = [
  "local cap = tonumber(ARGV[1])",
  "local refill = tonumber(ARGV[2])",
  "local now = tonumber(ARGV[3])",
  "local cost = tonumber(ARGV[4])",
  "local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')",
  "local tokens = tonumber(b[1])",
  "local ts = tonumber(b[2])",
  "if tokens == nil then tokens = cap; ts = now end",
  "local delta = math.max(0, now - ts)",
  "tokens = math.min(cap, tokens + delta * refill)",
  "local allowed = 0",
  "if tokens >= cost then allowed = 1; tokens = tokens - cost end",
  "redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now)",
  "redis.call('EXPIRE', KEYS[1], 3600)",
  "return allowed"
].join('\n');

async function take(userId) {
  const now = Date.now() / 1000;
  // capacity 10, refill 1 token/sec, cost 1
  const allowed = await redis.eval(LUA, 1, 'tb:' + userId, 10, 1, now, 1);
  return allowed === 1;
}

take('user:42').then((ok) => console.log(ok ? 'allowed' : 'throttled'));

Wiring the Limiter into Express Middleware

In a real Node backend the limiter lives in middleware so every route is protected. On rejection you return HTTP 429 and, as a courtesy, a Retry-After header telling the client when to try again.

Best practices to remember:

Key by a stable identity (API key or authenticated user id), not just IP — IPs are shared behind NAT.
Fail open or closed deliberately: if Redis is unreachable, decide whether to allow traffic (availability) or block it (protection). Make it an explicit choice, not an accident.
Return rate-limit headers so well-behaved clients can self-throttle.

function rateLimit(redis, take) {
  return async (req, res, next) => {
    const id = req.user?.id || req.ip;
    try {
      if (await take(id)) return next();
      res.set('Retry-After', '1');
      return res.status(429).json({ error: 'Too Many Requests' });
    } catch (err) {
      // Redis down: fail OPEN here (prioritize availability)
      console.error('rate limiter degraded', err.message);
      return next();
    }
  };
}

module.exports = { rateLimit };

Quick Check: Choosing the Right Primitive

You are building an order-processing pipeline. Worker processes can crash and restart, and no order event may ever be lost; each event must be processed by exactly one of several workers, with crashed work automatically retried. Which Redis feature fits?

Recap: Pick the Tool That Matches the Guarantee

You now have three Redis messaging and control patterns and, more importantly, the judgment to choose between them:

Pub/Sub — instant fan-out, but ephemeral. Use a dedicated subscriber connection. Great for cache invalidation and live notifications where a missed message is harmless.
Streams — durable, replayable log. Consumer groups give exactly-one-of-N delivery; XACK plus the Pending Entries List and XAUTOCLAIM give at-least-once processing with crash recovery. Cap growth with MAXLEN ~.
Rate limiting — never check-then-set in app code; that races. Use atomic INCR+EXPIRE for fixed windows, or an EVAL Lua token bucket for smooth bursts. Wrap it in middleware, return 429 with Retry-After, and decide consciously whether to fail open or closed.

The unifying principle: let Redis do the atomic work in a single round-trip, and choose the primitive by the durability guarantee your use case actually requires.

常见问题解答

「使用 Redis 实现发布/订阅、流与速率限制」课时是免费的吗？

是的 — 「使用 Redis 实现发布/订阅、流与速率限制」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Node.js Backend Development Bootcamp 课程的其余内容，请升级到 CoddyKit PRO。 Node.js Backend Development Bootcamp 课程共包含 4 节课。

「使用 Redis 实现发布/订阅、流与速率限制」这节课中我会学到什么？

广播事件、构建持久化 Redis Streams，并原子化实现令牌桶速率限制器。你通过在浏览器中直接运行的动手代码来练习 Node.js Backend Development Bootcamp，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Node.js Backend Development Bootcamp 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Node.js Backend Development Bootcamp 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「使用 Redis 实现发布/订阅、流与速率限制」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Node.js Backend Development Bootcamp 课中编写并运行代码吗？

能。每节 Node.js Backend Development Bootcamp 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Node.js Backend Development Bootcamp