Distributed Locks and the Redlock Algorithm

When your Node.js API runs as a single process, a simple in-memory mutex is enough to serialize access to a critical section. But production backends run many instances behind a load balancer, often across multiple machines.

• Two pods may both try to charge the same invoice.
• Two workers may both pick up the same job from a queue.
• Two requests may both regenerate the same expensive cache entry (a cache stampede).

An in-memory lock lives inside one process only — the other instances know nothing about it. To coordinate exclusive access across instances you need a lock that lives in shared external storage, and Redis is a popular home for it.

The core primitive is the atomic SET key value NX PX ttl command. NX means "only set if the key does not exist", and PX sets an expiry in milliseconds so the lock auto-releases if the holder crashes.

• If SET returns OK, you acquired the lock.
• If it returns null, someone else holds it.

The value must be a unique random token per acquisition — you will need it later to release safely.

Releasing is the dangerous part. A naive DEL key can delete someone else's lock: if your work overran the TTL, the lock expired, another instance acquired it, and then your late DEL wipes their lock.

The fix: only delete if the stored value still equals your token. Check-then-delete must be atomic, so we run it as a Lua script — Redis executes scripts without interleaving other commands.

The TTL is a bet on how long your critical section takes.

• Too short and the lock expires mid-work, allowing a second worker in — your mutual exclusion is broken.
• Too long and if a holder crashes, everyone waits the full TTL before anyone can proceed.

Rules of thumb: set the TTL to a few times your p99 critical-section duration, keep the protected work short, and for long tasks use a watchdog that periodically extends the lock instead of a single huge TTL.

For work whose duration is uncertain, acquire a modest TTL and renew it on a timer while you still hold the lock. Like release, extension must be guarded by your token so you never extend a lock that has already moved on.

The watchdog runs at roughly one third of the TTL, giving you margin against clock jitter and GC pauses.

Everything so far assumes one Redis node. That node is a single point of failure, so teams add a replica with failover. But Redis replication is asynchronous, and that quietly breaks mutual exclusion:

• Client A acquires the lock on the master.
• The master crashes before replicating the write to the replica.
• The replica is promoted; it has no record of the lock.
• Client B acquires "the same" lock on the new master.

Now two clients hold the lock simultaneously. The Redlock algorithm was designed to address this failover window.

Redlock uses N independent Redis masters (typically 5), with no replication between them. To acquire a lock a client:

• Records the start time, then tries to SET NX PX the same key+token on all N nodes, using a short per-node timeout.
• Counts successes. The lock is considered acquired only if it got a quorum (N/2 + 1, i.e. 3 of 5) and the total elapsed time is less than the TTL.
• The effective validity = TTL minus elapsed time minus a clock-drift allowance.

If it fails to reach quorum (or runs out of time), it unlocks all nodes and retries after a small random delay.

You rarely implement Redlock by hand. The redlock npm package takes an array of independent Redis clients and exposes using(), which acquires, auto-extends, and releases the lock around your callback.

• retryCount / retryDelay control how hard it tries before giving up.
• using() hands you a signal — check signal.aborted to detect that the lock was lost mid-work.

Martin Kleppmann's well-known critique: no lock based on timeouts can guarantee safety if a holder is paused (GC, VM stall) past the TTL. The lock expires, another client proceeds, and the paused client wakes up still believing it holds the lock.

The robust defense is a fencing token: a monotonically increasing number issued with each lock grant. The protected resource itself rejects any write carrying a token lower than the highest it has already seen — so a stale, paused writer is fenced out at the destination.

A lock reduces the chance of concurrent execution, but timeouts and failovers mean you can never make it a hard guarantee. Treat the lock as an optimization, not the last line of defense.

• Make the protected operation idempotent — running it twice produces the same result.
• Use database unique constraints or conditional updates (compare-and-set) so a duplicate write fails loudly.
• Use fencing tokens where the resource supports them.

Best practice: lock to avoid wasted work and contention, but design the system so a rare double-execution is still correct.

Redlock adds operational cost: five independent Redis deployments, careful clock management, and retry tuning. The redis maintainers themselves note that for efficiency use cases (avoid doing the same work twice), a single-node lock is fine — an occasional double-run just wastes a little work.

• Efficiency lock (cache rebuild, dedup): single Redis SET NX PX is enough.
• Correctness lock (money, inventory): don't rely on any timeout lock alone — add idempotency and fencing, regardless of Redlock.

Reach for Redlock only when single-node HA failover is genuinely unacceptable and you cannot fence at the resource.

Test your understanding of distributed locking safety.

You learned how to coordinate exclusive access across Node.js instances — and where the limits are.

• Acquire with atomic SET key token NX PX ttl; the token must be unique per acquisition.
• Release and extend only via a token-checked Lua script so you never touch someone else's lock; renew long tasks with a watchdog.
• TTL is a tradeoff: too short breaks exclusion, too long delays recovery after a crash.
• Redlock uses a quorum across N independent masters to survive single-node failover, but it is still timeout-based.
• No timeout lock is safe against long pauses — add fencing tokens and idempotency for correctness-critical work.
• Use a single-node lock for efficiency; reserve Redlock for genuine HA needs.