Backpressure, pipe() and the pipeline() Utility
Diagnose memory bloat and wire streams safely with pipeline() to propagate errors and honor backpressure.
Why Memory Bloats in Streams
A Node.js Readable stream can produce data faster than a Writable can consume it. If you never tell the producer to slow down, unconsumed chunks pile up in an internal buffer and your process memory grows until the GC can't keep up.
- A slow disk, slow network socket, or slow database write is the typical consumer.
- A fast file read or HTTP upload is the typical producer.
The mechanism that makes the producer wait for the consumer is called backpressure. Misusing streams almost always means backpressure was ignored.
The Naive (Broken) Copy
Here is the classic memory bug. We listen for data and call dst.write() for every chunk, ignoring its return value.
If dst is slower than src, the unwritten chunks queue up inside dst's buffer with no upper bound. For a multi-gigabyte file this can exhaust RAM.
const fs = require('fs');
const src = fs.createReadStream('big.bin');
const dst = fs.createWriteStream('copy.bin');
// BUG: return value of write() is ignored, so backpressure is never honored
src.on('data', (chunk) => {
dst.write(chunk);
});
src.on('end', () => dst.end());All lessons in this course
- Readable, Writable, Duplex & Transform Stream Internals
- Implementing Custom Transform Streams with _transform and _flush
- Backpressure, pipe() and the pipeline() Utility
- Async Iterators and for-await-of Over Streams