Readable, Writable, Duplex & Transform Stream Internals
Understand the four stream types and how the internal buffer and highWaterMark govern their behavior.
Why Streams Exist
Node.js streams let you process data piece by piece instead of loading everything into memory at once. This is essential for backend work like serving large files, proxying HTTP bodies, or piping database exports.
- Readable — a source you read FROM (file read, HTTP request)
- Writable — a sink you write TO (file write, HTTP response)
- Duplex — both readable and writable, independent channels (TCP socket)
- Transform — a Duplex where the output is a function of the input (gzip, encryption)
Every one of these is backed by an internal buffer governed by a single number: highWaterMark.
The Internal Buffer & highWaterMark
Each stream keeps an internal buffer in its _readableState or _writableState. The highWaterMark (HWM) is the threshold, not a hard limit, at which the stream signals it has buffered "enough".
- Default HWM for byte streams is 16 KB (16384 bytes).
- In object mode the HWM counts objects, defaulting to 16.
When a Readable's buffer fills to the HWM it stops pulling from the source. When a Writable's buffer exceeds the HWM, write() returns false — the signal known as backpressure.
const fs = require('fs');
const rs = fs.createReadStream('/etc/hostname', { highWaterMark: 4 });
console.log('configured HWM:', rs.readableHighWaterMark);
rs.on('data', (chunk) => {
console.log('chunk of', chunk.length, 'bytes:', JSON.stringify(chunk.toString()));
});
rs.on('end', () => console.log('done'));All lessons in this course
- Readable, Writable, Duplex & Transform Stream Internals
- Implementing Custom Transform Streams with _transform and _flush
- Backpressure, pipe() and the pipeline() Utility
- Async Iterators and for-await-of Over Streams