0Pricing
DevOps Bootcamp · Lesson

Streaming Pipelines and Named Pipes for Throughput

Use FIFOs and process substitution to stream data between stages without intermediate files.

Streaming Pipelines and Named Pipes for Throughput is a free DevOps Bootcamp lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Intermediate Files Hurt Throughput

When you chain commands like sort file.txt > tmp.txt && uniq tmp.txt > result.txt, you pay a hidden tax: disk writes, disk reads, and the pipeline stalls until the first stage finishes completely before the next begins.

Streaming pipelines eliminate that tax. Data flows directly from producer to consumer in memory, stage by stage, concurrently. This is the core idea behind Unix pipes — and named pipes (FIFOs) extend it further.

  • Anonymous pipe (|): connects two adjacent commands in the same shell line.
  • Named pipe (FIFO): a special file in the filesystem that lets unrelated processes stream to each other.
  • Process substitution: lets a command treat another command's output as if it were a file.

This lesson shows you how to apply all three to maximize throughput in real-world Bash workflows.

Anatomy of a Streaming Pipeline

An anonymous pipe connects stdout of one process to stdin of the next. The kernel keeps both processes running simultaneously in a fixed-size in-memory buffer (typically 64 KB on Linux).

The key insight: the pipeline is as fast as its slowest stage. If the producer is faster, it blocks on a full buffer. If the consumer is faster, it blocks on an empty buffer. This back-pressure is free, automatic flow control.

The example below counts unique IP addresses in a large access log without ever writing a temporary file. Each stage runs concurrently:

#!/usr/bin/env bash
# Stream a 2 GB access log — all stages run in parallel
grep '"GET' /var/log/nginx/access.log \
  | awk '{print $1}' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -20

Creating Named Pipes with mkfifo

A named pipe (FIFO — First In, First Out) is created with mkfifo. It appears in the filesystem like a regular file, but data written to it is never stored on disk — it flows directly to the reading process.

Key behaviours to remember:

  • A write to a FIFO blocks until a reader opens it, and vice versa.
  • The FIFO entry persists in the filesystem; you must delete it with rm when done.
  • Multiple writers are allowed, but ordering between them is not guaranteed.

Below: a producer compresses data into a FIFO while a consumer uploads it to S3 simultaneously — no temp file needed.

#!/usr/bin/env bash
mkfifo /tmp/stream_pipe

# Producer: compress in background
gzip -c /var/log/syslog > /tmp/stream_pipe &

# Consumer: read from FIFO (runs in foreground)
wc -l < /tmp/stream_pipe

wait
rm /tmp/stream_pipe

Process Substitution: Treat a Command as a File

Process substitution uses the syntax <(command) or >(command). Bash creates a FIFO (or /dev/fd/N file descriptor) behind the scenes and passes the path to the outer command.

This is powerful when a tool expects a filename argument rather than stdin. Without process substitution you would need a temporary file; with it, you stream directly.

  • <(cmd) — the outer command reads from cmd's output.
  • >(cmd) — the outer command writes to cmd's input.
#!/usr/bin/env bash
# diff two sorted streams without creating temp files
diff <(sort /etc/passwd) <(sort /etc/group)

# Compare live command output against a baseline
diff <(ls /usr/bin | sort) <(cat ~/bin_baseline.txt | sort)

Tee: Splitting a Stream to Multiple Consumers

tee reads stdin and writes it to both stdout and one or more files. Combined with process substitution, you can fan a single stream out to multiple processing pipelines simultaneously — all without touching disk.

This pattern is useful when you want to, for example, both log raw data and process it at the same time.

#!/usr/bin/env bash
# Generate 100000 random numbers, then simultaneously:
#  1. compute the sum
#  2. find the maximum
#  3. count lines (saved to a variable)
seq 1 100000 \
  | tee >(awk '{s+=$1} END{print "Sum:", s}') \
        >(awk 'BEGIN{m=0} $1>m{m=$1} END{print "Max:", m}') \
  | wc -l | xargs echo "Count:"

Fan-Out Pattern: One Producer, Many Consumers

When a single data source must feed multiple independent consumers, combine tee with multiple >() process substitutions. Each consumer gets the full stream and runs concurrently.

This avoids reading the source file multiple times. For a 10 GB file that difference is enormous — one disk read instead of N reads.

#!/usr/bin/env bash
# Read a large CSV once; simultaneously:
#  - count rows
#  - extract column 2 to a file
#  - pass column 3 to a stats script
cat large_data.csv \
  | tee \
      >(wc -l > /tmp/row_count.txt) \
      >(cut -d',' -f2 > /tmp/col2.txt) \
      >(cut -d',' -f3 | awk '{sum+=$1} END{print sum}' > /tmp/col3_sum.txt) \
  > /dev/null

echo "Rows:" $(cat /tmp/row_count.txt)
echo "Col3 sum:" $(cat /tmp/col3_sum.txt)

Fan-In Pattern: Many Producers, One Consumer

The reverse of fan-out is fan-in: multiple independent sources streaming into a single consumer. Named FIFOs make this straightforward.

A common use-case: merging log streams from several servers in real time, or aggregating partial results from parallel workers.

Note that with multiple writers the consumer sees interleaved output — fine for line-oriented data where each line is self-contained, but you must handle ordering yourself if sequence matters.

#!/usr/bin/env bash
mkfifo /tmp/fanin_pipe

# Three producers write concurrently into the same FIFO
for host in web1 web2 web3; do
  ssh "$host" 'tail -n 500 /var/log/app.log' > /tmp/fanin_pipe &
done

# Single consumer reads all merged output
grep 'ERROR' /tmp/fanin_pipe | sort | uniq -c | sort -rn

wait
rm /tmp/fanin_pipe

Using mkfifo for Parallel Compression

One of the most practical uses of FIFOs is parallel compression. Tools like pigz (parallel gzip) or pbzip2 read a stream; you can pipe the raw data directly without staging an uncompressed file.

The pattern below archives a directory, compresses it with all CPU cores, and streams the result to a remote host — all simultaneously:

#!/usr/bin/env bash
# Tar + parallel compress + stream to remote — no temp files
# Requires: pigz (parallel gzip)
tar cf - /data/large_dir \
  | pigz -p 4 \
  | ssh backup-host 'cat > /backups/large_dir.tar.gz'

# Verify the remote file exists
ssh backup-host 'ls -lh /backups/large_dir.tar.gz'

Controlling Buffer Size and Blocking

Pipes have a kernel buffer (usually 64 KB). When the buffer is full, the writer blocks; when empty, the reader blocks. This is usually what you want, but in some cases the blocking causes a deadlock.

Deadlock risk: if process A writes to FIFO1 and reads from FIFO2, while process B writes to FIFO2 and reads from FIFO1, both may block waiting for the other to consume first.

Solutions:

  • Put at least one side in the background (&) so it does not block the shell.
  • Use mbuffer or pv to add a larger in-memory buffer between stages.
  • Use pv -q -B 128m to insert a 128 MB buffer, smoothing throughput spikes.
#!/usr/bin/env bash
# pv adds a 64 MB buffer and shows throughput
# Useful when producer and consumer have bursty speeds
dd if=/dev/urandom bs=1M count=200 \
  | pv -B 64m \
  | gzip \
  | wc -c

Practical Example: Real-Time Log Aggregator

Here is a complete, realistic pattern: tail multiple log files, merge the streams through a named pipe, filter for errors, and write a live summary — all without any intermediate files and with all stages running in parallel.

This is the kind of pipeline that would be run as a background monitoring script on a production server.

#!/usr/bin/env bash
FIFO=/tmp/log_aggregator
mkfifo "$FIFO"

cleanup() { rm -f "$FIFO"; }
trap cleanup EXIT INT TERM

# Fan-in: tail multiple logs into the FIFO
tail -F /var/log/syslog /var/log/auth.log > "$FIFO" &
TAIL_PID=$!

# Consumer: filter and timestamp errors in real time
grep --line-buffered -i 'error\|fail\|crit' "$FIFO" \
  | while IFS= read -r line; do
      printf '[%s] %s\n' "$(date '+%H:%M:%S')" "$line"
    done

kill "$TAIL_PID" 2>/dev/null

Benchmarking Pipelines vs Temp Files

You can measure the real throughput difference between the streaming and temp-file approaches using time. The pipeline approach wins on large datasets because:

  • Stages run concurrently — CPU and I/O overlap.
  • No disk I/O for intermediate data — only final output hits disk.
  • Memory footprint stays constant regardless of input size (stream, not buffer).

A simple benchmark to compare both approaches:

#!/usr/bin/env bash
# Approach 1: Temp file (sequential)
time bash -c '
  seq 1 5000000 > /tmp/nums.txt
  sort -n /tmp/nums.txt > /tmp/sorted.txt
  uniq /tmp/sorted.txt | wc -l
  rm /tmp/nums.txt /tmp/sorted.txt
'

echo '---'

# Approach 2: Streaming pipeline (concurrent)
time bash -c 'seq 1 5000000 | sort -n | uniq | wc -l'

Knowledge Check: Named Pipe Blocking Behavior

Consider the following script:

mkfifo /tmp/mypipe
echo 'hello' > /tmp/mypipe
echo 'done'

What happens when this script is run without any background processes or readers?

Recap: Streaming Pipelines and Named Pipes

In this lesson you explored how to move data efficiently between processes without intermediate files:

  • Anonymous pipes (|) connect adjacent commands and run all stages concurrently with automatic back-pressure.
  • Named pipes (mkfifo) create a FIFO filesystem entry that lets unrelated or background processes stream to each other — writes block until a reader is present.
  • Process substitution (<(cmd), >(cmd)) lets commands that expect filenames consume or produce streams transparently.
  • tee + >() fans one stream out to multiple concurrent consumers without re-reading the source.
  • Fan-in merges multiple producers into one consumer via a shared FIFO.
  • Back-pressure and blocking are features, not bugs — but always background at least one side of a FIFO pair to avoid deadlock.
  • Use pv or mbuffer to add larger buffers and monitor throughput when stages are bursty.

These techniques are the foundation of high-throughput Bash data engineering: process GB-scale data with constant memory and maximum CPU/IO parallelism.

Frequently asked questions

Is the “Streaming Pipelines and Named Pipes for Throughput” lesson free?

Yes — the full text of “Streaming Pipelines and Named Pipes for Throughput” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Streaming Pipelines and Named Pipes for Throughput”?

Use FIFOs and process substitution to stream data between stages without intermediate files. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Streaming Pipelines and Named Pipes for Throughput” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Profiling Scripts and Avoiding Useless Subshells
  2. Parallelism with xargs -P and Background Jobs
  3. Orchestrating Workloads with GNU parallel
  4. Streaming Pipelines and Named Pipes for Throughput
← Back to DevOps Bootcamp