Orchestrating Workloads with GNU parallel

GNU parallel is a shell tool that lets you run jobs in parallel on one or multiple machines. Instead of processing a large list of items one by one in a for loop, parallel spreads that work across all available CPU cores simultaneously.

• Speed: A task that takes 8 minutes sequentially can finish in ~1 minute on an 8-core machine.
• Simplicity: It accepts input from stdin, files, or argument lists — no manual process management.
• Safety: Output from different jobs is kept separate; results are never interleaved.

Install it with sudo apt install parallel (Debian/Ubuntu) or brew install parallel (macOS). Verify with parallel --version.

The simplest form of parallel reads items from stdin and runs a command for each one. The placeholder {} represents the current input item.

The example below compresses five log files concurrently using gzip. Without parallel, each file would be compressed one after the other. With it, up to N files (where N = number of CPU cores) are compressed at the same time.

By default, parallel runs one job per CPU core. You can override this with the -j (or --jobs) flag.

• -j 4 — run exactly 4 jobs at once
• -j 0 — run as many jobs as there are inputs (use carefully!)
• -j 200% — run twice as many jobs as there are CPU cores (useful for I/O-bound work)
• -j 50% — use only half the available cores

For CPU-bound tasks, -j $(nproc) is often optimal. For network or disk I/O tasks, you can safely exceed the core count because jobs spend most of their time waiting.

parallel is flexible about where it reads its input list. You are not limited to piping from stdin.

• From a file: parallel -a urls.txt wget {}
• Inline argument list: parallel echo ::: apple banana cherry
• Multiple argument sources (cartesian product): parallel echo {1}-{2} ::: a b c ::: 1 2 — produces a-1, a-2, b-1, b-2, c-1, c-2
• From stdin explicitly: cat list.txt | parallel -j4 process {}

The ::: separator tells parallel to use the following values as an argument source rather than reading from stdin.

parallel provides several placeholder substitutions that let you extract parts of the input string automatically — very useful when inputs are file paths.

• {} — the full input item
• {.} — input without its file extension (report.csv → report)
• {/} — basename only (strips the directory path)
• {//} — directory path only
• {/.} — basename without extension

These eliminate the need for basename / dirname calls inside the job command, making pipelines cleaner and faster.

When jobs finish at different times, their stdout output appears in whichever order they complete. This can make logs hard to read and downstream parsing unreliable.

Two flags control output ordering:

• --keep-order (-k) — prints each job's output in the same order as the input, even if a later job finishes first. The output is buffered until earlier jobs complete.
• --line-buffer — a middle ground: outputs complete lines as they arrive, without waiting for job completion, but never interleaves half-written lines.

Use -k when the downstream consumer expects results in input order (e.g., building a sorted report). Omit it when order does not matter and you want to see results as soon as possible.

Even with ordered output, if a job prints multiple lines, those lines can interleave with lines from another job running at the same time. parallel solves this automatically by buffering each job's complete stdout and stderr, then printing them as a single atomic block once the job finishes.

This behaviour is on by default. You can disable it with --ungroup if you need live streaming output (e.g., long-running jobs with progress bars), but then interleaving becomes possible again.

• Default: output is grouped per job — safe for parsing.
• --ungroup: output streams live — good for interactive monitoring.
• --line-buffer: compromise — lines are never split, but jobs can interleave at line boundaries.

Sometimes the work you want to parallelise is more than a single command — it is a multi-step shell function. You can pass a function to parallel using export -f combined with env_parallel, or by calling bash -c directly.

The safest portable approach for complex jobs is the bash -c '...' pattern. The {} placeholder is passed as $1 when you end with _ {}.

When hitting external services (APIs, remote servers, databases) in parallel, you often need rate limiting and fault tolerance.

• --delay N — wait N seconds between starting each new job (fractional values like 0.5 are allowed). Prevents flooding a service.
• --retries N — if a job exits with a non-zero status, retry it up to N times before giving up. Each retry counts as a new job slot.
• --timeout N — kill a job if it runs longer than N seconds. Combined with --retries this handles hanging jobs gracefully.

Example: downloading 50 URLs with at most 4 concurrent connections, a 0.5 s ramp-up delay between starts, and 3 retries on failure.

parallel can transparently distribute jobs to remote machines over SSH, making it a lightweight cluster compute tool without any special cluster software.

• --sshlogin user@host — run jobs on a specific remote host.
• --sshloginfile machines.txt — read a list of hosts from a file (one per line). Use : as a special entry to also use the local machine.
• --transfer — copy the input file to the remote host before processing.
• --return {} — copy the result file back after the job finishes.
• --cleanup — delete transferred files from the remote host after retrieval.

The remote host must have parallel installed and SSH key-based authentication configured (no password prompts).

For long-running workloads it is essential to monitor progress and diagnose failures after the fact.

• --progress — prints a live summary line showing how many jobs are running, completed, and remaining.
• --eta — estimates time to completion based on average job duration so far.
• --joblog results.log — writes a tab-separated log file with one row per completed job, including exit code, runtime, and the command run. Invaluable for auditing failures.
• --resume --joblog results.log — skip jobs that already appear (with exit code 0) in the log file. If a batch run is interrupted, you can resume it without re-doing successful work.

The --joblog + --resume combination is one of the most powerful features of GNU parallel for robust production pipelines.

Test your understanding of how parallel controls concurrency.

You have covered the core toolkit for distributing large input sets across CPU cores with GNU parallel. Here is what to take away:

• Basic usage: pipe a list into parallel command {} — {} is replaced by each input item.
• Job slots (-j): control concurrency precisely — use core count for CPU-bound work, higher percentages for I/O-bound work.
• Placeholders ({.}, {/}, {//}, {/.}) cleanly extract path components without extra commands.
• Output control: -k preserves input order; default grouping prevents interleaved lines; --ungroup gives live streaming.
• Resilience: --retries, --timeout, and --delay make parallel pipelines robust against flaky jobs and rate limits.
• Auditability: --joblog records every job's outcome; --resume lets you pick up where you left off after an interruption.
• Scale-out: --sshloginfile distributes jobs to remote machines over SSH with zero cluster overhead.

Mastering these options turns parallel into a production-grade workload orchestrator built right into your shell.