Filtering and Selecting JSON with jq Pipelines

jq is a lightweight, powerful command-line tool for parsing, filtering, and transforming JSON data. It is the sed of JSON — you pipe JSON into it and get structured output back.

• Pre-installed on most Linux distros or available via apt install jq / brew install jq
• Works seamlessly in shell pipelines with curl, cat, and other tools
• Supports filtering, mapping, reduction, and format conversion

The basic invocation is: jq '<filter>' file.json or piped as cat file.json | jq '<filter>'. The filter . (dot) is the identity — it pretty-prints the whole document.

To access a field in a JSON object, use dot notation: .fieldName. You can chain selectors to navigate nested objects.

• .name — top-level field
• .address.city — nested field
• ."field-with-dash" — fields with special characters need quotes

If the field does not exist, jq returns null rather than erroring. This makes it safe to use in scripts without extra null-checks for optional fields.

JSON arrays are accessed with bracket notation. jq uses zero-based indexing.

• .items[0] — first element
• .items[-1] — last element
• .items[1:3] — slice (index 1 up to but not including 3)
• .items[] — explode the array: outputs each element as a separate value (this is the iterator)

The iterator [] is fundamental to jq pipelines — it lets you apply subsequent filters to every element independently.

Just like the shell pipe |, jq has its own internal pipe operator. It passes the output of one filter as the input to the next.

• jq '.users[] | .name' — iterate users, then extract name from each
• jq '.data | .items[] | .id' — navigate to data, explode items, extract id

Pipes inside a jq expression allow complex transformations to be built step by step. Each stage receives whatever the previous stage produced — including multiple values from an iterator.

Key insight: when an iterator produces N values, every downstream filter runs N times, once per value.

The select(condition) filter passes a value through only if the condition is true; otherwise it produces no output. It is the jq equivalent of grep or WHERE in SQL.

• select(.age > 18) — keep objects where age is greater than 18
• select(.status == "active") — equality check
• select(.name | startswith("A")) — nested string test

Combine select with the iterator to filter arrays: .items[] | select(.active) outputs only elements where .active is truthy.

jq lets you reshape data by constructing new objects with {} and new arrays with [].

• {name: .name, city: .address.city} — pick and rename fields into a new object
• [.items[] | .id] — collect iterated values back into an array
• Shorthand: {name, age} is equivalent to {name: .name, age: .age}

Wrapping a pipeline in [...] is called array construction and is essential when you want a JSON array as output rather than a stream of values.

Real-world JSON is often deeply nested. jq provides two tools for deep navigation:

• .a.b.c — explicit path when structure is known
• .. | .fieldName? — recursive descent: walks every node in the tree and outputs values where the key exists

The ? (try) operator suppresses errors when a field does not exist at a given node, which is critical when using recursive descent on heterogeneous trees.

Use recursive descent sparingly on large documents — it visits every node and can be slow. Prefer explicit paths when the structure is predictable.

One of the most common jq use-cases is parsing REST API responses fetched with curl. Combining curl -s (silent) with a jq pipeline gives you clean, scriptable data extraction.

• Extract a single value: curl -s URL | jq '.field'
• Build a summary table: iterate an array, reconstruct objects with only the fields you need
• Use -r (raw output) to strip the surrounding quotes from string values — essential when assigning to shell variables

Tip: always add -r when the jq output will be used as a shell variable or piped to another command.

jq provides two convenient higher-order functions for transforming collections:

• map(f) — applies filter f to every element of an array, returning a new array. Equivalent to [.[] | f].
• map_values(f) — applies f to every value in an object or array, preserving the keys/indices.

These are more readable than manually wrapping pipelines in [] and are idiomatic jq style for transformations that should stay as arrays.

JSON data from external sources is often inconsistent — fields may be missing or null. jq provides the alternative operator // (double slash) to supply a default value.

• .nickname // "anonymous" — use .nickname if it is not null/false, otherwise use "anonymous"
• .count // 0 — numeric default
• Combine with select: select((.status // "inactive") == "active")

This is far more concise than the shell equivalent of ${VAR:-default} and composes cleanly inside longer pipelines.

Structured JSON logging is standard in modern systems. Here is a realistic script that reads a newline-delimited JSON log file, filters for error entries, and formats a human-readable summary.

Key patterns used:

• -c (compact output) — one JSON object per line, useful for piping to shell loops
• --arg name value — inject a shell variable as a jq string argument
• select for log-level filtering
• -r for raw string output suitable for echo

Test your understanding of how select() works inside a jq pipeline.

Given the following command:

What will the output be?

You have covered the core jq toolkit for navigating and filtering JSON from the command line:

• Dot notation (.field, .a.b.c) selects fields from objects
• Array access (.[0], .[]) indexes and iterates arrays
• Pipe operator (|) chains filters; each stage processes all values from the previous one
• select(cond) filters values, passing only those where the condition is truthy
• Object/array construction ({}, [], map()) reshapes data into new structures
• Alternative operator (//) supplies defaults for null or missing fields
• -r flag strips quotes for shell variable assignment; --arg injects shell variables safely
• Recursive descent (.. | .field?) searches deeply nested trees when the path is unknown

With these building blocks you can transform any JSON API response, log file, or configuration into exactly the data your scripts need — all without leaving the terminal.