0Pricing
DevOps Bootcamp · Lesson

Consuming REST APIs with curl and jq Together

Chain curl requests into jq to extract, paginate, and reformat live API responses in scripts.

Consuming REST APIs with curl and jq Together is a free DevOps Bootcamp lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why curl + jq Is the Power Combo

REST APIs return JSON. curl fetches the raw response; jq slices, filters, and reshapes it — all in a single pipeline. No Python script, no Postman, no intermediate file needed.

  • curl handles HTTP: methods, headers, auth, redirects.
  • jq handles JSON: filtering, mapping, transforming, formatting.
  • Piping them together creates concise, composable API workflows.

This lesson builds that skill from first principles through real-world pagination and scripting patterns.

Basic curl Pipeline into jq

The simplest pattern: pipe curl output directly into jq. Use -s (silent) to suppress curl's progress meter so only the JSON body reaches jq.

  • -s — silent mode, no progress bar.
  • . — jq's identity filter; pretty-prints the full response.
  • -r on jq — raw output (no surrounding quotes on strings).
#!/usr/bin/env bash
# Fetch a public endpoint and pretty-print the JSON
curl -s 'https://jsonplaceholder.typicode.com/todos/1' | jq '.'

# Extract just the title field as a plain string
curl -s 'https://jsonplaceholder.typicode.com/todos/1' | jq -r '.title'

Setting Request Headers and Passing Auth Tokens

Most production APIs require an Authorization header or an API key. Pass headers with -H and store secrets in environment variables — never hardcode them.

  • -H 'Authorization: Bearer $TOKEN' — injects the auth header.
  • -H 'Accept: application/json' — explicitly requests JSON back.
  • Variables expand inside double quotes; use " around the header string.
#!/usr/bin/env bash
TOKEN="${GITHUB_TOKEN}"   # set in your shell environment
USER="octocat"

curl -s \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/users/${USER}" \
| jq '{login: .login, repos: .public_repos, followers: .followers}'

Filtering Arrays — .[] and select()

APIs often return arrays. Use .[] to iterate over every element, then select() to keep only items matching a condition.

  • .[] — explodes an array into a stream of objects.
  • select(.field == value) — keeps only matching objects.
  • Chain multiple filters with |.
#!/usr/bin/env bash
# Fetch all todos and keep only the completed ones
curl -s 'https://jsonplaceholder.typicode.com/todos' \
| jq '[.[] | select(.completed == true) | {id, title}]'

# Count how many are completed
curl -s 'https://jsonplaceholder.typicode.com/todos' \
| jq '[.[] | select(.completed == true)] | length'

Extracting Multiple Fields with map()

map() applies a transformation to every element of an array and returns a new array — equivalent to [.[] | ...] but more readable.

  • map({key: .field}) — reshape every object.
  • Combine with @csv or @tsv to produce tabular output.
  • Use -r with @csv/@tsv to get raw text (no JSON quoting).
#!/usr/bin/env bash
# Reshape a posts list into id + title pairs
curl -s 'https://jsonplaceholder.typicode.com/posts' \
| jq 'map({id, title: .title[0:40]})'

# Output as CSV for import into a spreadsheet
curl -s 'https://jsonplaceholder.typicode.com/posts' \
| jq -r '.[] | [.id, .userId, .title] | @csv'

Pagination Pattern — Loop Until Empty Page

Most APIs paginate results. A common pattern is a while loop that increments a page counter and stops when the returned array is empty.

  • Capture the curl response into a variable with $(curl ...).
  • Use jq 'length' to check whether the page has items.
  • Accumulate results with jq -s (slurp) or append to a file.
#!/usr/bin/env bash
# Paginate through jsonplaceholder posts (simulated: page stops at page 2
# because the API returns the full list regardless of ?_page, but the
# pattern is correct for real paginated APIs)

PAGE=1
PER_PAGE=10
OUTPUT="all_posts.json"
echo '[]' > "$OUTPUT"

while true; do
  RESPONSE=$(curl -s "https://jsonplaceholder.typicode.com/posts?_page=${PAGE}&_limit=${PER_PAGE}")
  COUNT=$(echo "$RESPONSE" | jq 'length')

  if [ "$COUNT" -eq 0 ]; then
    echo "No more pages. Stopping at page $((PAGE - 1))."
    break
  fi

  # Merge new items into the accumulated JSON array
  CURRENT=$(cat "$OUTPUT")
  echo "$CURRENT" "$RESPONSE" | jq -s '.[0] + .[1]' > "$OUTPUT"
  echo "Page $PAGE: fetched $COUNT items."
  PAGE=$((PAGE + 1))
done

echo "Total collected: $(jq 'length' "$OUTPUT")"

Link-Header Pagination (GitHub Style)

GitHub and many other APIs use a Link response header to provide the URL of the next page. You must parse the header rather than guessing the URL.

  • curl -i includes response headers in stdout; or use -D - to dump headers to stdout.
  • Parse the Link: <url>; rel="next" header with grep and sed.
  • Loop until no rel="next" link is present.
#!/usr/bin/env bash
# Follow Link-header pagination (GitHub repos example)
# Requires GITHUB_TOKEN in environment
TOKEN="${GITHUB_TOKEN}"
NEXT_URL="https://api.github.com/users/torvalds/repos?per_page=5"
ALL_REPOS="[]"

while [ -n "$NEXT_URL" ]; do
  # Capture full response (headers + body) to a temp file
  TMPFILE=$(mktemp)
  curl -sD "$TMPFILE" \
    -H "Authorization: Bearer ${TOKEN}" \
    -H "Accept: application/vnd.github+json" \
    "$NEXT_URL" \
  | {
      BODY=$(cat)
      ALL_REPOS=$(echo "$ALL_REPOS" "$BODY" | jq -s '.[0] + .[1]')
      echo "$ALL_REPOS" > /tmp/repos_acc.json
    }

  # Extract next URL from Link header
  NEXT_URL=$(grep -i '^link:' "$TMPFILE" \
    | sed -E 's/.*<([^>]+)>; rel="next".*/\1/;t;d')
  rm -f "$TMPFILE"
done

echo "Total repos: $(jq 'length' /tmp/repos_acc.json)"

Chaining Requests — Use Output of One Call as Input to Another

A common workflow: fetch a list, extract an ID, then fetch the detail for each ID. Store intermediate values with $() command substitution and pass them into the next URL.

  • Extract a single value with jq -r '.field'.
  • Loop over multiple IDs with jq -r '.[].id' inside a while read loop.
  • Use sleep between requests to respect rate limits.
#!/usr/bin/env bash
# Step 1: get all user IDs from the /users endpoint
# Step 2: for each user, fetch their posts and count them

curl -s 'https://jsonplaceholder.typicode.com/users' \
| jq -r '.[].id' \
| while read -r USER_ID; do
    POST_COUNT=$(curl -s "https://jsonplaceholder.typicode.com/posts?userId=${USER_ID}" \
                 | jq 'length')
    echo "User ${USER_ID}: ${POST_COUNT} posts"
    sleep 0.1   # be polite to the API
done

POST Requests — Sending JSON Payloads

To create or update resources, send a POST or PUT with a JSON body. Use -X POST, -H 'Content-Type: application/json', and -d for the body. Build the payload with jq -n to avoid quoting pitfalls.

  • jq -n --arg key value '{key: $key}' — safe variable interpolation in jq.
  • Pipe the constructed JSON directly into curl's -d @- (read body from stdin).
  • Parse the response immediately with another jq filter.
#!/usr/bin/env bash
TITLE="My New Post"
BODY_TEXT="Written via curl and jq."
USER_ID=1

# Build the JSON payload safely and POST it
RESPONSE=$(jq -n \
  --arg title   "$TITLE" \
  --arg body    "$BODY_TEXT" \
  --argjson userId "$USER_ID" \
  '{title: $title, body: $body, userId: $userId}' \
| curl -s \
    -X POST \
    -H 'Content-Type: application/json' \
    -d @- \
    'https://jsonplaceholder.typicode.com/posts')

echo "Created post ID: $(echo "$RESPONSE" | jq '.id')"
echo "Full response:"
echo "$RESPONSE" | jq '.'

Error Handling — HTTP Status Codes and API Errors

A successful HTTP connection does not mean a successful API call. Check the HTTP status code and the JSON error field separately.

  • curl -w '%{http_code}' appends the status code to stdout; use -o to write the body to a file.
  • Compare the code in your script and handle 4xx/5xx differently.
  • Many APIs embed {"error": "..."} in the body — check with jq's has() or type.
#!/usr/bin/env bash
API_URL='https://jsonplaceholder.typicode.com/todos/99999'
TMPBODY=$(mktemp)

HTTP_CODE=$(curl -s -o "$TMPBODY" -w '%{http_code}' "$API_URL")

if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
  echo "Success ($HTTP_CODE):"
  jq '.' "$TMPBODY"
elif [ "$HTTP_CODE" -eq 404 ]; then
  echo "Resource not found (404). Body:"
  jq '.' "$TMPBODY"
elif [ "$HTTP_CODE" -ge 500 ]; then
  echo "Server error ($HTTP_CODE). Retrying later."
else
  echo "Unexpected status: $HTTP_CODE"
  cat "$TMPBODY"
fi

rm -f "$TMPBODY"

Building a Reusable API Helper Function

Wrap the curl + error-checking boilerplate into a shell function. The function handles auth, status checking, and JSON extraction — callers just pass the endpoint and a jq filter.

  • Return non-zero exit codes on HTTP errors so callers can use || or set -e.
  • Accept a jq filter argument so the same function serves many endpoints.
  • Source this function file from any script that needs the API.
#!/usr/bin/env bash
# api_get <endpoint_path> <jq_filter>
# Returns filtered JSON or exits non-zero on error.
api_get() {
  local PATH_PART="$1"
  local JQ_FILTER="${2:-.}"
  local BASE_URL='https://jsonplaceholder.typicode.com'
  local TMPBODY
  TMPBODY=$(mktemp)

  local HTTP_CODE
  HTTP_CODE=$(curl -s \
    -o "$TMPBODY" \
    -w '%{http_code}' \
    "${BASE_URL}${PATH_PART}")

  if [ "$HTTP_CODE" -lt 200 ] || [ "$HTTP_CODE" -ge 300 ]; then
    echo "[ERROR] HTTP $HTTP_CODE for ${PATH_PART}" >&2
    rm -f "$TMPBODY"
    return 1
  fi

  jq -r "$JQ_FILTER" "$TMPBODY"
  rm -f "$TMPBODY"
}

# Usage examples
api_get '/todos/1' '.title'
api_get '/posts?userId=1' '[.[] | .title]'
api_get '/users' '.[] | "\(.id) \(.name) <\(.email)>"'

Knowledge Check: Pagination Strategy

Test your understanding of how to handle paginated REST API responses in a Bash script using curl and jq.

Lesson Recap: curl + jq API Scripting

You now have a complete toolkit for consuming REST APIs from the Bash command line:

  • Basic pipeline: curl -s URL | jq 'filter' — the foundation of everything.
  • Auth headers: pass tokens via -H from environment variables, never hardcoded.
  • Array handling: .[], select(), and map() slice and reshape API responses.
  • Pagination: loop with a page counter (empty-array sentinel) or parse Link headers for next-URL style APIs.
  • Chaining: extract IDs from one response and feed them into the next request inside a while read loop.
  • POST with safe payloads: build JSON bodies using jq -n --arg and pipe into curl's -d @-.
  • Error handling: separate HTTP status (-w '%{http_code}') from API-level errors in the body.
  • Reusable helper: wrap boilerplate in a shell function so scripts stay concise and DRY.

Combine these patterns and you can automate any JSON API workflow entirely from the shell — no additional runtime required.

Frequently asked questions

Is the “Consuming REST APIs with curl and jq Together” lesson free?

Yes — the full text of “Consuming REST APIs with curl and jq Together” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Consuming REST APIs with curl and jq Together”?

Chain curl requests into jq to extract, paginate, and reformat live API responses in scripts. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Consuming REST APIs with curl and jq Together” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Filtering and Selecting JSON with jq Pipelines
  2. Transforming and Building JSON Objects with jq
  3. Consuming REST APIs with curl and jq Together
  4. Editing YAML Configuration Files with yq
← Back to DevOps Bootcamp