0Pricing
DevOps Bootcamp · Lesson

Querying journald with journalctl in Scripts

Filter systemd journal entries by unit, priority, and time for automated incident triage.

Querying journald with journalctl in Scripts is a free DevOps Bootcamp lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why journald for Incident Triage?

Modern Linux systems running systemd centralise all log output in the journal — a structured, binary log store managed by systemd-journald. Unlike plain text files in /var/log, every journal entry carries rich metadata: unit name, priority level, PID, UID, timestamp, and more.

In automated incident-triage scripts, this metadata lets you:

  • Filter logs to a single service without grep chains
  • Scope queries to exact time windows (last 15 minutes, since a deploy)
  • Emit only critical/error messages, ignoring noise
  • Feed structured output directly into alerting pipelines

The tool that exposes all of this is journalctl. This lesson teaches you to drive it programmatically inside Bash scripts.

Basic journalctl Invocation

The simplest form of journalctl dumps the entire journal. In scripts you almost never want that — always add at least one filter. Here are the most common flags you will chain together:

  • -u <unit> — filter by systemd unit (e.g. nginx.service)
  • -p <priority> — filter by syslog priority (0=emerg … 7=debug)
  • --since / --until — time window
  • -n <N> — last N lines
  • --no-pager — disable interactive paging (essential in scripts)
  • -o <format> — output format (short, json, cat, etc.)

Always pass --no-pager in non-interactive scripts so journalctl does not try to invoke less and hang.

#!/usr/bin/env bash
# Print the last 20 lines of the nginx service journal
journalctl --no-pager -u nginx.service -n 20

Filtering by Systemd Unit

The -u flag accepts any valid unit name. You can supply it multiple times to combine units, which is useful when a single application spans several services (e.g., an API and its database sidecar).

Unit names follow the pattern <name>.service, <name>.socket, <name>.timer, etc. Globbing is supported: -u 'myapp*' matches myapp-api.service, myapp-worker.service, and so on.

In a triage script you typically receive the unit name as an argument, making the filter dynamic.

#!/usr/bin/env bash
# Usage: ./unit_logs.sh nginx.service
UNIT="${1:?Usage: $0 <unit>}"

echo "=== Journal for ${UNIT} (last 50 lines) ==="
journalctl --no-pager -u "${UNIT}" -n 50

# Combine two related units
echo "=== Combined: api + worker ==="
journalctl --no-pager -u myapp-api.service -u myapp-worker.service -n 30

Priority Levels and the -p Flag

The -p flag maps to standard syslog priority levels:

  • 0 — emerg
  • 1 — alert
  • 2 — crit
  • 3 — err
  • 4 — warning
  • 5 — notice
  • 6 — info
  • 7 — debug

You can specify a single level (-p err) to see only that level, or a range (-p emerg..err) to capture everything from emergency through errors — the most common choice for automated alerting.

Named aliases (err, warning, crit) are accepted alongside numeric values.

#!/usr/bin/env bash
# Extract only error-level and above entries for sshd
journalctl --no-pager \
  -u sshd.service \
  -p emerg..err \
  --since "1 hour ago"

# Exit non-zero if any errors were found (useful in CI health checks)
ERROR_COUNT=$(journalctl --no-pager -u sshd.service -p emerg..err \
  --since "1 hour ago" --output=cat | wc -l)

if [[ "${ERROR_COUNT}" -gt 0 ]]; then
  echo "[ALERT] ${ERROR_COUNT} error(s) detected in sshd" >&2
  exit 1
fi

Time-Window Filtering with --since and --until

Time filters are the backbone of incident-window queries. journalctl accepts flexible human-readable timestamps:

  • Relative: "10 minutes ago", "2 hours ago", "yesterday"
  • Absolute: "2026-06-11 14:00:00"
  • Special keywords: today, yesterday, -1h (shorthand)

In deploy scripts a common pattern is to capture the timestamp just before a deploy, then query the journal from that point to detect regressions introduced by the release.

#!/usr/bin/env bash
# Record deploy start time, then check logs afterwards
DEPLOY_START=$(date +"%Y-%m-%d %H:%M:%S")

echo "Deploying at ${DEPLOY_START}..."
# ... your deploy steps here ...
sleep 2  # simulate deploy

echo "=== Journal since deploy start ==="
journalctl --no-pager \
  -u myapp.service \
  --since "${DEPLOY_START}" \
  -p emerg..warning

Structured Output with JSON Format

For machine-readable pipelines, pass -o json (one JSON object per line, NDJSON) or -o json-pretty (formatted). Each object contains all journal fields:

  • MESSAGE — the log text
  • PRIORITY — numeric priority (0–7)
  • _SYSTEMD_UNIT — originating unit
  • __REALTIME_TIMESTAMP — microseconds since epoch
  • _PID, _UID, _HOSTNAME — process metadata

You can pipe this NDJSON stream into jq to extract, filter, or reformat fields for downstream alerting systems such as PagerDuty, Slack webhooks, or SIEM ingestors.

#!/usr/bin/env bash
# Extract error messages as a clean list for a Slack notification
MESSAGES=$(journalctl --no-pager \
  -u nginx.service \
  -p emerg..err \
  --since "30 minutes ago" \
  -o json \
  | jq -r '.MESSAGE' \
  | sort -u)

if [[ -n "${MESSAGES}" ]]; then
  echo "Errors detected:"
  echo "${MESSAGES}"
fi

Following the Journal in Real Time

The -f flag makes journalctl tail the journal live — analogous to tail -f on a log file. Combined with unit and priority filters this becomes a targeted real-time monitor.

In scripted pipelines the more useful pattern is cursor-based polling: save the current journal cursor, then on each poll pass --after-cursor=<cursor> to read only new entries since the last check. This avoids re-processing old lines.

Retrieve the latest cursor with --show-cursor -n 0 and parse the -- cursor: line from the output.

#!/usr/bin/env bash
# Cursor-based polling: read only new journal entries each run
CURSOR_FILE="/tmp/triage_cursor"

if [[ -f "${CURSOR_FILE}" ]]; then
  CURSOR=$(cat "${CURSOR_FILE}")
  NEW_ENTRIES=$(journalctl --no-pager \
    -u myapp.service \
    -p emerg..err \
    --after-cursor="${CURSOR}" \
    -o json)
else
  # First run: look back 5 minutes
  NEW_ENTRIES=$(journalctl --no-pager \
    -u myapp.service \
    -p emerg..err \
    --since "5 minutes ago" \
    -o json)
fi

# Save updated cursor for next poll
journalctl --no-pager -n 0 --show-cursor 2>&1 \
  | grep '^-- cursor:' \
  | sed 's/-- cursor: //' \
  > "${CURSOR_FILE}"

echo "${NEW_ENTRIES}" | jq -r '.MESSAGE // empty'

Boot-Scoped Queries with -b

The -b flag scopes a query to a specific boot session. This is essential after a crash or unexpected reboot to retrieve logs from the previous boot rather than the current one.

  • -b 0 — current boot (default)
  • -b -1 — previous boot
  • -b -2 — two boots ago
  • --list-boots — show all recorded boot sessions with timestamps

Post-mortem scripts commonly dump critical logs from the previous boot (-b -1) to diagnose why the system crashed or why a service failed on startup.

#!/usr/bin/env bash
# Post-mortem: collect critical logs from the previous boot
echo "=== Previous boot sessions ==="
journalctl --list-boots

echo ""
echo "=== Critical entries from previous boot ==="
journalctl --no-pager \
  -b -1 \
  -p emerg..crit \
  -o short-iso

Grepping Inside journalctl vs Native Matches

You can pass a raw grep pattern after all flags, but journalctl also supports native field matches using the syntax FIELD=value. Native matches are evaluated against structured metadata — much faster than post-processing text with grep.

Common useful matches:

  • _PID=1234 — logs from a specific process
  • _COMM=python3 — logs from any process named python3
  • SYSLOG_IDENTIFIER=myapp — logs tagged with a custom identifier

Multiple FIELD=value arguments are ANDed; a + between them creates an OR. Use -g <regex> for full-text grep when structured metadata is insufficient.

#!/usr/bin/env bash
# Native field match: errors from the postgres process only
journalctl --no-pager \
  _COMM=postgres \
  -p emerg..err \
  --since "1 hour ago"

# Full-text grep for a specific error string
journalctl --no-pager \
  -u postgresql.service \
  --since "1 hour ago" \
  -g "FATAL|PANIC" \
  --output=cat

Building a Reusable Triage Function

Once you master individual flags, composing them into a reusable Bash function keeps your triage scripts clean and consistent. A well-designed function should:

  • Accept unit, priority range, and time window as parameters
  • Default to safe, low-noise values when arguments are omitted
  • Return a non-zero exit code when errors are found (integrates with CI pipelines)
  • Write findings to both stdout and a timestamped log file for audit trails
#!/usr/bin/env bash
# triage.sh — reusable journal triage function

triage_unit() {
  local unit="${1:?unit required}"
  local priority="${2:-emerg..err}"
  local since="${3:-1 hour ago}"
  local logfile="/tmp/triage_${unit//[^a-zA-Z0-9]/_}_$(date +%s).log"

  echo "[$(date -Iseconds)] Triaging ${unit} | prio=${priority} | since='${since}'" | tee "${logfile}"

  journalctl --no-pager \
    -u "${unit}" \
    -p "${priority}" \
    --since "${since}" \
    -o short-iso \
    | tee -a "${logfile}"

  local count
  count=$(wc -l < "${logfile}")
  # Subtract 1 for the header line
  (( count-- ))

  if [[ "${count}" -gt 0 ]]; then
    echo "[ALERT] ${count} line(s) logged to ${logfile}" >&2
    return 1
  fi
  return 0
}

# Example: triage nginx errors in the last 30 minutes
triage_unit nginx.service "emerg..err" "30 minutes ago"

Automated Incident Triage Script

The following complete script ties all concepts together into a practical automated triage tool. It reads a list of critical services, queries the journal for each one over a configurable lookback window, aggregates findings, and exits with a failure code if any errors were detected — making it suitable as a cron job or a CI health-check step.

#!/usr/bin/env bash
# incident_triage.sh — automated multi-service journal triage
set -euo pipefail

LOOKBACK="${1:-15 minutes ago}"
PRIORITY="emerg..err"
SERVICES=(nginx.service postgresql.service myapp-api.service myapp-worker.service)
REPORT="/tmp/incident_report_$(date +%Y%m%d_%H%M%S).txt"
FAILED=0

{
  echo "Incident Triage Report"
  echo "Generated : $(date -Iseconds)"
  echo "Lookback  : ${LOOKBACK}"
  echo "Priority  : ${PRIORITY}"
  echo "-----------------------------------"
} > "${REPORT}"

for svc in "${SERVICES[@]}"; do
  ENTRIES=$(journalctl --no-pager \
    -u "${svc}" \
    -p "${PRIORITY}" \
    --since "${LOOKBACK}" \
    --output=cat 2>/dev/null || true)

  COUNT=$(echo "${ENTRIES}" | grep -c . || true)

  if [[ "${COUNT}" -gt 0 ]]; then
    echo "[FAIL] ${svc}: ${COUNT} error(s)" | tee -a "${REPORT}"
    echo "${ENTRIES}" >> "${REPORT}"
    echo "-----------------------------------" >> "${REPORT}"
    FAILED=1
  else
    echo "[OK]   ${svc}"
  fi
done

echo ""
echo "Full report: ${REPORT}"
exit "${FAILED}"

Knowledge Check: Priority Range Filtering

You are writing a script to alert on-call engineers only when a service logs messages at error severity or worse (i.e., error, critical, alert, or emergency). Which journalctl flag combination correctly captures exactly that range?

Lesson Recap: journalctl in Scripts

In this lesson you learned how to query the systemd journal programmatically for automated incident triage:

  • Always pass --no-pager in scripts to prevent interactive blocking.
  • Unit filtering (-u) scopes queries to one or more services; globbing and multiple -u flags are supported.
  • Priority filtering (-p emerg..err) captures only the severity levels you care about — remember lower numbers are more severe.
  • Time windows (--since / --until) isolate log output to a deployment window or lookback period using human-readable timestamps.
  • Boot scoping (-b -1) lets post-mortem scripts read logs from a previous crash session.
  • JSON output (-o json) and jq enable structured pipelines feeding alerting or SIEM systems.
  • Cursor-based polling with --after-cursor avoids re-processing old entries on repeated runs.
  • Native field matches (_COMM=, SYSLOG_IDENTIFIER=) are faster than piping to grep.

Combining these flags inside a reusable Bash function gives you a production-grade triage tool that integrates cleanly with cron, CI pipelines, and on-call alerting workflows.

Frequently asked questions

Is the “Querying journald with journalctl in Scripts” lesson free?

Yes — the full text of “Querying journald with journalctl in Scripts” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Querying journald with journalctl in Scripts”?

Filter systemd journal entries by unit, priority, and time for automated incident triage. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Querying journald with journalctl in Scripts” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Parsing Web and Application Logs at Scale
  2. Real-Time Log Following and Streaming Alerts
  3. Querying journald with journalctl in Scripts
  4. Computing Metrics and Histograms from Log Streams
← Back to DevOps Bootcamp