0Pricing
DevOps Bootcamp · Lesson

Real-Time Log Following and Streaming Alerts

Tail and filter live log streams to fire alerts the moment error patterns appear.

Real-Time Log Following and Streaming Alerts is a free DevOps Bootcamp lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Real-Time Log Following Matters

In production systems, log files grow continuously. Waiting until a problem is reported before examining logs means downtime has already cost you. Real-time log following lets you watch events unfold the moment they are written, enabling immediate response to failures, security events, and performance degradation.

  • Web servers write one line per request — errors appear instantly
  • Application daemons log stack traces the moment an exception fires
  • Auth systems record failed login attempts in real time

The foundational Unix tool for this is tail -f, combined with filtering and alerting utilities to turn raw log streams into actionable signals.

tail -f: Following a Live Log

tail -f (follow) keeps the file open and prints new lines as they are appended. It is the simplest and most universally available real-time log tool.

Common usage patterns:

  • tail -f /var/log/syslog — follow the system log
  • tail -n 50 -f app.log — show last 50 lines then follow
  • tail -f /var/log/nginx/access.log — follow nginx requests live

Press Ctrl+C to stop. The process stays attached until you cancel or the terminal closes.

#!/usr/bin/env bash
# Simulate a live log and follow it
LOGFILE='/tmp/demo_app.log'

# Write a header
echo '[INFO] Application started' >> "$LOGFILE"

# In a real scenario you would run:
# tail -f "$LOGFILE"
# Below we demonstrate tail showing the last 3 lines then exit
tail -n 3 "$LOGFILE"

tail -F: Surviving Log Rotation

Many systems rotate log files at midnight or when they reach a size limit. When that happens, the original file is renamed (e.g. app.log.1) and a new app.log is created. With tail -f you silently keep reading the old renamed file and miss all new output.

tail -F (capital F) solves this by watching the filename, not the file descriptor. When the file disappears and reappears, tail -F automatically reopens it and continues following.

  • Always prefer tail -F over tail -f in production scripts
  • Works with logrotate, newsyslog, and Docker log drivers that rotate files
# Follow nginx access log, surviving log rotation
tail -F /var/log/nginx/access.log

# Follow multiple files simultaneously
tail -F /var/log/nginx/access.log /var/log/nginx/error.log

Filtering the Stream with grep

Following a busy log raw is overwhelming — an active web server can write hundreds of lines per second. Pipe the output of tail -F into grep to isolate only the patterns you care about.

Key flags for streaming grep:

  • --line-buffered — flush each matched line immediately instead of buffering; required in pipelines or output will be delayed or lost
  • -i — case-insensitive match
  • -E — extended regex for alternation (error|warn|crit)
  • -v — invert match (exclude lines)
# Show only ERROR and WARN lines from a live application log
tail -F /var/log/myapp/app.log | grep --line-buffered -Ei 'error|warn|critical'

# Follow nginx and exclude health-check requests
tail -F /var/log/nginx/access.log | grep --line-buffered -v '/health'

Adding Timestamps and Context with awk

Log lines sometimes lack context that helps with triage. You can enrich the stream in real time using awk — adding a local timestamp, extracting fields, or reformatting output for readability.

awk also runs in streaming (line-buffered) mode when piped, making it safe to use in live pipelines without extra flags.

# Prepend a reception timestamp to every ERROR line
tail -F /var/log/myapp/app.log | \
  grep --line-buffered -i 'error' | \
  awk '{ print strftime("[%Y-%m-%d %H:%M:%S]"), $0; fflush() }'

# Extract HTTP status code (field 9) and URL (field 7) from nginx combined log
tail -F /var/log/nginx/access.log | \
  awk '{ print $9, $7; fflush() }' | \
  grep --line-buffered '^5'

Sending Alerts with Slack Webhooks

Filtering is only half the job — once you detect an error pattern you need to notify someone. A Slack incoming webhook lets you POST a message to a channel with a single curl call, requiring no Slack SDK or credentials beyond the webhook URL.

The pattern is: filter the stream, and for each matching line send an HTTP POST.

#!/usr/bin/env bash
# Real-time alert: send every ERROR line to a Slack channel
LOGFILE='/var/log/myapp/app.log'
WEBHOOK_URL='https://hooks.slack.com/services/T000/B000/XXXX'

tail -F "$LOGFILE" | grep --line-buffered -i 'error' | while IFS= read -r line; do
  payload=$(printf '{"text":"*[ERROR ALERT]*\n%s"}' "$line")
  curl -s -o /dev/null -X POST -H 'Content-Type: application/json' \
    -d "$payload" "$WEBHOOK_URL"
done

Rate-Limiting Alerts to Prevent Noise

A log storm can generate thousands of error lines per minute. Sending one Slack message per line will flood the channel and cause alert fatigue. You need rate limiting — fire the alert, then suppress further notifications for a cooldown period.

This is achieved with a simple timestamp file: record when the last alert was sent, and skip firing if the cooldown has not elapsed.

#!/usr/bin/env bash
# Alert on ERROR lines but no more than once every 60 seconds
LOGFILE='/var/log/myapp/app.log'
WEBHOOK_URL='https://hooks.slack.com/services/T000/B000/XXXX'
COOLDOWN=60
LAST_ALERT_FILE='/tmp/last_alert_ts'

tail -F "$LOGFILE" | grep --line-buffered -i 'error' | while IFS= read -r line; do
  now=$(date +%s)
  last=0
  [ -f "$LAST_ALERT_FILE" ] && last=$(cat "$LAST_ALERT_FILE")

  if (( now - last >= COOLDOWN )); then
    echo "$now" > "$LAST_ALERT_FILE"
    payload=$(printf '{"text":"*[ERROR ALERT]*\n%s"}' "$line")
    curl -s -o /dev/null -X POST -H 'Content-Type: application/json' \
      -d "$payload" "$WEBHOOK_URL"
    echo "[$(date)] Alert sent: $line"
  else
    echo "[$(date)] Suppressed (cooldown): $line"
  fi
done

Counting Error Bursts with a Sliding Window

Sometimes a single error line is not significant — but 20 errors in 30 seconds is a serious problem. A sliding-window counter lets you trigger alerts only when an error rate threshold is breached, reducing false positives.

The technique stores each matching event's epoch timestamp in a temp file, then counts how many fall within the window before deciding whether to alert.

#!/usr/bin/env bash
# Alert when more than 10 errors occur within any 60-second window
LOGFILE='/var/log/myapp/app.log'
WINDOW=60
THRESHOLD=10
TS_FILE='/tmp/error_timestamps'
WEBHOOK_URL='https://hooks.slack.com/services/T000/B000/XXXX'

tail -F "$LOGFILE" | grep --line-buffered -i 'error' | while IFS= read -r line; do
  now=$(date +%s)
  echo "$now" >> "$TS_FILE"

  # Keep only timestamps within the window
  cutoff=$(( now - WINDOW ))
  tmp=$(mktemp)
  awk -v c="$cutoff" '$1 > c' "$TS_FILE" > "$tmp" && mv "$tmp" "$TS_FILE"

  count=$(wc -l < "$TS_FILE")
  if (( count > THRESHOLD )); then
    msg="*[BURST ALERT]* ${count} errors in ${WINDOW}s — last: ${line}"
    curl -s -o /dev/null -X POST -H 'Content-Type: application/json' \
      -d "{\"text\":\"$msg\"}" "$WEBHOOK_URL"
    # Clear to avoid re-alerting until next burst
    > "$TS_FILE"
  fi
done

journalctl -f: Following systemd Journals

On modern Linux systems (RHEL, Ubuntu 20.04+, Debian 10+), services write to the systemd journal rather than plain text files. journalctl -f is the equivalent of tail -F for the journal.

Useful flags:

  • -u myapp.service — follow a specific unit only
  • -p err — filter by priority (emerg, alert, crit, err, warning, notice, info, debug)
  • --since '5 min ago' — start from a relative time
  • -o json — output structured JSON for machine parsing
# Follow only error-and-above entries for nginx
journalctl -f -u nginx.service -p err

# Stream journal as JSON and extract MESSAGE field with jq
journalctl -f -u myapp.service -o json | \
  jq --unbuffered -r 'select(.PRIORITY <= "3") | .MESSAGE'

multitail and color-coded Multi-Source Monitoring

When you need to watch several log sources simultaneously, multitail splits the terminal into panes — each following a different file or command — with optional color-coding by pattern.

If multitail is not installed, a lightweight pure-Bash alternative is to prefix each stream with the source name and merge them into one view.

# multitail: watch nginx access + error + app log in split panes
# (requires: apt install multitail  or  brew install multitail)
multitail /var/log/nginx/access.log /var/log/nginx/error.log /var/log/myapp/app.log

# Pure-Bash alternative — merge three streams with labeled prefixes
(
  tail -F /var/log/nginx/access.log | sed --unbuffered 's/^/[nginx-access] /' &
  tail -F /var/log/nginx/error.log  | sed --unbuffered 's/^/[nginx-error]  /' &
  tail -F /var/log/myapp/app.log    | sed --unbuffered 's/^/[myapp]        /' &
  wait
)

Building a Self-Contained Alert Daemon

Bringing together everything in this lesson, a production-grade alert daemon should:

  • Follow the log file robustly with tail -F
  • Filter for critical patterns with buffered grep
  • Rate-limit notifications to avoid alert fatigue
  • Log its own activity so you can audit what was sent
  • Run as a background process managed by systemd or a supervisor

The script below is a minimal but complete daemon you can drop into /usr/local/bin/ and manage with systemd.

#!/usr/bin/env bash
# log_alert_daemon.sh — tail a log and fire Slack alerts with cooldown
set -euo pipefail

LOGFILE=${1:-'/var/log/myapp/app.log'}
PATTERN=${2:-'error|critical|fatal'}
WEBHOOK_URL=${SLACK_WEBHOOK_URL:?'Set SLACK_WEBHOOK_URL env var'}
COOLDOWN=${ALERT_COOLDOWN:-120}
DAEMON_LOG='/var/log/log_alert_daemon.log'
LAST_SENT_FILE='/tmp/log_alert_last_sent'

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$DAEMON_LOG"; }

log "Starting alert daemon: watching $LOGFILE for pattern: $PATTERN"

tail -F "$LOGFILE" | grep --line-buffered -Ei "$PATTERN" | while IFS= read -r line; do
  now=$(date +%s)
  last=0
  [ -f "$LAST_SENT_FILE" ] && last=$(cat "$LAST_SENT_FILE")

  if (( now - last >= COOLDOWN )); then
    echo "$now" > "$LAST_SENT_FILE"
    msg=$(printf '{"text":"*[ALERT]* %s\n%s"}' "$(hostname)" "$line")
    if curl -s -o /dev/null -w '%{http_code}' -X POST \
         -H 'Content-Type: application/json' -d "$msg" "$WEBHOOK_URL" | grep -q '^200$'; then
      log "Alert sent: $line"
    else
      log "Alert FAILED to send: $line"
    fi
  else
    log "Suppressed (cooldown ${COOLDOWN}s): $line"
  fi
done

Knowledge Check: Streaming Log Pipelines

Test your understanding of real-time log following and streaming alerts.

A Bash pipeline follows a log file and sends a Slack alert for every matched line. During a log storm, 3,000 error lines are written in 10 seconds. Which single change best prevents the script from flooding the Slack channel with 3,000 messages?

Lesson Recap: Real-Time Log Following and Streaming Alerts

In this lesson you built a complete real-time log observability pipeline from first principles:

  • tail -F follows a log file by name, surviving log rotation — always prefer it over tail -f in production
  • grep --line-buffered filters the live stream without introducing latency; always add this flag in piped grep commands
  • awk with fflush() enriches each line with timestamps or extracted fields in a streaming-safe way
  • Slack webhooks via curl deliver alerts with a single HTTP POST — no SDK required
  • Cooldown files prevent alert fatigue during log storms by enforcing a minimum interval between notifications
  • Sliding-window counters detect error bursts (rate-based alerting) rather than reacting to every individual line
  • journalctl -f is the systemd-native equivalent of tail -F, with built-in priority filtering and JSON output
  • A self-contained alert daemon script combines all these patterns and can be managed by systemd for production reliability

These primitives compose into the foundation of any custom observability pipeline — no third-party agent required.

Frequently asked questions

Is the “Real-Time Log Following and Streaming Alerts” lesson free?

Yes — the full text of “Real-Time Log Following and Streaming Alerts” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Real-Time Log Following and Streaming Alerts”?

Tail and filter live log streams to fire alerts the moment error patterns appear. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Real-Time Log Following and Streaming Alerts” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Parsing Web and Application Logs at Scale
  2. Real-Time Log Following and Streaming Alerts
  3. Querying journald with journalctl in Scripts
  4. Computing Metrics and Histograms from Log Streams
← Back to DevOps Bootcamp