0Pricing
DevOps Bootcamp · Lesson

Computing Metrics and Histograms from Log Streams

Aggregate request rates, percentiles, and top-N reports directly from streaming log data.

Computing Metrics and Histograms from Log Streams is a free DevOps Bootcamp lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Compute Metrics from Raw Logs?

Production systems emit thousands of log lines per second. Rather than shipping raw logs to expensive analytics platforms, you can compute request rates, percentiles, and top-N reports directly in the shell — at near-zero cost.

  • Request rate: How many requests per second/minute does your service handle?
  • Latency percentiles: What is the p50/p95/p99 response time?
  • Top-N reports: Which endpoints, IPs, or error codes appear most frequently?

Shell tools like awk, sort, uniq, and bc form a powerful, composable pipeline that can answer these questions from a live log stream or a historical file without leaving the terminal.

Anatomy of a Common Access Log

Most web servers write logs in Combined Log Format. Understanding its fields is the foundation of every metric pipeline:

127.0.0.1 - frank [10/Oct/2024:13:55:36 -0700] "GET /api/users HTTP/1.1" 200 2326 0.042
  • Field 1: client IP
  • Field 4 (brackets): timestamp
  • Field 7 (quoted): HTTP method + path
  • Field 9: status code
  • Field 10: response bytes
  • Field 11: response time in seconds (custom field, not always present)

Use awk to reference fields by position ($1, $9, etc.). Fields inside quotes count as one token only when you split carefully — wrap the log line in awk -F'"' or use multiple passes.

Counting Request Rate Per Minute

To compute requests per minute, extract the minute portion of each timestamp and count occurrences. The pattern [day/Mon/year:HH:MM gives you the hour:minute bucket.

The pipeline below reads access.log and prints a table of minute → request count:

#!/usr/bin/env bash
# Requests per minute from an nginx/apache access log
# Usage: bash req_per_min.sh access.log

LOG="${1:-access.log}"

awk '{
    # Extract [day/Mon/year:HH:MM from field 4
    match($0, /\[([^:]+:[0-9]+:[0-9]+)/, arr)
    minute = arr[1]
    if (minute != "") count[minute]++
} END {
    for (m in count) print count[m], m
}' "$LOG" | sort -k2

Sliding-Window Rate with a Live Stream

For a live tail, you need a sliding window. The trick is to use tail -f piped into awk that resets its counter every N seconds using the system clock (systime()).

This example prints a rate line every 10 seconds:

#!/usr/bin/env bash
# Live request rate — prints lines/10s from a tailed log
# Usage: bash live_rate.sh /var/log/nginx/access.log

LOG="${1:-/var/log/nginx/access.log}"
WINDOW=10

tail -f "$LOG" | awk -v win="$WINDOW" '
BEGIN { start = systime(); count = 0 }
{
    count++
    now = systime()
    if (now - start >= win) {
        printf "[%s] %d req/%ds (%.1f req/s)\n",
               strftime("%H:%M:%S", now), count, win, count/win
        count = 0
        start = now
    }
}'

Extracting Latency Values for Percentile Calculation

Percentiles require sorting all observed latency values. The standard approach:

  1. Extract the latency column into a plain number list.
  2. Sort numerically.
  3. Pick the value at the correct rank using line count math.

The script below extracts field 11 (response time in seconds) and saves it to a temp file for percentile computation in the next step:

#!/usr/bin/env bash
# Extract latency column from access log (field 11)
# Assumes last field on each line is response time in seconds

LOG="${1:-access.log}"
TMP=$(mktemp /tmp/latency_XXXXXX.txt)

awk '{ if ($NF ~ /^[0-9]+\.?[0-9]*$/) print $NF }' "$LOG" \
    | sort -n > "$TMP"

echo "Extracted $(wc -l < "$TMP") latency samples -> $TMP"
echo "$TMP"   # callers can read this file

Computing p50, p95, and p99 with awk

Once latency values are sorted, picking a percentile is arithmetic: the pth percentile sits at row ceil(p/100 * N). Pure awk can do this in a single pass after loading the sorted file into an array:

#!/usr/bin/env bash
# Compute p50 / p95 / p99 from a sorted latency file
# Usage: bash percentiles.sh latency_sorted.txt

# Demo: generate 1000 random latencies if no file given
if [[ $# -eq 0 ]]; then
    SORTED=$(mktemp)
    for i in $(seq 1 1000); do
        awk 'BEGIN { srand(); printf "%.4f\n", 0.001 + rand()*0.999 }'
    done | sort -n > "$SORTED"
else
    SORTED="$1"
fi

awk '
{ values[NR] = $1 }
END {
    n = NR
    if (n == 0) { print "No data"; exit }
    p50  = values[int(n * 0.50 + 0.9999)]
    p95  = values[int(n * 0.95 + 0.9999)]
    p99  = values[int(n * 0.99 + 0.9999)]
    printf "p50  = %.4fs\np95  = %.4fs\np99  = %.4fs\nN    = %d\n",
           p50, p95, p99, n
}' "$SORTED"

Building a Top-N Endpoints Report

A top-N report answers "which paths are hit most?" The classic shell idiom is:

  • awk to print the field of interest (e.g., the URL path)
  • sort to group identical values
  • uniq -c to count consecutive duplicates
  • sort -rn to rank by count descending
  • head -n N to take the top N
#!/usr/bin/env bash
# Top-10 most requested URL paths from access log
# Usage: bash top_endpoints.sh access.log [N]

LOG="${1:-access.log}"
N="${2:-10}"

echo "=== Top $N endpoints ==="
awk -F'"' '{ print $2 }' "$LOG" \
    | awk '{ print $2 }' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -n "$N" \
    | awk '{ printf "%6d  %s\n", $1, $2 }'

Top-N by Status Code: Spotting Errors at a Glance

HTTP status codes tell you the health of your service. Grouping log lines by status code and counting them reveals whether errors are rare or systemic.

The script below generates a breakdown across all 2xx/3xx/4xx/5xx buckets:

#!/usr/bin/env bash
# Status code frequency breakdown
# Usage: bash status_breakdown.sh access.log

LOG="${1:-access.log}"

echo "=== HTTP Status Code Distribution ==="
awk '{ print $9 }' "$LOG" \
    | grep -E '^[0-9]{3}$' \
    | sort \
    | uniq -c \
    | sort -rn \
    | awk '{ printf "%6d  %s\n", $1, $2 }'

echo
echo "=== 5xx Error Spike Check ==="
awk '$9 ~ /^5/ { print $9, $7 }' "$LOG" \
    | sort | uniq -c | sort -rn | head -20

ASCII Histograms with awk

A text histogram makes latency distributions instantly readable in a terminal or CI log. The approach:

  1. Bucket each latency value into a bin (e.g., 0-50ms, 50-100ms, …).
  2. Count observations per bin.
  3. Print a bar of # characters scaled to the max count.

This is extremely useful for spotting bimodal distributions or outliers without a graphing tool.

#!/usr/bin/env bash
# ASCII latency histogram from a list of latency values in seconds
# Demo: generates synthetic data if no input file is provided

if [[ $# -eq 0 ]]; then
    # Generate 500 synthetic latencies (ms converted to s)
    python3 -c "
import random, math
for _ in range(500):
    # bimodal: fast cluster ~50ms, slow cluster ~300ms
    if random.random() < 0.75:
        v = max(1, random.gauss(50, 15))
    else:
        v = max(1, random.gauss(300, 80))
    print(f'{v/1000:.4f}')
" | sort -n | awk '
{ values[NR] = $1 * 1000 }  # convert to ms
END {
    buckets = 10; maxVal = 500
    step = maxVal / buckets
    for (i = 0; i < buckets; i++) hist[i] = 0
    for (i = 1; i <= NR; i++) {
        b = int(values[i] / step)
        if (b >= buckets) b = buckets - 1
        hist[b]++
    }
    maxCount = 0
    for (i = 0; i < buckets; i++) if (hist[i] > maxCount) maxCount = hist[i]
    print "Latency (ms)   Count   Distribution"
    print "---------------------------------------"
    for (i = 0; i < buckets; i++) {
        lo = i * step; hi = lo + step
        barLen = int(hist[i] / maxCount * 40)
        bar = ""
        for (j = 0; j < barLen; j++) bar = bar "#"
        printf "%4d-%4dms  %5d  %s\n", lo, hi, hist[i], bar
    }
}'
else
    echo "Usage: pipe a sorted latency file (in seconds) to this pattern"
fi

Combining Metrics: The One-Shot Summary Report

Production runbooks often need a single command that emits all key metrics at once: rate, latency percentiles, top endpoints, and error rate. You can compose everything learned so far into one script that a human or alerting system can call.

#!/usr/bin/env bash
# One-shot log summary report
# Usage: bash log_summary.sh access.log

LOG="${1:-access.log}"
[[ -f "$LOG" ]] || { echo "File not found: $LOG"; exit 1; }

TOTAL=$(wc -l < "$LOG")
ERRORS=$(awk '$9 ~ /^[45]/' "$LOG" | wc -l)
ERR_RATE=$(awk "BEGIN { printf \"%.1f\", ($ERRORS / ($TOTAL || 1)) * 100 }")

echo "====================================="
echo " Log Summary: $LOG"
echo "====================================="
printf " Total requests : %d\n" "$TOTAL"
printf " 4xx/5xx errors : %d (%.1f%%)\n" "$ERRORS" "$ERR_RATE"

echo
echo "--- Top 5 Endpoints ---"
awk -F'"' '{ print $2 }' "$LOG" | awk '{ print $2 }' \
    | sort | uniq -c | sort -rn | head -5 \
    | awk '{ printf "  %6d  %s\n", $1, $2 }'

echo
echo "--- Latency Percentiles ---"
awk '{ if ($NF ~ /^[0-9]+\.?[0-9]*$/) print $NF * 1000 }' "$LOG" \
    | sort -n \
    | awk '
{ v[NR]=$1 }
END {
  if (NR==0) { print "  No latency data"; exit }
  printf "  p50 = %.1fms\n", v[int(NR*0.50+0.9999)]
  printf "  p95 = %.1fms\n", v[int(NR*0.95+0.9999)]
  printf "  p99 = %.1fms\n", v[int(NR*0.99+0.9999)]
}'

Streaming Metrics via Named Pipes and tee

In observability pipelines you often need to fan out a log stream: write raw lines to disk and simultaneously compute metrics. tee with a named pipe (mkfifo) makes this possible without buffering the entire stream in memory.

  • mkfifo /tmp/log_pipe — create the named pipe
  • tee /tmp/log_pipe | metric_consumer & — fork one branch to the metric consumer
  • The other branch writes to the archive file

This pattern keeps disk writes and metric aggregation decoupled and allows either side to restart independently.

#!/usr/bin/env bash
# Fan-out: write to archive AND count errors in real time
# Run: bash fanout_pipeline.sh /var/log/nginx/access.log

LOG="${1:-/var/log/nginx/access.log}"
ARCHIVE="/tmp/access_archive.log"
PIPE="/tmp/log_metrics_pipe"

# Clean up on exit
trap 'rm -f "$PIPE"' EXIT

mkfifo "$PIPE"

# Branch 1: count 5xx errors per minute from the pipe
awk '$9 ~ /^5/ {
    match($0, /\[([^:]+:[0-9]+:[0-9]+)/, arr)
    errors[arr[1]]++
} END {
    for (m in errors) printf "5xx errors at %s: %d\n", m, errors[m]
}' "$PIPE" &

# Branch 2: archive to disk + feed Branch 1 via pipe
tail -f "$LOG" | tee "$PIPE" >> "$ARCHIVE"

Which awk technique correctly computes the p95 percentile from a sorted array of N latency values?

You have loaded N sorted latency values into an awk array v[1..N]. Which expression correctly retrieves the p95 percentile?

Lesson Recap: Metrics and Histograms from Log Streams

In this lesson you built a complete metrics pipeline directly in Bash:

  • Request rate: awk extracts the minute bucket from timestamps and counts occurrences; tail -f + systime() gives a live sliding-window rate.
  • Latency percentiles: extract the latency column, sort -n, then pick the row at ceil(p/100 * N) with int(N * p + 0.9999) in awk.
  • Top-N reports: the classic awk | sort | uniq -c | sort -rn | head -N pipeline works for any categorical field (path, IP, status code).
  • ASCII histograms: bucket values, count per bin, and scale bar length to the maximum bin count for at-a-glance distribution views.
  • Fan-out with named pipes: mkfifo + tee lets you archive raw logs and feed metric consumers simultaneously without loading the stream into memory.

These composable primitives eliminate the need for external tooling during incidents and form the backbone of lightweight observability scripts that run anywhere Bash runs.

Frequently asked questions

Is the “Computing Metrics and Histograms from Log Streams” lesson free?

Yes — the full text of “Computing Metrics and Histograms from Log Streams” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Computing Metrics and Histograms from Log Streams”?

Aggregate request rates, percentiles, and top-N reports directly from streaming log data. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Computing Metrics and Histograms from Log Streams” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Parsing Web and Application Logs at Scale
  2. Real-Time Log Following and Streaming Alerts
  3. Querying journald with journalctl in Scripts
  4. Computing Metrics and Histograms from Log Streams
← Back to DevOps Bootcamp