Parsing Web and Application Logs at Scale

Every HTTP server — Apache, Nginx, Caddy — writes one line to an access log for each request. Understanding the structure of these lines is the foundation of all log analysis work.

A typical Combined Log Format (CLF) line looks like this:

• Client IP — who made the request
• Timestamp — when it happened
• Request line — method, path, protocol
• Status code — HTTP response (200, 404, 500…)
• Bytes sent — response body size
• Referer — origin page
• User-Agent — browser or bot string

Example line from /var/log/nginx/access.log:

192.168.1.10 - alice [11/Jun/2026:14:32:01 +0000] "GET /api/orders HTTP/1.1" 200 1482 "-" "curl/7.88.1"

At scale, these files grow to millions of lines per day. The goal of this lesson is to extract, filter, and aggregate fields from them efficiently using standard BASH tools.

Before writing any pipeline, inspect the log to understand its shape. tail lets you watch a live stream; grep narrows it to relevant lines immediately.

Common patterns:

• tail -n 1000 access.log — last 1000 lines
• tail -f access.log — follow in real time
• tail -f access.log | grep '" 5' — only 5xx errors as they arrive

The key insight is that grep matches against the entire line, so anchoring your pattern matters. Matching ' 500 ' (with spaces) avoids accidentally matching a URL path that contains the string 500.

cut splits each line by a delimiter and prints selected fields. In Combined Log Format the status code sits at field 9 when you split on spaces — but quotes around the request line mean it is safer to count from a known anchor.

A reliable trick: since the request line is always quoted, the status code is always the first token after the closing quote of the request field. Using cut -d'"' -f3 isolates everything after the request quote, then a second cut -d' ' -f2 picks the status code.

This two-stage cut is a classic idiom for CLF logs — fast, no external dependencies.

awk is more powerful than cut because it can accumulate state across lines. The idiomatic pattern for counting occurrences is an associative array keyed on the value you care about.

In CLF, field $9 (1-indexed, space-delimited) is the status code. awk processes each line, increments a counter, then prints a sorted summary in the END block.

Why prefer awk over cut | sort | uniq -c? Because awk does it in a single pass without sorting the full file first — critical when the log is hundreds of gigabytes.

One of the most common operational tasks is finding which client IPs are generating the most errors. This combines filtering (only error lines) with field extraction (the IP at field 1).

The pipeline strategy:

• Use awk to filter on status code range and extract the IP in one step — avoid a separate grep pass
• Pipe to sort | uniq -c | sort -rn | head for a quick top-N view

This pattern is fast enough to run against a 10 GB log file on a single server without loading the file into memory.

Application servers (Rails, Gunicorn, Express with morgan, etc.) often log request duration. Nginx can be configured to emit $request_time as an extra field at the end of each line.

A custom Nginx log format example in nginx.conf:

log_format timed '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" rt=$request_time';

Once latency is in the log you can use awk to compute average, max, and percentile approximations across millions of requests without loading the data into a database.

A single average hides tail latency. A histogram reveals the distribution — whether most requests are fast and a few are very slow (long tail) or whether the distribution is uniform.

The trick is to bucket each value into a rounded range using integer arithmetic inside awk. Multiplying by 1000 (converting seconds to milliseconds) then using integer division gives clean bucket boundaries.

This produces a text histogram you can read directly in a terminal, which is often faster than shipping data to Grafana for a quick investigation.

The User-Agent field (field 6 when splitting on ") identifies clients. Crawlers, scrapers, and malicious bots often pollute your metrics and inflate error counts. Filtering them out gives a cleaner picture of real-user traffic.

Common bot signatures: bot, crawler, spider, curl, python-requests, Googlebot, Bingbot.

Use grep -iv (case-insensitive invert) to exclude known bots, or use awk to split on " and match the UA field directly.

Knowing which endpoints receive the most traffic — and generate the most errors — helps prioritise optimisation and capacity planning. The request path lives inside the quoted request field.

Split on ", take field 2 (the request line), then cut out the method and protocol to isolate the path. For APIs with path parameters like /users/12345 you may also want to normalise IDs to /users/:id using sed or a more complex awk pattern.

The most powerful single-pass analysis combines multiple fields at once: endpoint, status code, and optionally latency. awk associative arrays keyed on composite values make this clean and fast.

The pattern below counts 5xx errors per endpoint in one pass — no temporary files, no intermediate sorts until the very end. This is the approach used in production observability scripts when you need answers in under a minute on a large log.

Logs are rotated daily on most servers. Older files are compressed with gzip as access.log.1.gz, access.log.2.gz, etc. Standard tools cannot read them directly, but two approaches work cleanly:

• zcat — decompress to stdout, pipe into your pipeline
• zgrep — grep directly inside gzip files without extracting

To analyse a full week of logs spanning both uncompressed and compressed files, use process substitution or concatenate with zcat. The snippet below processes the last 7 rotated files plus the current live log in a single awk invocation — no temp files needed.

You are writing an awk one-liner to filter only HTTP 4xx responses from a standard Nginx access log in Combined Log Format (space-delimited, with the request line quoted). Which field number correctly identifies the HTTP status code?

In this lesson you built a complete toolkit for analysing web and application logs at scale using only standard BASH utilities.

Key techniques covered:

• Structure first — Combined Log Format has a predictable field layout; knowing it lets you split reliably with cut -d'"' or awk field references.
• Status code extraction — awk '{ count[$9]++ }' counts all codes in a single pass; filter with $9 ~ /^5/ for server errors.
• Latency analysis — parse the rt= custom field with awk to compute averages, maximums, and histogram buckets without any external tool.
• Client and bot analysis — split on " with -F'"' to reach the User-Agent field; pipe through grep -iv to exclude bots before aggregating.
• Endpoint normalisation — use gsub(/\/[0-9]+/, "/:id") inside awk to collapse parameterised paths before counting.
• Rotated logs — combine cat and zcat in a subshell to feed all rotation files into a single pipeline pass.

These patterns compose: you can chain filtering, extraction, normalisation, and aggregation in a single pipeline that processes hundreds of millions of lines on commodity hardware. Master these primitives and you rarely need a dedicated log-aggregation service for ad-hoc incident investigation.