Patterns, Ranges, and BEGIN/END Blocks

In awk, a pattern is a condition that controls whether an action block executes for a given input line. The general form is:

awk 'pattern { action }' file

If the pattern matches the current record, the action runs. If no action is given, awk prints the entire line by default. Patterns can be:

• Regular expressions — /regex/
• Relational expressions — $3 > 100
• Compound expressions — $1 == "ERROR" && $4 > 500
• Range patterns — /start/,/end/
• Special patterns — BEGIN and END

Understanding patterns is the gateway to using awk as a powerful data filter rather than just a column printer.

The most common pattern type is a regular expression enclosed in forward slashes. awk tests each line against the regex and runs the action only on matches.

Below we filter an /var/log/syslog-style file to show only lines containing the word ERROR:

Prefix a regex pattern with ! to invert it — matching lines that do not contain the pattern. Combine multiple conditions with && (AND) or || (OR) for fine-grained control.

• !/DEBUG/ — skip debug lines
• $3 > 500 && $5 == "POST" — large POST requests
• /WARN/ || /ERROR/ — either severity

The example below parses a simplified access log and prints only POST requests with a response size above 1000 bytes:

A range pattern selects a contiguous block of records — from the line matching /start/ through the line matching /end/, inclusive. The syntax is:

awk '/start/,/end/ { action }' file

Key behaviours to know:

• The range activates on the first line that matches /start/ and deactivates after the first subsequent line matching /end/.
• If /end/ never matches, the range stays active until EOF.
• Multiple non-overlapping ranges in the same file are each processed independently.

This makes range patterns ideal for extracting marked sections from config files, logs with delimiter headers, or multi-block reports.

Consider a server config that has multiple named sections delimited by [section] headers. We want only the lines inside [database]:

The BEGIN block runs once before any input is read. It is ideal for:

• Printing a report header
• Initialising variables and counters
• Setting the field separator (FS) or other built-in variables
• Running stand-alone awk programs that need no input

Syntax:

awk 'BEGIN { setup } pattern { action }' file

Setting FS in BEGIN is the clean alternative to the -F flag — particularly useful when writing multi-rule programs where the separator logic lives inside the script.

The END block runs once after the last record is processed. It is the natural place to:

• Print totals, averages, and summaries accumulated during processing
• Flush or close output files
• Print footer lines for formatted reports

Variables set during the main rules are still in scope inside END — this is how running totals work. The example below sums salary values from a CSV and prints the total in the footer:

The real power emerges when you combine all three blocks into a single self-contained awk program that prints a header, processes each record, and prints a summary. This pattern is the backbone of shell-based reporting pipelines.

Below is a complete report generator that reads a CSV of web requests and prints per-line details plus a final count of requests that exceeded 1 second:

Two often-overlooked output variables can be set cleanly in BEGIN:

• OFS (Output Field Separator) — inserted between fields when you rebuild a record with $1=$1 or use print $1, $2
• ORS (Output Record Separator) — appended after each print call (default is \n)

Setting OFS="," lets you reformat a TSV into CSV in one pass without string concatenation. Changing ORS to "\n---\n" adds dividers between records automatically.

Range patterns shine in operational log analysis. A common task is extracting all log lines between the start and end of an incident, identified by sentinel messages like INCIDENT START and INCIDENT END.

In the example below, awk extracts only the lines inside the incident window and counts the ERROR events within it — combining a range pattern with an accumulator and an END summary:

By default, range patterns include the lines that match /start/ and /end/ themselves. To skip those delimiter lines and keep only the content in between, guard the action with a negation or a flag variable:

• Negation approach: /start/,/end/ { if (!/start/ && !/end/) print }
• Flag approach: manually set a variable to 1 on /start/ and 0 on /end/

The flag approach is more robust when the delimiter text might also appear in the body content:

Test your understanding of how BEGIN and END blocks interact with input processing in awk.

In this lesson you mastered the three most powerful structural features of awk programs:

• Patterns — regex (/ERROR/), relational ($3 > 1000), negated (!/DEBUG/), and compound (&&, ||) — control which records trigger an action without needing explicit if statements.
• Range patterns (/start/,/end/) — select contiguous blocks of records, perfect for extracting log incident windows, config sections, and report segments. Use a flag variable (next + boolean) when you need to exclude the delimiter lines themselves.
• BEGIN block — runs once before any input; ideal for setting FS, OFS, ORS, initialising counters, and printing report headers.
• END block — runs once after all input; the right place to print totals, averages, and footers using variables accumulated in the main rules.

Together these features let you write complete, self-documenting awk programs that ingest raw text and emit structured, human-readable reports — all in a single pipeline stage without temporary files or external tools.