Aggregation with awk Arrays and Grouping

In awk, an associative array is a key-value store where keys can be any string or number. Unlike indexed arrays in most languages, awk arrays are hash maps under the hood — perfect for grouping and aggregating data by field values.

• Declare implicitly: just assign arr[key] = value
• Keys are strings by default (numbers are coerced)
• No size limit — awk grows the array as needed
• Ideal for computing sums, counts, and group-by rollups in a single pass

You do not need to initialize a key before incrementing it — awk treats an unset key as zero or an empty string automatically.

The most common aggregation pattern is counting how many times each unique value in a field appears. Use field $1 (or any field) as the array key and increment a counter on every matching row.

The END block runs after all input is consumed — that is where you print the accumulated results.

count[$1]++

After processing, count holds the total number of lines for each unique value of $1.

Counting is just one form of aggregation. To sum a numeric field grouped by another field, accumulate the numeric value into an array keyed by the group field.

The pattern is:

• Key = the group-by field (e.g., $1 for username)
• Value = running total of the numeric field (e.g., $2 for bytes)

This computes a full group-by sum in a single awk pass — no sorting or pre-processing required.

awk lets you maintain multiple arrays simultaneously, giving you count and sum (and therefore average) for each group in one scan of the file. This is far more efficient than running the pipeline twice.

• count[key]++ — tracks occurrences
• total[key] += $N — tracks the running sum
• In END, divide total[k] / count[k] for the per-group average

You can create a composite key by concatenating multiple fields with a separator. This gives you a two-dimensional group-by without any special syntax.

Choose a separator that cannot appear in the data (e.g., SUBSEP, the built-in awk record separator \034, or a literal pipe |).

• Composite key: arr[$1 SUBSEP $2] or arr[$1"|"$2]
• Split key back when printing: split(key, parts, "|")

Associative arrays make it easy to track min and max values per group. The trick is to initialize only on the first occurrence of each key using the special condition !(key in arr).

• !(key in min_arr) is true the first time a key is seen
• After initialization, compare and overwrite with the standard less-than / greater-than check

This pattern avoids a spurious zero that would corrupt your minimum.

A common real-world task is finding the top-N most frequent values. awk handles the counting; pipe through sort and head to rank and slice.

This pipeline pattern is idiomatic in shell engineering:

1. awk counts occurrences into an array, prints count key in END
2. sort -rn sorts numerically in descending order
3. head -n 5 keeps only the top 5

Keeping awk focused on aggregation and delegating sorting to sort follows the Unix philosophy.

When processing multiple files, awk's built-in NR (total records read) and FNR (records in the current file) let you tag which file each record came from using FILENAME.

• FILENAME — name of the file currently being read
• FNR == 1 — triggers at the start of each new file (useful for skipping headers)

This allows per-file group-by aggregation across an entire directory of logs in one awk invocation.

The for (key in array) loop in awk does not guarantee order. For deterministic output, pipe awk's END print to sort, or collect values and sort within awk using the asorti / asort functions (GNU awk only).

The portable shell-pipeline approach is usually preferred for cross-platform scripts:

• sort -k1,1 — sort by the first field (group key) alphabetically
• sort -k2,2rn — sort by second field (count/sum) numerically descending

Sometimes you need to reset aggregation state mid-stream — for example, when processing log files where each section is separated by a blank line or a sentinel value.

• delete arr[key] — removes a single entry
• delete arr — clears the entire array (GNU awk)
• Check existence with (key in arr) before accessing to avoid creating ghost entries

This technique enables sliding-window or per-section aggregation without restarting awk.

Putting it all together — here is a production-grade awk one-pass aggregation over an Nginx combined log format. It computes request count, total bytes, and error rate per virtual host in a single scan.

Key techniques used:

• Composite key: vhost extracted from a field
• Parallel arrays: reqs[], bytes[], errors[]
• Conditional accumulation: $9 >= 400 to count only error responses
• Formatted printf table in END

Test your understanding of a common awk aggregation pitfall.

You have learned the core patterns for computing group-by aggregations entirely within awk:

• Counting: count[$key]++ — increment on each row, print in END
• Summing: total[$key] += $N — accumulate numeric fields per group
• Average: maintain both count[] and total[], divide in END
• Min/Max: seed on first occurrence using !(key in arr), then compare
• Composite keys: concatenate fields with a separator for multi-dimensional grouping
• Sorted output: pipe awk's END print to sort for deterministic ordering
• Section resets: use delete arr to clear state between logical sections of input

These patterns let you replace heavy database queries or multiple pipeline passes with a single, efficient awk invocation — an essential skill for production shell engineering.