Records, Fields, and Custom Separators in awk

awk reads input line by line. Each line is called a record, and each whitespace-separated chunk within that line is called a field.

• $0 — the entire current record (the full line)
• $1 — the first field
• $2 — the second field
• $NF — the last field (NF = number of fields)

By default, awk splits fields on any run of whitespace (spaces or tabs). This makes it immediately useful for log parsing, command output, and space-delimited data.

The built-in variable FS (Field Separator) tells awk how to split each record into fields. Its default value is a single space, which triggers whitespace-splitting mode.

You can set FS in two ways:

• With the -F flag on the command line: awk -F':'
• Inside a BEGIN block: BEGIN { FS = ":" }

Both are equivalent. The BEGIN block approach is preferred in longer scripts because it keeps configuration inside the script itself, making it more readable and portable.

For scripts longer than a one-liner, placing FS inside a BEGIN block is the standard practice. The BEGIN block runs before awk reads any input, so the separator is ready for the first record.

This pattern also allows you to set multiple variables at once, add comments, and keep your logic self-contained in a script file.

Common separators you will encounter in the wild:

• Colon : — /etc/passwd, /etc/shadow
• Tab \t — TSV exports, compiler output
• Comma , — CSV files (simple, no quoted fields)
• Pipe | — some log formats, database dumps

CSV (comma-separated values) is one of the most common data formats in shell engineering. Setting FS="," lets awk treat commas as delimiters.

Important caveat: awk's built-in CSV handling does not account for quoted fields that contain commas (e.g., "Smith, John"). For simple CSVs without quoted commas this works perfectly. For RFC 4180-compliant CSVs with quoted fields, use a proper CSV parser or miller/csvkit.

A common real-world task is extracting specific columns and filtering rows by value — both trivial with awk once FS is set.

awk's FS is not limited to a single character — it can be a regular expression. This is powerful for real-world log formats where fields are separated by variable-length delimiters.

Examples of regex separators:

• FS = "[,;]" — split on either comma or semicolon
• FS = "[ \t]+" — split on one or more spaces/tabs (same as default but explicit)
• FS = "::" — split on a literal double colon
• FS = "\\|" — split on a pipe character (must be escaped)

When FS is a regex, awk (gawk) treats it as a pattern, not a literal string.

OFS (Output Field Separator) controls what awk places between fields when you reconstruct a record with print. Its default is a single space.

The key insight: OFS is only inserted between fields when you use the comma syntax in print or when you assign to a field and awk rebuilds $0. If you concatenate with spaces in the source code, OFS is not used.

• print $1, $2, $3 — commas trigger OFS between fields
• print $1 " " $2 — explicit space, OFS ignored

A common pattern: read CSV, transform, write TSV by setting FS="," and OFS="\t".

There is a subtle but important trick: assigning to any field (even assigning a field to itself, $1=$1) forces awk to rebuild $0 by joining all fields with OFS.

This is the idiomatic way to reformat a record's delimiters without manually printing every field:

• Set FS to the input delimiter
• Set OFS to the desired output delimiter
• Trigger a rebuild with $1=$1
• Print $0

This approach scales to any number of fields and avoids hard-coding $1, $2, $3, ....

Just as FS splits fields within a record, RS (Record Separator) defines what separates records from each other. By default RS is a newline, so awk processes one line at a time.

Changing RS unlocks powerful multi-line record processing:

• RS="" — paragraph mode: blank lines separate records (fields can span multiple lines)
• RS=";" — split on semicolons (useful for SQL dumps)
• RS="\n" — explicit newline (the default)

In paragraph mode (RS=""), FS is still used to split fields within the multi-line record, and newlines within a record are also treated as field separators automatically.

ORS (Output Record Separator) is printed after each print statement. By default it is a newline (\n), which is why every print goes to a new line.

Changing ORS lets you:

• Join records onto a single line: ORS=" "
• Add blank lines between output records: ORS="\n\n"
• Create custom output formats like JSON or XML fragments

Note that printf does not use ORS — it only applies to the bare print statement. Use printf when you need full control over formatting.

Apache/nginx access logs have a well-known format with space-delimited fields. Some fields (like the timestamp) contain spaces and are wrapped in brackets or quotes — making direct awk field-splitting tricky.

A practical approach is to use a regex FS that accounts for the structure, or to extract specific fields by position knowing the exact format.

The Combined Log Format fields are roughly:

1. Client IP
2. Ident (usually -)
3. Auth user (usually -)
4. Timestamp (in brackets, counts as one token)
5. Request method+path+protocol (in quotes)
6. HTTP status code
7. Response bytes

In real shell engineering you rarely use these separators in isolation. A common pipeline pattern is to use awk as a format converter in the middle of a pipeline, transforming one structured format into another.

Key takeaways for combining separators:

• Always set FS and OFS together in BEGIN when converting formats
• Use $1=$1 to trigger a rebuild of $0 when you want all fields reformatted without listing them
• Change RS only when your records genuinely span multiple lines
• Use ORS to control spacing between output records
• Prefer printf for precise formatting; use print when ORS auto-termination is convenient

Test your understanding of how OFS interacts with field assignment in awk.

You now have full control over how awk reads and writes structured data. Here is a summary of the four separator variables:

• FS — Input Field Separator. Set with -F or in BEGIN. Can be a character, string, or regex. Default: whitespace.
• OFS — Output Field Separator. Inserted between fields in print $1, $2 calls and when awk rebuilds $0 after a field assignment. Default: single space.
• RS — Input Record Separator. Defines what separates records (lines). Default: newline. Set to empty string for paragraph mode.
• ORS — Output Record Separator. Appended after every print. Default: newline. Change to join lines or add spacing.

The most important pattern to remember: set both FS and OFS in BEGIN, then use $1=$1 to rebuild $0 whenever you want to reformat delimiters across all fields without hard-coding field positions. This pattern works for files with any number of columns and is the foundation of awk-based format conversion pipelines.