Advanced Text Manipulation (sed, awk)

sed, short for "stream editor," is a powerful command-line utility for parsing and transforming text. It processes text streams line by line. Think of it as a non-interactive text editor.

sed reads a line, applies a command (like substitution or deletion), prints the result, and then moves to the next line. This makes it very efficient for large files or piped data.

The most common use of sed is for substituting text. We use the s command, followed by a delimiter (often /), the pattern to find, another delimiter, the replacement text, and a final delimiter.

Syntax: sed 's/pattern/replacement/' filename

Try replacing "apple" with "orange" in a sample file:

By default, sed only replaces the first occurrence of a pattern on each line. To replace all occurrences on a line, use the g (global) flag.

To perform a case-insensitive search, use the i flag. You can combine flags, like gi.

Example: Replace all "apple" (case-insensitive) with "banana".

You can also use sed to delete entire lines that match a specific pattern. The d command is used for this.

Syntax: sed '/pattern/d' filename

This is useful for removing unwanted lines, like log entries containing "DEBUG" or empty lines.

Let's remove lines containing "error":

awk is another powerful text processing tool, often used for data extraction and reporting. While sed is a stream editor, awk is more like a programming language designed for text.

awk processes text files line by line, just like sed. However, it treats each line as a "record" and splits records into "fields" (columns) based on a separator.

By default, awk uses whitespace (spaces, tabs) as the field separator.

In awk, fields are referred to by $1, $2, $3, and so on. $0 represents the entire line.

You can print specific fields using the print command within curly braces {}. The syntax is: awk '{ print $FIELD_NUMBER }' filename

Let's print the first and third fields from a simple data file:

Just like sed, awk can also filter lines based on patterns. You specify a pattern before the action block. The action will only be performed on lines that match the pattern.

Syntax: awk '/pattern/ { action }' filename

Let's find and print only the names of people from "London":

awk allows you to specify actions to be performed before processing any lines (BEGIN block) and after all lines have been processed (END block).

BEGIN blocks are great for printing headers or initializing variables. END blocks are perfect for printing summaries, totals, or footers.

Let's add a header and footer to our output:

You can use if statements within awk's action blocks to apply logic based on field values or other conditions. This makes awk very powerful for complex data filtering and transformation.

Syntax: awk '{ if (condition) { action } }' filename

Let's print only users older than 25:

Consider the following content in a file named names.txt:

John Doe,30,New York
Jane Smith,25,London
Peter Jones,40,Paris

Which of the following commands would output only the names (first field) of people older than 30?

You've now explored the power of sed for stream editing and awk for advanced text processing!

• sed is perfect for simple substitutions, deletions, and transformations on a line-by-line basis.
• awk excels at parsing structured text, working with fields, and applying conditional logic for data extraction and reporting.

These tools are indispensable for scripting, log analysis, and data manipulation in the Linux environment.