Text Manipulation with `sed` and `awk`

Welcome to a powerful lesson on text manipulation! We'll explore two essential Linux command-line tools: sed and awk.

These tools are incredibly useful for editing and processing text files, making them indispensable for system administrators, developers, and data analysts.

• sed (Stream Editor): Great for simple text transformations like finding and replacing text or deleting lines.
• awk (Pattern Scanning and Processing Language): A more robust language for advanced data extraction, reporting, and conditional processing.

The most common use for sed is substitution. You can replace specific text patterns on a line using the s command, which stands for substitute.

The basic syntax is sed 's/pattern/replacement/'. By default, it replaces only the first occurrence of the pattern on each line.

What if you want to replace all occurrences on a line, or ignore case? sed has flags for that!

• The g flag (global) replaces all matches on a line.
• The i flag (case-insensitive) makes the pattern matching ignore uppercase/lowercase differences.

You can combine flags, like gi.

Beyond substitution, sed can also delete entire lines based on a pattern or line number. The d command is used for deletion.

To delete lines containing a specific word, you'd use sed '/pattern/d'. You can also delete specific line numbers, like sed '2d' for the second line.

sed can also insert new lines of text before or after existing lines. This is useful for adding headers, footers, or context to your files.

• The i command inserts text before a matched line.
• The a command appends text after a matched line.

The text to insert/append must be on a new line after the command, or escaped with a backslash.

Now, let's dive into awk. It's more than just a stream editor; it's a programming language designed for pattern scanning and processing!

awk processes text files line by line. For each line, it checks if it matches a given pattern. If it does, it performs an associated action. The basic structure is awk 'pattern { action }'.

A key feature of awk is its ability to automatically split each line into fields (columns). By default, whitespace (spaces, tabs) acts as the field separator.

• $1, $2, $3, ...: Refer to the first, second, third field, and so on.
• $0: Refers to the entire line.
• -F option: Use this to specify a custom field separator, e.g., -F',' for comma-separated values.

awk allows you to use conditional statements, like if, within its actions. This means you can process lines or fields only if certain criteria are met.

This is extremely powerful for filtering data based on numeric comparisons, string matches, and more.

awk provides several built-in variables that offer useful information during processing:

• NR (Number of Record): The current line number being processed.
• NF (Number of Fields): The total number of fields in the current line.
• FILENAME: The name of the current input file.

These variables help you create dynamic and context-aware scripts.

Time to test your knowledge of sed!

Great job! You've learned the basics of two incredibly powerful text processing tools: sed and awk.

• sed is your go-to for stream editing, perfect for quick substitutions (s), deleting lines (d), and inserting/appending text. Remember the g (global) and i (case-insensitive) flags!
• awk is a versatile language for pattern scanning. It excels at splitting lines into fields ($1, $2, ...), using custom delimiters (-F), applying conditional logic (if), and leveraging built-in variables like NR and NF.

Practice these commands to master your text manipulation skills!