Combining Regex with sed/awk
Integrate regular expressions into 'sed' and 'awk' commands for advanced text filtering, substitution, and data extraction.
Combining Regex with sed/awk is a free DevOps Bootcamp lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Regex with sed and awk
Welcome to the final lesson on Regular Expressions! We've learned about the power of regex for pattern matching.
Now, let's combine that power with two essential Linux tools: sed (stream editor) and awk (pattern scanning and processing language). This integration allows for highly advanced text filtering, substitution, and data extraction.
sed: Filtering with Regex
sed can use regular expressions to filter lines. The /pattern/ address allows sed to apply commands only to lines that match the given regex.
The -n option suppresses default output, and p explicitly prints matching lines.
echo -e "apple
banana
orange" | sed -n '/an/p'sed: Substituting with Regex
The s/regex/replacement/ command in sed is where regex truly shines for text transformation. You can replace patterns found by your regex with new text.
The g flag replaces all occurrences on a line, and i makes the match case-insensitive.
echo "Hello world, hello universe" | sed 's/hello/hi/gi'sed: Capturing Groups & Backreferences
Regular expressions allow you to define capturing groups using parentheses (). You can then refer to the text matched by these groups in the replacement string using backreferences like \1, \2, etc.
Use -E with sed for extended regex syntax (e.g., for () without escaping).
echo "Doe, John" | sed -E 's/([A-Za-z]+), ([A-Za-z]+)/\2 \1/'awk: Filtering Records with Regex
awk uses regex patterns to select records (lines) before performing actions. If a pattern is provided before an action block {...}, the action is only executed for lines matching the pattern.
This is similar to grep but allows for more complex processing.
echo -e "Line 1: apple
Line 2: banana
Line 3: orange" | awk '/banana/'awk: Regex on Specific Fields
You can apply regex patterns to specific fields in awk using the ~ (matches) and !~ (does not match) operators. This is very powerful for structured data.
For example, $1 ~ /pattern/ checks if the first field matches the regex.
echo "Name: John, Age: 30
Name: Jane, Age: 25" | awk '$1 ~ /Name/ {print $2}'awk: Extracting Matches (match, substr)
awk provides functions like match(string, regex) to find the position and length of a match, and substr(string, start, length) to extract it.
After match(), RSTART holds the start position and RLENGTH holds the length of the match.
echo "ItemCode: ABC123 Price: 45.67" | awk '{match($0, /[0-9]+\.[0-9]+/); print substr($0, RSTART, RLENGTH)}'awk: In-Place Substitution (sub, gsub)
awk also has built-in functions for substitution: sub(regex, replacement, target) for the first match, and gsub(regex, replacement, target) for all matches.
If target is omitted, $0 (the entire record) is used. You can specify a field like $1.
echo "apple,banana,orange" | awk '{gsub(/a/, "X", $0); print}'Choosing Between sed and awk
While both sed and awk can use regex for substitution and filtering, they excel in different areas:
sedis generally better for simple line-by-line transformations and substitutions. It's a stream editor.awkis more powerful for processing structured data (fields), performing calculations, and handling more complex logic or multi-line patterns. It's a programming language in itself.
Often, they can be chained together with pipes | for very complex tasks.
Challenge: Extract & Transform
Consider the following line of text:
Order#12345: ProductX - Price $19.99Which command would correctly extract the price (e.g., '19.99') and then change the currency symbol to 'EUR'?
Recap: Regex with sed & awk
You've mastered combining Regular Expressions with sed and awk!
sedexcels at filtering and replacing text on a line-by-line basis, using regex for powerful pattern matching and substitution.awkprovides a more programmatic approach, allowing regex to filter records, match within specific fields, and extract/modify data using functions likematch(),substr(),sub(), andgsub().
This skill is crucial for advanced text processing and scripting in Linux.
Frequently asked questions
Is the “Combining Regex with sed/awk” lesson free?
Yes — the full text of “Combining Regex with sed/awk” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.
What will I learn in “Combining Regex with sed/awk”?
Integrate regular expressions into 'sed' and 'awk' commands for advanced text filtering, substitution, and data extraction. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start DevOps Bootcamp?
No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Combining Regex with sed/awk” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this DevOps Bootcamp lesson?
Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Introduction to Regular Expressions (Regex)
- Advanced Grep with Regex
- Combining Regex with sed/awk
- Capture Groups, Backreferences & Substitution