Sorting & Deduplicating Data (sort, uniq, cut)

Log files and CSVs are just lines of text. Three classic tools shape them: sort orders lines, uniq removes duplicates, and cut extracts columns.

sort orders lines alphabetically by default and prints the result.

Plain sort treats numbers as text (10 before 2). Use -n for true numeric order and -r to reverse.

Use -k to sort by a specific column and -t to set the delimiter.

uniq collapses adjacent duplicate lines. It only works on sorted input, so it is almost always paired with sort.

uniq -c prefixes each line with how many times it appeared — perfect for frequency analysis.

Combine sort, uniq -c, and another sort to rank items by frequency — a one-line "top N" report.

cut pulls fields from each line. Use -d for the delimiter and -f for the field numbers.

cut -c extracts character ranges by position, useful for fixed-width data.

uniq -d shows only lines that repeat, while uniq -u shows only lines that appear exactly once.

The real power is chaining: extract a column, sort it, then count uniques to summarize a dataset in one command.

Test your understanding.

You learned to shape data with sort (alphabetic, numeric -n, by field -k), uniq (dedupe adjacent lines, -c count, -d/-u), and cut (fields with -d/-f, characters with -c) — chained into powerful pipelines.