Selecting Columns by Pattern
Use filter(like=), filter(regex=), and list comprehensions to select columns matching a name pattern.
Selecting Columns by Pattern is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Select Columns by Pattern?
DataFrames from real-world sources often have dozens or hundreds of columns, and you rarely need all of them. When columns follow a naming convention — prefixes like sales_, suffixes like _2023, or keyword patterns — selecting them by name pattern is far more maintainable than listing every column explicitly. If the schema changes, pattern-based selection adapts automatically.
Pandas provides two primary tools: the filter() method and list comprehensions on df.columns.
import pandas as pd
df = pd.DataFrame({
'sales_jan': [100, 200],
'sales_feb': [150, 250],
'cost_jan': [50, 80],
'cost_feb': [60, 90],
'profit': [140, 280]
})
print(df.columns.tolist())
# ['sales_jan', 'sales_feb', 'cost_jan', 'cost_feb', 'profit']filter(like=) for Substring Matching
DataFrame.filter(like='substring') selects columns whose names contain the given substring anywhere in the name (case-sensitive). It operates on the column axis by default and returns a new DataFrame with only the matching columns.
This is the simplest pattern-selection tool: no regex knowledge required, just a substring to look for.
import pandas as pd
df = pd.DataFrame({
'sales_jan': [100, 200],
'sales_feb': [150, 250],
'cost_jan': [50, 80],
'revenue_q1': [110, 210]
})
# Select columns whose name contains 'sales'
sales_cols = df.filter(like='sales')
print(sales_cols)
# sales_jan sales_feb
# 0 100 150
# 1 200 250filter(regex=) for Pattern Matching
DataFrame.filter(regex='pattern') selects columns whose names match the given regular expression. This is more powerful than like= because regex lets you match prefixes, suffixes, digit positions, or complex patterns. The regex is matched anywhere in the column name (it is not anchored to the start).
import pandas as pd
df = pd.DataFrame({
'q1_revenue': [100],
'q2_revenue': [200],
'q1_cost': [40],
'q2_cost': [60],
'annual_total': [360]
})
# Select columns ending with '_revenue'
revenue_cols = df.filter(regex='_revenue$')
print(revenue_cols)
# q1_revenue q2_revenue
# 0 100 200
# Select columns starting with 'q' (quarter columns)
quarter_cols = df.filter(regex='^q')
print(quarter_cols.columns.tolist())
# ['q1_revenue', 'q2_revenue', 'q1_cost', 'q2_cost']List Comprehension on df.columns
For maximum flexibility, iterate over df.columns with a list comprehension and apply any Python string method to build your column list. This works for startswith, endswith, contains, string length checks, or any custom logic that regex alone cannot express.
import pandas as pd
df = pd.DataFrame({
'age': [25, 30],
'age_group': ['young', 'mid'],
'income': [50000, 70000],
'income_tax': [8000, 12000],
'name': ['Alice', 'Bob']
})
# Select columns that start with 'income'
income_cols = [c for c in df.columns if c.startswith('income')]
print(df[income_cols])
# income income_tax
# 0 50000 8000
# 1 70000 12000Selecting by Column Name Suffix
The str.endswith() method on the column Index is the cleanest way to select columns by suffix. Pandas Index objects support .str accessor methods, so you can apply string operations across all column names at once and use the resulting boolean array to slice the DataFrame.
import pandas as pd
df = pd.DataFrame({
'revenue_2022': [100],
'cost_2022': [40],
'revenue_2023': [150],
'cost_2023': [55],
'notes': ['ok']
})
# Use Index.str.endswith for suffix selection
cols_2023 = df.columns[df.columns.str.endswith('_2023')]
print(df[cols_2023])
# revenue_2023 cost_2023
# 0 150 55Dropping Columns by Pattern
Sometimes it is easier to exclude a set of columns rather than name the ones to keep. Combine pattern selection with drop() or use the complement: build a list of columns to drop, then call df.drop(columns=cols_to_drop). Alternatively, get the columns you don't want and pass them to drop.
import pandas as pd
df = pd.DataFrame({
'id': [1, 2],
'name': ['A', 'B'],
'temp_col1': [0, 0],
'temp_col2': [0, 0],
'score': [90, 80]
})
# Drop columns whose name starts with 'temp_'
temp_cols = [c for c in df.columns if c.startswith('temp_')]
cleaned = df.drop(columns=temp_cols)
print(cleaned)
# id name score
# 0 1 A 90
# 1 2 B 80Selecting Numeric Columns with select_dtypes
When you want columns of a particular data type rather than a name pattern, use df.select_dtypes(include=). Passing include='number' selects all numeric columns (int and float), include='object' selects string columns, and include='datetime' selects datetime columns.
This is especially useful before applying numeric aggregations — you never accidentally call mean() on a text column.
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob'],
'age': [25, 30],
'salary': [50000.0, 70000.0],
'dept': ['Eng', 'HR']
})
# Select only numeric columns
numeric_df = df.select_dtypes(include='number')
print(numeric_df)
# age salary
# 0 25 50000.0
# 1 30 70000.0
# Exclude numeric columns
text_df = df.select_dtypes(exclude='number')
print(text_df)
# name dept
# 0 Alice Eng
# 1 Bob HRfilter() on Row Index with axis=0
By default, filter() operates on columns (axis=1). But you can also filter rows by index label by passing axis=0. This is useful when your DataFrame has meaningful string index labels and you want to select rows whose index label matches a pattern — a less common but handy trick.
import pandas as pd
df = pd.DataFrame({
'value': [10, 20, 30, 40]
}, index=['alpha_a', 'beta_b', 'alpha_c', 'gamma_d'])
# Filter rows by index label pattern
alpha_rows = df.filter(like='alpha', axis=0)
print(alpha_rows)
# value
# alpha_a 10
# alpha_c 30Combining Column Selection Patterns
Real column selection tasks often combine multiple strategies: regex to find a base set, then further filtering with a list comprehension. Building the column list in steps and inspecting it before slicing prevents silent bugs where you accidentally drop needed columns or include unwanted ones.
import pandas as pd
df = pd.DataFrame(columns=[
'user_id', 'user_name', 'user_email',
'product_id', 'product_name', 'product_price',
'order_total', 'order_date'
])
# Step 1: columns that start with 'user_' or 'order_'
step1 = [c for c in df.columns if c.startswith('user_') or c.startswith('order_')]
# Step 2: exclude email (PII)
final_cols = [c for c in step1 if 'email' not in c]
print(final_cols)
# ['user_id', 'user_name', 'order_total', 'order_date']Reordering Columns with Pattern
Pattern-based selection is also useful for reordering columns. Select the columns you want first (e.g., ID and metadata), then append the remaining columns in a specific order. Reordering makes DataFrames easier to read and ensures outputs have a consistent structure.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'metric_b': [1], 'metric_a': [2],
'id': [10], 'label': ['x'],
'metric_c': [3]
})
# Put id and label first, then sort metric columns alphabetically
id_cols = ['id', 'label']
metric_cols = sorted([c for c in df.columns if c.startswith('metric')])
ordered = df[id_cols + metric_cols]
print(ordered.columns.tolist())
# ['id', 'label', 'metric_a', 'metric_b', 'metric_c']Practical Example: Wide Survey Data
A classic use case for pattern selection is a wide survey dataset where each question generates multiple columns like q1_score, q1_text, q2_score, etc. You can extract all score columns with one regex, compute their mean, and keep the analysis columns separate from free-text responses.
import pandas as pd
survey = pd.DataFrame({
'respondent_id': [1, 2, 3],
'q1_score': [4, 3, 5],
'q2_score': [3, 4, 4],
'q3_score': [5, 5, 3],
'q1_comment': ['good', 'ok', 'great'],
'q2_comment': ['fine', 'nice', 'meh']
})
# All score columns
score_cols = survey.filter(regex='_score$').columns
survey['avg_score'] = survey[score_cols].mean(axis=1)
print(survey[['respondent_id', 'avg_score']])
# respondent_id avg_score
# 0 1 4.000000
# 1 2 4.000000
# 2 3 4.000000Quick Check
Test your understanding of selecting columns by pattern.
Lesson Recap
In this lesson you learned: filter(like=) selects columns containing a substring, filter(regex=) uses regular expressions for flexible pattern matching, and list comprehensions on df.columns with Python string methods offer maximum flexibility. Use select_dtypes() to select by data type. Next up we explore detecting missing values (NaN) in DataFrames.
Frequently asked questions
Is the “Selecting Columns by Pattern” lesson free?
Yes — the full text of “Selecting Columns by Pattern” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “Selecting Columns by Pattern”?
Use filter(like=), filter(regex=), and list comprehensions to select columns matching a name pattern. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Selecting Columns by Pattern” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Boolean Indexing on DataFrames
- The query() Method
- isin() and between() Filters
- Selecting Columns by Pattern