Pandas & NumPy Academy · Lesson

GroupBy Transform and Filter

Use transform() to add group-level statistics back as a column and filter() to keep only groups meeting a condition.

GroupBy Transform and Filter is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Beyond Aggregation

After mastering agg(), the natural question is: what if you want to keep all original rows but enrich them with group-level statistics? Or keep only the groups that meet a condition? This is where transform() and filter() come in. These two methods extend GroupBy beyond simple summarisation into feature engineering and data selection.

Understanding transform()

transform() applies a function to each group and returns a result with the same shape as the original DataFrame — one value per original row. The group result is broadcast back to each row that belongs to that group. This makes it ideal for adding group-level statistics as new columns without changing the row count.

import pandas as pd

df = pd.DataFrame({
    'dept':   ['Eng', 'HR', 'Eng', 'HR', 'Eng'],
    'salary': [90000, 60000, 95000, 62000, 88000]
})

# Add a column with each employee's department average salary
df['dept_avg'] = df.groupby('dept')['salary'].transform('mean')
print(df)
#    dept  salary    dept_avg
# 0   Eng   90000  91000.000
# 1    HR   60000  61000.000
# 2   Eng   95000  91000.000
# 3    HR   62000  61000.000
# 4   Eng   88000  91000.000

Common transform() Use Cases

Common applications of transform() include: adding a group mean to normalise values, adding a group sum to compute per-row percentage of total, and adding a group rank to see how each member compares within its group. All of these preserve the original shape and index, making the result immediately usable alongside the original columns.

# Percentage of each employee's salary within their department total
df['pct_of_dept'] = (
    df['salary'] / df.groupby('dept')['salary'].transform('sum') * 100
).round(1)
print(df[['dept', 'salary', 'pct_of_dept']])
# dept  salary  pct_of_dept
# Eng   90000        33.1
# HR    60000        49.2
# Eng   95000        34.9

Using a Custom Function in transform()

Just like agg(), transform() accepts any callable in addition to string names. The function receives a Series of values for one group and must return a Series of the same length, or a scalar (which is then broadcast). Returning a scalar is the most common use case — returning a Series of a different length will raise an error.

# Z-score normalisation within each department
def zscore(s):
    return (s - s.mean()) / s.std()

df['salary_zscore'] = df.groupby('dept')['salary'].transform(zscore)
print(df[['dept', 'salary', 'salary_zscore']].round(2))
# dept  salary  salary_zscore
# Eng   90000          -0.51
# HR    60000          -0.71
# Eng   95000           1.03

agg() vs transform() Side by Side

The key difference: agg() reduces the number of rows (one per group), while transform() preserves the number of rows (one per original row). Use agg() to build a summary table. Use transform() to add group-level information as a new feature column in the original DataFrame.

g = df.groupby('dept')['salary']

# agg: 2 rows (one per unique dept)
print(g.agg('mean'))
# dept
# Eng    91000.0
# HR     61000.0

# transform: 5 rows (one per original row)
print(g.transform('mean'))
# 0    91000.0
# 1    61000.0
# 2    91000.0
# 3    61000.0
# 4    91000.0

Understanding filter()

filter() keeps or discards entire groups based on a boolean function. You pass a function that receives a sub-DataFrame for one group and returns True (keep the group) or False (drop the group). The result is a subset of the original DataFrame containing only rows from the groups that passed the test.

df2 = pd.DataFrame({
    'dept':   ['Eng', 'HR', 'Eng', 'HR', 'Eng', 'Legal'],
    'salary': [90000, 60000, 95000, 62000, 88000, 70000]
})

# Keep only departments with at least 2 employees
big_depts = df2.groupby('dept').filter(lambda g: len(g) >= 2)
print(big_depts)
# dept, salary rows: Eng(3) and HR(2) remain; Legal(1) dropped

Filter by Group Aggregate Value

A very common use of filter() is keeping groups whose aggregate value meets a threshold. For example, keep only departments where the average salary exceeds a target, or keep only product categories with total sales above a minimum. This lets you remove low-volume groups before further analysis.

# Keep only departments where average salary > 80000
high_paying = df2.groupby('dept').filter(
    lambda g: g['salary'].mean() > 80000
)
print(high_paying)
#    dept  salary
# 0   Eng   90000
# 2   Eng   95000
# 4   Eng   88000
# (HR avg is 61000, filtered out)

Combining transform() and filter()

You can apply transform() and filter() sequentially to enrich your data and then narrow it down. First use filter() to remove irrelevant groups, then use transform() on the filtered result to add group-level features. The combination gives you a clean, feature-rich subset ready for modelling or reporting.

# Step 1: keep only large departments
filtered = df2.groupby('dept').filter(lambda g: len(g) >= 2)

# Step 2: add group mean salary to the filtered result
filtered = filtered.copy()
filtered['dept_avg'] = filtered.groupby('dept')['salary'].transform('mean')
print(filtered)

transform() for Forward Filling Within Groups

transform() is also useful with non-numeric functions. A popular pattern is filling missing values within a group using the group's forward fill or median, which is much better than a global fill. Pass a lambda that calls fillna() on the group Series, and the result has the same index as the original DataFrame.

import numpy as np

df3 = pd.DataFrame({
    'dept':   ['Eng', 'Eng', 'HR', 'HR', 'Eng'],
    'salary': [90000, np.nan, 60000, np.nan, 88000]
})

# Fill NaN with the group mean
df3['salary_filled'] = df3.groupby('dept')['salary'].transform(
    lambda s: s.fillna(s.mean())
)
print(df3)

Practical Pattern: Relative Standing

One powerful business use case is computing each row's standing relative to its group. By combining transform() with arithmetic, you can add columns like: salary minus group average (deviation), salary as a fraction of group total, or boolean flag for whether this employee earns above the group median. These features are extremely useful for dashboards and ML models.

df['dept_total'] = df.groupby('dept')['salary'].transform('sum')
df['pct_of_total'] = (df['salary'] / df['dept_total'] * 100).round(1)
df['above_avg'] = df['salary'] > df.groupby('dept')['salary'].transform('mean')
print(df[['dept', 'salary', 'pct_of_total', 'above_avg']])

Performance Considerations

Both transform() and filter() with built-in string functions are fast because they use optimised code paths. However, when you pass a lambda or custom Python function, Pandas must call that function once per group, which can be slow on data with many groups. For maximum performance with large datasets, check whether your custom logic can be expressed using a built-in string function instead.

# Slower: custom lambda (called once per group)
df['dept_mean_slow'] = df.groupby('dept')['salary'].transform(lambda s: s.mean())

# Faster: built-in string shortcut (vectorised C path)
df['dept_mean_fast'] = df.groupby('dept')['salary'].transform('mean')

# Both give identical results, but the built-in is significantly faster

Quick Check

Test your understanding of GroupBy transform() and filter() from this lesson.

Lesson Recap

In this lesson you learned: transform() returns group statistics broadcast back to the original row count, making it ideal for adding group-level features; filter() keeps or removes entire groups based on a boolean condition; and combining both lets you build rich, filtered datasets for downstream analysis. Next up we explore how to combine DataFrames using pd.concat.

Frequently asked questions

Is the “GroupBy Transform and Filter” lesson free?

Yes — the full text of “GroupBy Transform and Filter” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.

What will I learn in “GroupBy Transform and Filter”?

Use transform() to add group-level statistics back as a column and filter() to keep only groups meeting a condition. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Pandas & NumPy Academy?

No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “GroupBy Transform and Filter” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Pandas & NumPy Academy lesson?

Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Pandas & NumPy Academy