apply() with GroupBy
Pass a multi-row function to groupby().apply() to compute complex group-level summaries that agg() cannot express.
apply() with GroupBy is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why GroupBy Needs apply()
The built-in agg() method handles simple aggregations like sum, mean, and count — one scalar output per group. But some group-level computations require looking at the entire sub-DataFrame for the group, not just a single column. groupby().apply(func) passes the full group DataFrame to your function and collects the results, enabling complex summaries that agg() cannot express.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'region': ['North', 'North', 'South', 'South', 'North'],
'product': ['A', 'B', 'A', 'A', 'C'],
'revenue': [100, 250, 180, 90, 300]
})
print(df)Returning a Scalar per Group
When the function passed to groupby().apply() returns a scalar, the result is a Series indexed by the group keys — identical to what agg() produces. This form is useful when the scalar requires multi-column logic, such as computing the ratio of top-product revenue to total group revenue, which cannot be expressed in a single agg() column spec.
def top_product_share(group):
top = group['revenue'].max()
total = group['revenue'].sum()
return top / total
share = df.groupby('region').apply(top_product_share)
print(share)Returning a Series per Group
When the function returns a pd.Series, the result has a MultiIndex: outer level is the group key and inner level is the Series index. This is useful for computing multiple statistics per group in a single apply call, producing a summary table where each group has multiple rows of metrics.
def group_stats(group):
return pd.Series({
'total': group['revenue'].sum(),
'top_product': group.loc[group['revenue'].idxmax(), 'product'],
'n_products': group['product'].nunique()
})
result = df.groupby('region').apply(group_stats)
print(result)Returning a DataFrame per Group
When the function returns a pd.DataFrame, groupby().apply() concatenates all the sub-DataFrames vertically. This is the most powerful form: it lets you filter or transform the rows within each group and return a modified subset. For example, keep only the top-2 products by revenue within each region.
def top2_per_region(group):
return group.nlargest(2, 'revenue')
top2 = df.groupby('region').apply(top2_per_region)
print(top2.reset_index(drop=True))Computing Within-Group Z-Scores
A common use of groupby().apply() is computing within-group standardisation. Instead of normalising revenue relative to all orders (which mixes regions), compute the Z-score of revenue within each region. A Z-score of 2 in the North means "2 standard deviations above the North average" — a more meaningful benchmark than 2 SDs above the global average.
def within_group_zscore(group):
mean = group['revenue'].mean()
std = group['revenue'].std()
group = group.copy()
group['revenue_z'] = (group['revenue'] - mean) / std
return group
df_z = df.groupby('region').apply(within_group_zscore).reset_index(drop=True)
print(df_z)Custom Aggregation: Winsorised Mean
The Winsorised mean clips extreme values before averaging, making it more robust than the simple mean for skewed distributions. This custom aggregation cannot be expressed with standard agg() functions but is straightforward with groupby().apply(): clip at the 5th and 95th percentile within each group, then compute the mean on the clipped values.
def winsorised_mean(group, lower_pct=0.05, upper_pct=0.95):
col = group['revenue']
lo = col.quantile(lower_pct)
hi = col.quantile(upper_pct)
clipped = col.clip(lo, hi)
return clipped.mean()
wins_mean = df.groupby('region').apply(winsorised_mean)
print('Winsorised mean by region:')
print(wins_mean)Custom Rolling Window per Group
Rolling windows applied to the full DataFrame ignore group boundaries. If you compute a 3-row rolling mean on a time series that interleaves two products, the window crosses product boundaries incorrectly. Apply the rolling window inside a groupby().apply() to ensure each rolling calculation stays within its group — for example, a 3-month rolling revenue average per region.
def group_rolling_mean(group, window=3):
group = group.sort_values('order_date').copy()
group['rolling_revenue'] = group['revenue'].rolling(window, min_periods=1).mean()
return group
# df_ts has order_date column
# df_rolled = df_ts.groupby('region').apply(group_rolling_mean).reset_index(drop=True)
print('Rolling window per group applied inside apply()')include_groups Parameter (Pandas 2.2+)
In Pandas 2.2 and later, groupby().apply() raises a FutureWarning if the groupby keys appear in the DataFrame that the function receives. Pass include_groups=False to exclude the groupby columns from the sub-DataFrame passed to the function. This avoids both the warning and accidental operations on the key columns inside the function body.
def revenue_only(group):
# group does not include 'region' column when include_groups=False
return group['revenue'].sum()
# result = df.groupby('region').apply(revenue_only, include_groups=False)
print('include_groups=False avoids FutureWarning in Pandas 2.2+')Combining apply() Results with reset_index
When groupby().apply() returns a DataFrame per group, the result has a MultiIndex that includes the group key as the outer level. Use reset_index(drop=True) to flatten the index back to a plain integer range. Alternatively, use reset_index(level=0) to promote the group key to a column so it is explicit in the output.
result = df.groupby('region').apply(top2_per_region)
print('With MultiIndex:')
print(result.head())
print('\nAfter reset_index:')
print(result.reset_index(drop=True))When to Prefer transform() Over apply()
groupby().apply() is for complex group-level computations that return a different shape from the input. groupby().transform() is for computations that add a new column aligned to the original index — such as broadcasting the group mean back to each row for normalisation. Use transform when you need the result in the same shape as the original DataFrame; use apply when the result shape differs.
# transform: adds group mean back to each row
df['group_mean_revenue'] = df.groupby('region')['revenue'].transform('mean')
# apply: computes one scalar per group
group_totals = df.groupby('region')['revenue'].apply(sum)
print(df[['region', 'revenue', 'group_mean_revenue']])Performance Tip: Avoid apply() for Simple Aggregations
groupby().apply() is significantly slower than groupby().agg() for simple operations because apply() creates a full Python object for each group. For sums, means, and counts, always use agg(). Save apply() for cases that genuinely require the full group DataFrame — multi-column conditional logic, custom statistics, or filtering within groups.
# SLOW: apply for simple sum
slow = df.groupby('region').apply(lambda g: g['revenue'].sum())
# FAST: native agg
fast = df.groupby('region')['revenue'].sum()
print('Both produce the same result:')
print(fast.equals(slow))Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: returning scalars, Series, and DataFrames from groupby().apply(), computing within-group Z-scores and custom robust statistics, and choosing between apply(), agg(), and transform() for group-level operations. Next up we explore map() and applymap() for element-wise operations on Series and DataFrames.
Frequently asked questions
Is the “apply() with GroupBy” lesson free?
Yes — the full text of “apply() with GroupBy” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “apply() with GroupBy”?
Pass a multi-row function to groupby().apply() to compute complex group-level summaries that agg() cannot express. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “apply() with GroupBy” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.