Outlier Detection and Treatment
Identify outliers with IQR fencing and Z-scores, decide whether to cap, remove, or flag them, and document decisions.
Outlier Detection and Treatment is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is an Outlier?
An outlier is a data point that differs substantially from the rest of the dataset. Outliers can be genuine (a single enterprise client with a $500,000 order in a dataset of $100 average orders) or erroneous (a negative price, or a typo that turned 120 into 12,000). The first step in outlier treatment is to decide whether the extreme value is real and meaningful or a data quality problem — the treatment differs dramatically between the two cases.
import pandas as pd
import numpy as np
df = pd.read_parquet('sales_clean.parquet')
print(df['revenue'].describe())Visual Outlier Detection: Box Plot
A box plot is the fastest visual tool for spotting outliers. The box spans the interquartile range (IQR, Q1–Q3), whiskers extend to 1.5× IQR, and points outside the whiskers are plotted individually as suspected outliers. Use df['revenue'].plot(kind='box') or Seaborn's sns.boxplot() to see the outliers immediately without computing thresholds manually.
import matplotlib.pyplot as plt
import seaborn as sns
fig, ax = plt.subplots(figsize=(6, 4))
df['revenue'].plot(kind='box', ax=ax)
ax.set_title('Revenue Distribution — Box Plot')
plt.tight_layout()
plt.show()IQR Fencing Method
The IQR fencing method computes the interquartile range and defines bounds at Q1 − 1.5×IQR and Q3 + 1.5×IQR. Values outside these bounds are flagged as outliers. The 1.5 multiplier is standard for mild outliers; use 3.0 for extreme outliers only. This method is robust: unlike Z-scores, it does not assume a normal distribution.
Q1 = df['revenue'].quantile(0.25)
Q3 = df['revenue'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = df[(df['revenue'] < lower) | (df['revenue'] > upper)]
print(f'Q1={Q1:.0f}, Q3={Q3:.0f}, IQR={IQR:.0f}')
print(f'Bounds: [{lower:.0f}, {upper:.0f}]')
print(f'Outliers: {len(outliers)}')Z-Score Method for Outlier Detection
The Z-score measures how many standard deviations a value is from the mean. A value with |Z| > 3 is conventionally considered an outlier, covering 99.7 % of normally distributed data. However, Z-scores are sensitive to the very outliers they are trying to detect — extreme values pull the mean and inflate the standard deviation, potentially masking other outliers.
mean = df['revenue'].mean()
std = df['revenue'].std()
df['revenue_z'] = (df['revenue'] - mean) / std
z_outliers = df[df['revenue_z'].abs() > 3]
print(f'Z-score outliers (|z|>3): {len(z_outliers)}')
print(z_outliers[['revenue', 'revenue_z']].head())Flagging vs. Removing Outliers
Never remove outliers without documenting and justifying the decision. Instead, first flag them with a boolean column (is_outlier), then analyse whether they share a pattern (same region, same product, same day). If they are genuine, keep them and model them explicitly. If they are errors, remove them and log the removal count in the pipeline audit trail.
df['is_revenue_outlier'] = (df['revenue'] < lower) | (df['revenue'] > upper)
print('Flagged rows:', df['is_revenue_outlier'].sum())
print(df.groupby('is_revenue_outlier')['revenue'].describe())Capping (Winsorizing) Outliers
Capping (or Winsorising) replaces values beyond the bounds with the bound value itself rather than removing the row. This preserves all rows in the dataset while reducing the distortion outliers cause in linear models and summary statistics. Use clip(lower=, upper=) to apply caps in a single vectorised operation.
df['revenue_capped'] = df['revenue'].clip(lower=lower, upper=upper)
print('Original max:', df['revenue'].max())
print('Capped max:', df['revenue_capped'].max())
print('Rows changed:', (df['revenue'] != df['revenue_capped']).sum())Log Transformation for Right-Skewed Data
Revenue and price distributions are often right-skewed with a long tail of large values. Applying a log transformation with np.log1p() (log of value + 1 to handle zeros) compresses the tail and makes the distribution more symmetric. Many statistical models and visualisations assume normality, so a log transform on the revenue column before modelling often improves results.
df['log_revenue'] = np.log1p(df['revenue'])
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
df['revenue'].hist(bins=50, ax=axes[0])
axes[0].set_title('Original Revenue')
df['log_revenue'].hist(bins=50, ax=axes[1])
axes[1].set_title('Log Revenue')
plt.tight_layout()
plt.show()Outliers in Multiple Columns
When analysing many columns at once, compute IQR outlier bounds programmatically for all numeric columns. Iterate over numeric columns, compute bounds, and store results in a dictionary. This lets you generate a full outlier report in a few lines rather than repeating the IQR calculation for every column manually.
numeric_cols = df.select_dtypes(include=['number']).columns
outlier_report = {}
for col in numeric_cols:
Q1 = df[col].quantile(0.25)
Q3 = df[col].quantile(0.75)
IQR = Q3 - Q1
n_out = ((df[col] < Q1 - 1.5*IQR) | (df[col] > Q3 + 1.5*IQR)).sum()
outlier_report[col] = n_out
print(pd.Series(outlier_report).sort_values(ascending=False))Bivariate Outliers with Scatter Plots
Some points are only outliers in combination: a unit_price of $10 is normal, a quantity of 500 is unusual but not impossible, but a unit_price of $10 with a quantity of 500 for a luxury item is suspicious. Bivariate outliers appear as isolated points in scatter plots. Use df.plot.scatter('quantity', 'unit_price') to spot points distant from the main cluster.
fig, ax = plt.subplots(figsize=(8, 5))
df.plot.scatter(x='quantity', y='unit_price', alpha=0.3, ax=ax)
ax.set_title('Quantity vs. Unit Price — Bivariate Outliers')
plt.tight_layout()
plt.show()Documenting Outlier Decisions
Every outlier decision should be logged: the column, the detection method, the threshold used, the number of rows flagged, and the treatment applied (keep, cap, or remove). This documentation protects the analyst in code reviews and audits. Store it in a cleaning log dictionary that is saved alongside the output dataset.
outlier_log = {
'column': 'revenue',
'method': 'IQR 1.5x',
'lower_bound': round(lower, 2),
'upper_bound': round(upper, 2),
'rows_flagged': int(df['is_revenue_outlier'].sum()),
'treatment': 'cap (winsorise)'
}
for k, v in outlier_log.items():
print(f'{k}: {v}')Saving the Treated Dataset
After flagging and treating outliers, save the updated DataFrame. Keep the is_outlier flag column in the output so downstream users can choose to exclude flagged rows for specific analyses. Save the capped version in a column with a clear suffix (_capped) alongside the original so the cleaning is reversible.
cols_to_save = [c for c in df.columns if c not in ['revenue_z']]
df[cols_to_save].to_parquet('sales_treated.parquet', index=False)
print('Outlier-treated dataset saved:', df.shape)Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: detecting outliers with IQR fencing and Z-scores, deciding whether to flag, cap, or remove outliers, and applying log transformations to right-skewed distributions. Next up we explore standardising inconsistent category labels in text columns.
Frequently asked questions
Is the “Outlier Detection and Treatment” lesson free?
Yes — the full text of “Outlier Detection and Treatment” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “Outlier Detection and Treatment”?
Identify outliers with IQR fencing and Z-scores, decide whether to cap, remove, or flag them, and document decisions. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Outlier Detection and Treatment” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Detecting and Removing Duplicates
- Outlier Detection and Treatment
- Standardising Inconsistent Categories
- Schema Validation and Assertions