异常值检测与处理
使用 IQR 围栏法和 Z 分数识别异常值，决定对其进行截断、删除或标记，并记录处理决策。
异常值检测与处理 是 CoddyKit 上的免费 Pandas & NumPy Academy 课时。 这是第 2 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Pandas & NumPy Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Pandas & NumPy Academy 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
What Is an Outlier?
An outlier is a data point that differs substantially from the rest of the dataset. Outliers can be genuine (a single enterprise client with a $500,000 order in a dataset of $100 average orders) or erroneous (a negative price, or a typo that turned 120 into 12,000). The first step in outlier treatment is to decide whether the extreme value is real and meaningful or a data quality problem — the treatment differs dramatically between the two cases.
import pandas as pd
import numpy as np
df = pd.read_parquet('sales_clean.parquet')
print(df['revenue'].describe())Visual Outlier Detection: Box Plot
A box plot is the fastest visual tool for spotting outliers. The box spans the interquartile range (IQR, Q1–Q3), whiskers extend to 1.5× IQR, and points outside the whiskers are plotted individually as suspected outliers. Use df['revenue'].plot(kind='box') or Seaborn's sns.boxplot() to see the outliers immediately without computing thresholds manually.
import matplotlib.pyplot as plt
import seaborn as sns
fig, ax = plt.subplots(figsize=(6, 4))
df['revenue'].plot(kind='box', ax=ax)
ax.set_title('Revenue Distribution — Box Plot')
plt.tight_layout()
plt.show()IQR Fencing Method
The IQR fencing method computes the interquartile range and defines bounds at Q1 − 1.5×IQR and Q3 + 1.5×IQR. Values outside these bounds are flagged as outliers. The 1.5 multiplier is standard for mild outliers; use 3.0 for extreme outliers only. This method is robust: unlike Z-scores, it does not assume a normal distribution.
Q1 = df['revenue'].quantile(0.25)
Q3 = df['revenue'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = df[(df['revenue'] < lower) | (df['revenue'] > upper)]
print(f'Q1={Q1:.0f}, Q3={Q3:.0f}, IQR={IQR:.0f}')
print(f'Bounds: [{lower:.0f}, {upper:.0f}]')
print(f'Outliers: {len(outliers)}')Z-Score Method for Outlier Detection
The Z-score measures how many standard deviations a value is from the mean. A value with |Z| > 3 is conventionally considered an outlier, covering 99.7 % of normally distributed data. However, Z-scores are sensitive to the very outliers they are trying to detect — extreme values pull the mean and inflate the standard deviation, potentially masking other outliers.
mean = df['revenue'].mean()
std = df['revenue'].std()
df['revenue_z'] = (df['revenue'] - mean) / std
z_outliers = df[df['revenue_z'].abs() > 3]
print(f'Z-score outliers (|z|>3): {len(z_outliers)}')
print(z_outliers[['revenue', 'revenue_z']].head())Flagging vs. Removing Outliers
Never remove outliers without documenting and justifying the decision. Instead, first flag them with a boolean column (is_outlier), then analyse whether they share a pattern (same region, same product, same day). If they are genuine, keep them and model them explicitly. If they are errors, remove them and log the removal count in the pipeline audit trail.
df['is_revenue_outlier'] = (df['revenue'] < lower) | (df['revenue'] > upper)
print('Flagged rows:', df['is_revenue_outlier'].sum())
print(df.groupby('is_revenue_outlier')['revenue'].describe())Capping (Winsorizing) Outliers
Capping (or Winsorising) replaces values beyond the bounds with the bound value itself rather than removing the row. This preserves all rows in the dataset while reducing the distortion outliers cause in linear models and summary statistics. Use clip(lower=, upper=) to apply caps in a single vectorised operation.
df['revenue_capped'] = df['revenue'].clip(lower=lower, upper=upper)
print('Original max:', df['revenue'].max())
print('Capped max:', df['revenue_capped'].max())
print('Rows changed:', (df['revenue'] != df['revenue_capped']).sum())Log Transformation for Right-Skewed Data
Revenue and price distributions are often right-skewed with a long tail of large values. Applying a log transformation with np.log1p() (log of value + 1 to handle zeros) compresses the tail and makes the distribution more symmetric. Many statistical models and visualisations assume normality, so a log transform on the revenue column before modelling often improves results.
df['log_revenue'] = np.log1p(df['revenue'])
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
df['revenue'].hist(bins=50, ax=axes[0])
axes[0].set_title('Original Revenue')
df['log_revenue'].hist(bins=50, ax=axes[1])
axes[1].set_title('Log Revenue')
plt.tight_layout()
plt.show()Outliers in Multiple Columns
When analysing many columns at once, compute IQR outlier bounds programmatically for all numeric columns. Iterate over numeric columns, compute bounds, and store results in a dictionary. This lets you generate a full outlier report in a few lines rather than repeating the IQR calculation for every column manually.
numeric_cols = df.select_dtypes(include=['number']).columns
outlier_report = {}
for col in numeric_cols:
Q1 = df[col].quantile(0.25)
Q3 = df[col].quantile(0.75)
IQR = Q3 - Q1
n_out = ((df[col] < Q1 - 1.5*IQR) | (df[col] > Q3 + 1.5*IQR)).sum()
outlier_report[col] = n_out
print(pd.Series(outlier_report).sort_values(ascending=False))Bivariate Outliers with Scatter Plots
Some points are only outliers in combination: a unit_price of $10 is normal, a quantity of 500 is unusual but not impossible, but a unit_price of $10 with a quantity of 500 for a luxury item is suspicious. Bivariate outliers appear as isolated points in scatter plots. Use df.plot.scatter('quantity', 'unit_price') to spot points distant from the main cluster.
fig, ax = plt.subplots(figsize=(8, 5))
df.plot.scatter(x='quantity', y='unit_price', alpha=0.3, ax=ax)
ax.set_title('Quantity vs. Unit Price — Bivariate Outliers')
plt.tight_layout()
plt.show()Documenting Outlier Decisions
Every outlier decision should be logged: the column, the detection method, the threshold used, the number of rows flagged, and the treatment applied (keep, cap, or remove). This documentation protects the analyst in code reviews and audits. Store it in a cleaning log dictionary that is saved alongside the output dataset.
outlier_log = {
'column': 'revenue',
'method': 'IQR 1.5x',
'lower_bound': round(lower, 2),
'upper_bound': round(upper, 2),
'rows_flagged': int(df['is_revenue_outlier'].sum()),
'treatment': 'cap (winsorise)'
}
for k, v in outlier_log.items():
print(f'{k}: {v}')Saving the Treated Dataset
After flagging and treating outliers, save the updated DataFrame. Keep the is_outlier flag column in the output so downstream users can choose to exclude flagged rows for specific analyses. Save the capped version in a column with a clear suffix (_capped) alongside the original so the cleaning is reversible.
cols_to_save = [c for c in df.columns if c not in ['revenue_z']]
df[cols_to_save].to_parquet('sales_treated.parquet', index=False)
print('Outlier-treated dataset saved:', df.shape)Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: detecting outliers with IQR fencing and Z-scores, deciding whether to flag, cap, or remove outliers, and applying log transformations to right-skewed distributions. Next up we explore standardising inconsistent category labels in text columns.
常见问题解答
「异常值检测与处理」课时是免费的吗？
是的 — 「异常值检测与处理」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Pandas & NumPy Academy 课程的其余内容，请升级到 CoddyKit PRO。 Pandas & NumPy Academy 课程共包含 4 节课。
「异常值检测与处理」这节课中我会学到什么？
使用 IQR 围栏法和 Z 分数识别异常值，决定对其进行截断、删除或标记，并记录处理决策。 你通过在浏览器中直接运行的动手代码来练习 Pandas & NumPy Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 Pandas & NumPy Academy 需要有经验吗？
无需任何先前经验。CoddyKit 上的 Pandas & NumPy Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 2 节课，共 4 节。
「异常值检测与处理」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 Pandas & NumPy Academy 课中编写并运行代码吗？
能。每节 Pandas & NumPy Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。