Outlier Detection and Removal
Z-score method, IQR method, isolation forest concept, handling outliers in practice.
Outlier Detection and Removal is a free Learn AI with Python lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What is an Outlier?
An outlier is a value far from the rest of the data. It may be a genuine extreme, a data-entry error, or a sensor glitch. Outliers can distort means, variances, and models, so detect them deliberately.
The Z-Score Method
A z-score measures how many standard deviations a value sits from the mean: z = (x - mean) / std. A common rule flags any point with |z| > 3 as an outlier.
import numpy as np
data = np.array([10, 12, 11, 13, 12, 100])
z = (data - data.mean()) / data.std()
print(np.round(z, 2))Flagging with scipy zscore
scipy.stats.zscore computes z-scores for an array. Combine with a threshold to build a boolean mask of outliers.
from scipy import stats
z = np.abs(stats.zscore(data))
outliers = z > 3
print(data[outliers])Z-Score Limitation
The z-score uses the mean and std, which are themselves dragged by outliers. With heavily skewed data or extreme values, z-scores can under-detect. The IQR method is more robust.
The IQR Method
The interquartile range IQR = Q3 - Q1 spans the middle 50 percent. It is computed from quartiles, which outliers barely move, making it robust.
q1 = np.percentile(data, 25)
q3 = np.percentile(data, 75)
iqr = q3 - q1
print(q1, q3, iqr)IQR Fences
Tukey's fences define bounds: lower = Q1 - 1.5*IQR, upper = Q3 + 1.5*IQR. Anything outside is flagged. The same rule draws the whiskers of a boxplot.
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
mask = (data < lower) | (data > upper)
print(data[mask]) # outliersBoxplot Visualization
A boxplot visualizes the IQR rule: the box is Q1 to Q3, the line is the median, whiskers reach the fences, and points beyond are drawn as individual outlier dots.
# import matplotlib.pyplot as plt
# plt.boxplot(data)
# plt.show()Removing Outliers
Once flagged, you can DROP outlier rows. Do this cautiously, removing real data can bias results. Keep a record of what you removed and why.
clean = data[~mask]
print(clean) # outlier removedWinsorizing with np.clip
Instead of deleting, winsorizing caps extreme values at chosen limits with np.clip. This keeps the row count while taming the influence of extremes.
capped = np.clip(data, lower, upper)
print(capped) # extreme value pulled to the upper fenceClip to Percentiles
A common winsorizing choice clips to the 1st and 99th percentiles, removing only the most extreme 2 percent of values symmetrically.
lo, hi = np.percentile(data, [1, 99])
print(np.clip(data, lo, hi))Remove or Cap?
Choose by context: REMOVE clear errors (impossible values), CAP genuine-but-extreme values you want to retain, and sometimes KEEP outliers when they are the signal (fraud detection). Document the decision.
Quick Check
Test your outlier knowledge.
Recap
Outlier handling toolkit:
- Z-score: flag
|z| > 3viascipy.stats.zscore(sensitive to outliers) - IQR fences: Q1-1.5*IQR to Q3+1.5*IQR (robust, basis of boxplots)
- Remove genuine errors; winsorize with
np.clipto cap extremes - Always document why values were removed or capped
Frequently asked questions
Is the “Outlier Detection and Removal” lesson free?
Yes — the full text of “Outlier Detection and Removal” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Outlier Detection and Removal”?
Z-score method, IQR method, isolation forest concept, handling outliers in practice. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Outlier Detection and Removal” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Outlier Detection and Removal
- Encoding Categorical Variables
- Feature Scaling: Normalization and Standardization
- Building Preprocessing Pipelines