Détection et traitement des valeurs aberrantes

An outlier is a data point that differs substantially from the rest of the dataset. Outliers can be genuine (a single enterprise client with a $500,000 order in a dataset of $100 average orders) or erroneous (a negative price, or a typo that turned 120 into 12,000). The first step in outlier treatment is to decide whether the extreme value is real and meaningful or a data quality problem — the treatment differs dramatically between the two cases.

A box plot is the fastest visual tool for spotting outliers. The box spans the interquartile range (IQR, Q1–Q3), whiskers extend to 1.5× IQR, and points outside the whiskers are plotted individually as suspected outliers. Use df['revenue'].plot(kind='box') or Seaborn's sns.boxplot() to see the outliers immediately without computing thresholds manually.

The IQR fencing method computes the interquartile range and defines bounds at Q1 − 1.5×IQR and Q3 + 1.5×IQR. Values outside these bounds are flagged as outliers. The 1.5 multiplier is standard for mild outliers; use 3.0 for extreme outliers only. This method is robust: unlike Z-scores, it does not assume a normal distribution.

The Z-score measures how many standard deviations a value is from the mean. A value with |Z| > 3 is conventionally considered an outlier, covering 99.7 % of normally distributed data. However, Z-scores are sensitive to the very outliers they are trying to detect — extreme values pull the mean and inflate the standard deviation, potentially masking other outliers.

Never remove outliers without documenting and justifying the decision. Instead, first flag them with a boolean column (is_outlier), then analyse whether they share a pattern (same region, same product, same day). If they are genuine, keep them and model them explicitly. If they are errors, remove them and log the removal count in the pipeline audit trail.

Capping (or Winsorising) replaces values beyond the bounds with the bound value itself rather than removing the row. This preserves all rows in the dataset while reducing the distortion outliers cause in linear models and summary statistics. Use clip(lower=, upper=) to apply caps in a single vectorised operation.

Revenue and price distributions are often right-skewed with a long tail of large values. Applying a log transformation with np.log1p() (log of value + 1 to handle zeros) compresses the tail and makes the distribution more symmetric. Many statistical models and visualisations assume normality, so a log transform on the revenue column before modelling often improves results.

When analysing many columns at once, compute IQR outlier bounds programmatically for all numeric columns. Iterate over numeric columns, compute bounds, and store results in a dictionary. This lets you generate a full outlier report in a few lines rather than repeating the IQR calculation for every column manually.

Some points are only outliers in combination: a unit_price of $10 is normal, a quantity of 500 is unusual but not impossible, but a unit_price of $10 with a quantity of 500 for a luxury item is suspicious. Bivariate outliers appear as isolated points in scatter plots. Use df.plot.scatter('quantity', 'unit_price') to spot points distant from the main cluster.

Every outlier decision should be logged: the column, the detection method, the threshold used, the number of rows flagged, and the treatment applied (keep, cap, or remove). This documentation protects the analyst in code reviews and audits. Store it in a cleaning log dictionary that is saved alongside the output dataset.

After flagging and treating outliers, save the updated DataFrame. Keep the is_outlier flag column in the output so downstream users can choose to exclude flagged rows for specific analyses. Save the capped version in a column with a clear suffix (_capped) alongside the original so the cleaning is reversible.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: detecting outliers with IQR fencing and Z-scores, deciding whether to flag, cap, or remove outliers, and applying log transformations to right-skewed distributions. Next up we explore standardising inconsistent category labels in text columns.