データの整理と探索的データ分析

Exploratory Data Analysis (EDA) is the practice of profiling a dataset before modelling to understand its structure, distributions, relationships, and quality issues. Coined by John Tukey, EDA follows a simple principle: let the data speak. Rushing past EDA and jumping straight to modelling is the second most common reason ML projects fail, after poor problem scoping. EDA prevents you from building models on faulty foundations.

Always start with a structural inspection: how many rows, columns, and what data types? Use df.shape, df.dtypes, df.head(), and df.info(). Look for columns that appear numeric but are stored as object (strings), and columns that appear categorical but are encoded as integers. Mismatched dtypes cause silent bugs downstream when scikit-learn treats an integer-encoded category as a continuous variable.

Univariate analysis examines one variable at a time. For numeric features, plot histograms and compute summary statistics (mean, median, std, skewness). Highly skewed distributions (skewness > 2) often benefit from a log transform before modelling. For categorical features, plot bar charts of value counts. Categories that appear in fewer than 1% of rows may cause issues with one-hot encoding and stratified splitting.

df.describe() produces count, mean, std, min, quartiles, and max for every numeric column. Examine it carefully: a max value orders of magnitude above the 99th percentile signals outliers. A min of zero in a 'days since last purchase' column may mean the customer purchased today — or it may be a sentinel for missing data. Understanding these edge cases prevents silent errors in downstream preprocessing.

Missing values must be profiled before imputation decisions are made. Compute the percentage of nulls per column with df.isnull().mean(). Columns with more than 50% missing may need to be dropped entirely. Understand the mechanism of missingness: is it missing completely at random (MCAR), or is it missing not at random (MNAR) — e.g., customers who churned deleted their account info? MNAR missingness is itself informative and should be encoded as a binary indicator.

Bivariate analysis examines relationships between pairs of variables. For numeric-numeric relationships, compute a correlation matrix and visualise it as a heat map with seaborn.heatmap. Correlations above 0.95 between two features signal multicollinearity — both features carry almost identical information, so one can be dropped without losing predictive power. Also compute point-biserial correlations between numeric features and the binary target to identify the most predictive features upfront.

Always analyse the target variable separately. For binary classification, compute the positive class rate (df['churned'].mean()). Rates below 5% or above 95% indicate severe imbalance that affects algorithm choice and evaluation metrics. For regression targets, check normality — skewed targets often benefit from a log transform before training, especially for tree-based models which are immune, but critical for linear regression.

Outliers can distort model training, especially for distance-based models (KNN, SVM) and linear regression. Use the IQR method (flag values outside [Q1 - 1.5·IQR, Q3 + 1.5·IQR]) or Z-score (flag values with |z| > 3) for initial detection. For each flagged outlier, investigate: is it a data entry error, a legitimate extreme value, or a sensor malfunction? Legitimate extreme values should be kept; errors should be corrected or removed.

For classification tasks, visualise how each numeric feature's distribution differs between classes using a violin plot or box plot grouped by the target. Features where the distributions strongly separate the two classes are likely to be predictive. For categorical features, a grouped bar chart of churn rate per category value quickly reveals which categories are associated with higher risk.

EDA findings must be documented in a reproducible Jupyter notebook shared with the team. At minimum, document: columns dropped and why, imputation decisions per column, outlier treatment, class imbalance magnitude, top predictive features, and any data quality issues escalated to the data engineering team. This notebook becomes the data quality section of the eventual model card and is invaluable when debugging production regressions months later.

For large datasets with hundreds of features, manual EDA is impractical. ydata-profiling (formerly pandas-profiling) generates an interactive HTML report with distributions, correlations, missing values, and outlier alerts for every column with a single function call. sweetviz produces comparison reports across train/test splits or target class segments. Use these as a starting point, but always follow up with domain-specific custom plots for the most important features.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: EDA profiles distributions, relationships, missing values, and outliers before any modelling, bivariate analysis (correlations, feature-target plots) identifies the most predictive features, and EDA findings must be documented in a reproducible notebook to guide preprocessing decisions and serve as the data quality record. Next up we run a model selection tournament to compare five algorithms on the same processed dataset.