Descriptive Stats and Normality Testing

Descriptive statistics summarise what is in your data: mean, median, standard deviation, skewness. Inferential statistics make claims about a population based on a sample: hypothesis tests, confidence intervals, p-values. SciPy's scipy.stats module bridges these worlds — it provides both descriptive measures that go beyond Pandas' describe() and the full suite of classical hypothesis tests. Combining Pandas for data manipulation and SciPy for statistical testing is the standard Python data science workflow.

Pandas describe() gives count, mean, std, min/max, and quartiles. scipy.stats adds skewness (asymmetry of the distribution) and kurtosis (tail heaviness). Skewness of 0 means symmetric; positive skewness means a longer right tail (many small values, few very large). Kurtosis of 3 (or 0 in excess form) is normal; higher values indicate heavier tails with more extreme outliers than a normal distribution would produce.

Skewness matters for choosing statistical tests and transformations. A right-skewed (positive skew) distribution has its mean larger than its median, typical of income data, reaction times, and sales amounts. A left-skewed (negative skew) distribution has its mean smaller than its median. Rule of thumb: |skewness| < 0.5 is approximately symmetric; 0.5–1.0 is moderately skewed; > 1.0 is highly skewed and may benefit from a log or square-root transformation before applying tests that assume normality.

Many statistical tests — t-tests, ANOVA, Pearson correlation — assume that the data (or the residuals) follow a normal (Gaussian) distribution. If this assumption is violated with small samples, the test's p-values may be inaccurate. Normality testing checks whether the assumption holds. For large samples (n > 100), normality tests become very sensitive and will almost always reject normality for trivial deviations — in that case, rely on the Central Limit Theorem (sample means are approximately normal) rather than testing the raw data.

The Shapiro-Wilk test (scipy.stats.shapiro(data)) is the most powerful normality test for sample sizes up to about 5,000. It returns a W-statistic and a p-value. If p > 0.05, you fail to reject normality — the data is consistent with a normal distribution. If p ≤ 0.05, you reject normality. Remember: failing to reject normality does not prove the data is normal; it just means you do not have enough evidence to say it is not.

The Kolmogorov-Smirnov (K-S) test (scipy.stats.kstest(data, 'norm', args)) tests whether a sample follows a specified distribution. Unlike Shapiro-Wilk, it works for any distribution (not just normal) and for large samples. It computes the maximum difference between the empirical CDF of your data and the theoretical CDF. A variation, the two-sample K-S test (stats.ks_2samp(a, b)), tests whether two samples come from the same distribution — useful for comparing before/after distributions.

The Q-Q (quantile-quantile) plot is a visual normality check: it plots the quantiles of your data against the quantiles of a normal distribution. If the data is normal, the points fall on a straight diagonal line. Deviations from the line indicate departure from normality — S-curves indicate kurtosis, bends indicate skewness. Statistical tests tell you if deviations are significant; Q-Q plots show you the nature and location of deviations. Use scipy.stats.probplot() to generate Q-Q plot data.

The Central Limit Theorem (CLT) states that the distribution of sample means approaches normality as sample size grows, regardless of the underlying distribution. For n ≥ 30, the sample mean is approximately normal even if individual observations are skewed. This is why t-tests are robust for large samples: you are testing whether the mean differs, and the mean is approximately normal even when the raw data is not. For small samples from non-normal populations, use non-parametric tests instead.

In group comparison tests (t-test, ANOVA), normality is required within each group, not in the overall data. When testing whether sales differ by region, test the normality of sales within each region separately. A distribution that is not normal overall can still satisfy normality within subgroups. Apply stats.shapiro() to each group using Pandas groupby() and iterate over groups to check the assumption systematically.

When normality is violated, use non-parametric tests that make no distributional assumptions. The Mann-Whitney U test (stats.mannwhitneyu) replaces the independent t-test; the Wilcoxon signed-rank test (stats.wilcoxon) replaces the paired t-test; the Kruskal-Wallis test (stats.kruskal) replaces one-way ANOVA. These tests use ranks instead of raw values and are robust to outliers, but they have less statistical power than their parametric counterparts when normality actually holds.

In EDA, you often need to check normality and skewness for all numeric columns at once. Combine Pandas select_dtypes with a loop over columns calling stats.shapiro() and stats.skew(). Build a summary DataFrame with columns for skewness, kurtosis, and Shapiro p-value to get a birds-eye view of which columns are non-normal and need transformation before modelling. This is part of a systematic data quality checklist.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: scipy.stats.skew() and kurtosis() describe the shape of a distribution beyond what describe() provides, scipy.stats.shapiro() tests normality with p > 0.05 meaning no evidence against normality, and the Central Limit Theorem makes t-tests robust for large samples even with non-normal data. Next up we apply hypothesis testing with t-tests to compare group means.