Fairness Metrics: Demographic Parity and Equal Opportunity
Learners will measure demographic parity difference and equal opportunity difference on a loan approval model, and identify whether the model disadvantages protected groups.
Why Fairness in ML Matters
Machine learning models trained on historical data can perpetuate and amplify existing societal biases. A loan approval model trained on past approvals may systematically deny credit to protected groups not because of their creditworthiness but because of discriminatory historical patterns in the training data. Fairness metrics quantify the extent to which a model treats different demographic groups differently, enabling auditors and engineers to detect and correct discriminatory behaviour.
Protected Attributes and Groups
A protected attribute is a demographic characteristic that should not influence high-stakes decisions — typically race, gender, age, religion, or disability status. A protected group is the subset of examples sharing a value of that attribute (e.g., female applicants). Fairness analysis compares model outcomes across these groups. Even if the protected attribute is removed from the feature set, proxy features (e.g., zip code) can still encode it.
import pandas as pd
import numpy as np
# Toy loan dataset
np.random.seed(42)
df = pd.DataFrame({
'income': np.random.normal(50000, 15000, 1000),
'credit_score': np.random.normal(650, 80, 1000),
'gender': np.random.choice(['M', 'F'], 1000),
'approved': np.random.choice([0, 1], 1000, p=[0.4, 0.6])
})
# Simulate a biased approval: female applicants approved 10% less
df.loc[df['gender'] == 'F', 'approved'] = (
df.loc[df['gender'] == 'F', 'approved'] * 0.85
).round().astype(int).clip(0, 1)
print(df.groupby('gender')['approved'].mean())All lessons in this course
- SHAP Values: Global and Local Feature Importance
- LIME: Local Interpretable Model-Agnostic Explanations
- Fairness Metrics: Demographic Parity and Equal Opportunity
- Bias Mitigation Strategies: Pre-processing, In-processing, and Post-processing