0Pricing
Machine Learning Academy · Lesson

Project Scoping: Defining the Problem and Success Criteria

Learners will write a one-page project charter that specifies the prediction target, success metrics, data sources, and deployment constraints before writing any code.

Project Scoping: Defining the Problem and Success Criteria is a free Machine Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Scoping Is the Most Important Step

The single most common reason ML projects fail is not a poorly chosen algorithm — it is a poorly defined problem. Project scoping translates a vague business need ('we want to use AI') into a precise, measurable specification that guides every subsequent technical decision. A well-scoped project defines the prediction target, success criteria, data requirements, and deployment constraints before writing a single line of modelling code.

Choosing the Right ML Formulation

The first scoping decision is the ML task type. Ask: is the output a number (regression), a category (classification), a ranking (learning to rank), a cluster (unsupervised), or a sequence (NLP/time-series)? Each leads to different algorithms, data requirements, and metrics. For example, 'predict customer churn' is a binary classification problem, not a regression problem, even though the business ultimately cares about revenue.

# Scoping checklist — ML task type
questions = [
    'What is the exact output the model should produce?',
    'Is the output a number (regression) or a category (classification)?',
    'Do we need probabilities, or just labels?',
    'Is there a natural ordering in the labels (ordinal vs nominal)?',
    'Is this supervised (labelled data exists) or unsupervised?',
    'Is time ordering important (time-series)?'
]
for q in questions:
    print('•', q)

Defining the Prediction Target

The prediction target (label or dependent variable) must be defined unambiguously. 'Will this customer churn?' is ambiguous: within what time window? Using what definition of churn (cancellation, inactivity, refund)? A precise target definition might be: 'will the customer not make any purchase in the 90 days following their last purchase?' Each word matters because it determines which rows in the database become training examples.

# Example: formalising target definition in code
import pandas as pd

def label_churn(df, observation_date, prediction_window_days=90):
    '''
    Label a customer as churned (1) if they made no purchase
    in the prediction_window_days after observation_date.
    '''
    cutoff = pd.Timestamp(observation_date)
    window_end = cutoff + pd.Timedelta(days=prediction_window_days)
    churned = (
        df.groupby('customer_id')['purchase_date']
        .apply(lambda dates: not any((dates > cutoff) & (dates <= window_end)))
        .reset_index(name='churned')
    )
    return churned

Identifying Data Sources and Feasibility

After defining the target, audit available data. Ask: does labelled historical data exist? How far back does it go — is that enough training examples? Are the features available at prediction time (not only after the event)? A feature that is only recorded after the outcome (e.g., 'complaint submitted') is a data leakage risk. Map each candidate feature to its source, collection frequency, and availability latency before committing to it.

# Data audit template
data_sources = [
    {'feature': 'days_since_last_purchase', 'source': 'orders DB', 'latency_hours': 1, 'leakage_risk': False},
    {'feature': 'total_spend_90d', 'source': 'orders DB', 'latency_hours': 1, 'leakage_risk': False},
    {'feature': 'support_tickets_opened', 'source': 'CRM', 'latency_hours': 24, 'leakage_risk': False},
    {'feature': 'refund_requested', 'source': 'finance DB', 'latency_hours': 1, 'leakage_risk': True}
]
import pandas as pd
audit = pd.DataFrame(data_sources)
print(audit)
print('\nFeatures with leakage risk:')
print(audit[audit['leakage_risk'] == True]['feature'].tolist())

Setting Quantitative Success Criteria

Vague success criteria ('the model should be accurate') make projects impossible to evaluate. Success criteria must be specific, measurable, achievable, relevant, and time-bound (SMART). Define a primary metric (e.g., recall ≥ 0.80 for fraud detection), a secondary metric (precision ≥ 0.70), and a business metric (reduce churn losses by 15% within 6 months). These determine when to launch and when to retrain.

# Formalised success criteria
success_criteria = {
    'primary_metric': {
        'name': 'recall (churn class)',
        'threshold': 0.80,
        'rationale': 'Missing a churner costs more than a false alert'
    },
    'secondary_metric': {
        'name': 'precision (churn class)',
        'threshold': 0.60,
        'rationale': 'Intervention budget: can only contact 40% of flagged customers'
    },
    'business_metric': {
        'name': 'retained revenue lift',
        'threshold': '+15% over no-model baseline',
        'measurement': 'A/B test over 90 days post-launch'
    }
}
for metric, spec in success_criteria.items():
    print(f'{metric}: {spec}')

Baseline Performance: The Floor to Beat

Before building any model, define a baseline — the simplest reasonable solution against which all models will be compared. For classification, the baseline is typically a DummyClassifier predicting the majority class or random predictions weighted by class frequency. For regression it is predicting the mean. Any model that cannot outperform this baseline offers zero business value and should not be deployed.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import classification_report
import numpy as np

# Simulate imbalanced dataset (10% churn rate)
np.random.seed(42)
y = np.random.choice([0, 1], size=1000, p=[0.9, 0.1])

# Majority-class baseline
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(np.zeros((1000, 1)), y)  # features irrelevant for dummy
y_dummy = dummy.predict(np.zeros((1000, 1)))
print(classification_report(y, y_dummy, target_names=['retained', 'churned']))

Deployment Constraints: Latency and Throughput

A model that is 95% accurate but takes 10 seconds per prediction is useless in a real-time recommendation system. Deployment constraints must be captured during scoping: latency (max time per prediction), throughput (predictions per second), memory footprint (mobile vs cloud), and update frequency (retraining every day vs every month). These constraints often rule out entire families of algorithms before any code is written.

# Deployment constraint spec
deployment = {
    'serving_mode': 'batch (nightly scoring of all active customers)',
    'max_latency_ms': None,   # batch: no real-time constraint
    'throughput_rps': None,
    'memory_budget_mb': 512,  # must run on a single EC2 t3.medium
    'retraining_frequency': 'weekly',
    'max_model_size_mb': 100,
    'explainability_required': True,
    'regulatory_audit_trail': True
}
for k, v in deployment.items():
    print(f'{k}: {v}')

Ethical and Legal Review Upfront

Before committing to a project, assess ethical and legal risks. Ask: does the model affect individuals' access to services, employment, or credit? Are protected attributes (race, gender, age) either directly present or recoverable from proxies? Are there GDPR, CCPA, or sector-specific regulations (HIPAA for healthcare, FCRA for credit) that apply? Identifying these issues at scoping time costs minutes; discovering them post-deployment can cost millions and destroy user trust.

# Ethics and legal checklist
ethics_checklist = [
    ('High-stakes decisions on individuals?', True,   'churn model triggers retention offers, low risk'),
    ('Protected attributes in data?',          True,   'age and region present — must audit for proxy bias'),
    ('GDPR data minimisation required?',       True,   'only collect features needed for prediction'),
    ('Right to explanation required?',         True,   'EU AI Act high-risk? Check with legal team'),
    ('Regulatory filing required?',            False,  'B2C retention model, not financial services')
]
for question, answer, note in ethics_checklist:
    print(f'[{"YES" if answer else "NO"}] {question}  -> {note}')

The One-Page Project Charter

All scoping decisions should be captured in a concise project charter — a single-page document that stakeholders, engineers, and data scientists can all read and agree on before work begins. The charter specifies: problem statement, ML task type, prediction target definition, data sources, success metrics, baseline, deployment constraints, and ethical considerations. It becomes the ground truth for scope creep discussions throughout the project.

charter_template = '''
PROJECT CHARTER
===============
Problem: Identify customers at risk of churning before they cancel.
ML Task: Binary classification (churned=1, retained=0)
Target: No purchase in 90 days after monthly observation date
Data Sources: orders DB, CRM (support tickets), product analytics
Primary Metric: Recall >= 0.80 on held-out test set
Secondary Metric: Precision >= 0.60
Baseline: DummyClassifier majority-class (recall = 0, precision = NaN)
Deployment: Batch scoring, weekly retraining, 100 MB model limit
Ethical Flags: Audit for age/region proxy bias; GDPR minimisation
Timeline: MVP in 6 weeks; A/B test for 90 days; review in month 5
'''
print(charter_template)

Iterating on the Scope

Scoping is not a one-time activity. After initial data exploration (EDA) you may discover the label is impossible to compute historically, the class imbalance is worse than expected, or the most informative feature has too high a latency for real-time serving. Plan for at least one scope revision after EDA and one after a baseline model run. Version the project charter with dates to track how the problem definition evolved.

# Charter revision log (tracked in version control)
revision_log = [
    {'version': 'v1.0', 'date': '2026-06-01',
     'change': 'Initial scope: 30-day churn window'},
    {'version': 'v1.1', 'date': '2026-06-05',
     'change': 'EDA: 30-day window has only 4% churn rate (too imbalanced); widened to 90 days'},
    {'version': 'v1.2', 'date': '2026-06-10',
     'change': 'Baseline run: dummy recall=0, real-time serving infeasible (10s latency); switched to batch'}
]
for rev in revision_log:
    print(f"{rev['version']} ({rev['date']}): {rev['change']}")

From Charter to Actionable Plan

A completed charter drives a concrete work plan. Break the project into phases: Phase 1 — EDA and data quality audit; Phase 2 — feature engineering and baseline; Phase 3 — model selection and tuning; Phase 4 — evaluation and fairness audit; Phase 5 — deployment, monitoring, and A/B test. Assign owners and deadlines to each phase. This structure is the same whether your timeline is two weeks or six months.

project_phases = [
    ('Phase 1', 'EDA + data quality audit', '2026-06-01', '2026-06-07'),
    ('Phase 2', 'Feature engineering + baseline model', '2026-06-08', '2026-06-14'),
    ('Phase 3', 'Model selection + hyperparameter tuning', '2026-06-15', '2026-06-21'),
    ('Phase 4', 'Evaluation, fairness audit, model card', '2026-06-22', '2026-06-25'),
    ('Phase 5', 'Deploy + A/B test + monitoring setup', '2026-06-26', '2026-07-31')
]
print(f'{'Phase':8} | {'Deliverable':42} | Start      | End')
print('-' * 80)
for phase, desc, start, end in project_phases:
    print(f'{phase:8} | {desc:42} | {start} | {end}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: project scoping translates business needs into precise ML specifications before any code is written, success criteria must be quantitative and tied to both model metrics and business outcomes, and deployment constraints, ethical risks, and data feasibility must be assessed upfront to prevent expensive pivots later. Next up we dive into data wrangling and exploratory data analysis to understand the raw dataset and uncover quality issues.

Frequently asked questions

Is the “Project Scoping: Defining the Problem and Success Criteria” lesson free?

Yes — the full text of “Project Scoping: Defining the Problem and Success Criteria” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Project Scoping: Defining the Problem and Success Criteria”?

Learners will write a one-page project charter that specifies the prediction target, success metrics, data sources, and deployment constraints before writing any code. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Project Scoping: Defining the Problem and Success Criteria” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Project Scoping: Defining the Problem and Success Criteria
  2. Data Wrangling and Exploratory Data Analysis
  3. Model Selection Tournament: Compare Five Algorithms
  4. Packaging, Documenting, and Presenting the Final Model
← Back to Machine Learning Academy