AI Prompt Engineering · Lesson

Evaluating the Decision

Measuring quality and cost.

Evaluating the Decision is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

You Cannot Decide What You Cannot Measure

The prompt-vs-tune-vs-hybrid choice is only as good as the evaluation behind it. Without a frozen eval set and a cost model, every comparison is anecdote.

Quality and cost are two axes - never collapse them into one number prematurely
The eval set must be held out and stable across every approach you compare
The winner is the approach on the best point of the quality-cost frontier for your constraints

Build the Frozen Eval Set First

Before comparing anything, construct a held-out eval set that no approach trains on. It must cover the real distribution: common cases, known edge cases, and adversarial inputs in roughly production proportions.

Freeze it. Every approach - prompt-only, tuned, hybrid - is scored on the identical set. If the eval shifts between comparisons, the numbers are not comparable and the decision is invalid.

def split_eval(labeled, holdout_ratio=0.2, seed=42):
    import random
    rng = random.Random(seed)        # fixed seed = reproducible split
    data = labeled[:]
    rng.shuffle(data)
    cut = int(len(data) * (1 - holdout_ratio))
    train, frozen_eval = data[:cut], data[cut:]
    return train, frozen_eval        # eval never enters any training run

Choose Metrics That Match the Task

Generic accuracy hides task-specific failure. Pick metrics that capture what actually matters:

Exact/schema match for structured output
Rubric-scored LLM-as-judge for open-ended quality, with a human-audited sample
Tail metrics - worst-case and p95, not just the mean
Safety/refusal rates as hard gates, scored separately from quality

A mean score that hides a catastrophic tail will lead you to the wrong decision.

Score Every Candidate Identically

Run prompt-only, tuned, and hybrid through the same scorer on the same frozen set. Record quality plus the full cost vector for each, so comparisons are apples to apples.

def evaluate(candidate, frozen_eval, scorer):
    results = []
    for ex in frozen_eval:
        out = candidate.run(ex['input'])
        results.append(scorer(out, ex['label']))
    mean = sum(results) / len(results)
    p95 = sorted(results)[int(0.95 * len(results)) - 1]
    return {'mean': mean, 'p95_worst': p95}

# Identical frozen_eval + scorer for prompt / tuned / hybrid

Model the Full Cost Vector

Cost is not one number. Capture every component so the comparison reflects reality at your volume:

Per-call inference: input + output tokens times price (long prompts cost more per call)
Amortized training: tuning cost spread over expected request volume
Maintenance: data pipeline, eval runs, re-tuning on base-model churn
Latency: priced separately when it affects conversion or UX

def monthly_cost(calls, in_tok, out_tok, price_in, price_out,
                 train_cost=0.0, months_amortized=12):
    inference = calls * ((in_tok/1000)*price_in + (out_tok/1000)*price_out)
    amortized_train = train_cost / months_amortized
    return inference + amortized_train

# Long prompt-only: high in_tok, train_cost=0
# Tuned: low in_tok, train_cost>0 amortized over volume

Plot the Quality-Cost Frontier

With quality and monthly cost for each candidate, place them on a frontier. A candidate is dominated if another has both higher quality and lower cost; drop the dominated ones.

Among the non-dominated set, the right choice depends on your constraint: pick the cheapest that clears the quality bar, or the highest quality within the cost ceiling. The decision is now explicit and defensible, not a matter of preference.

def non_dominated(candidates):
    # candidate: {'name','quality','cost'} -- higher quality, lower cost better
    keep = []
    for c in candidates:
        dominated = any(o['quality'] >= c['quality'] and o['cost'] <= c['cost']
                        and o != c for o in candidates)
        if not dominated:
            keep.append(c)
    return keep

Statistical Significance, Not Noise

A two-point lift on a 200-example eval may be noise. Before declaring a winner, check that the quality gap is statistically meaningful given your eval size.

Use a paired comparison (same examples through both candidates) and a confidence interval on the difference. If the interval crosses zero, you do not have a real improvement - and tuning's extra cost is unjustified.

def paired_diff_ci(scores_a, scores_b):
    import statistics
    diffs = [a - b for a, b in zip(scores_a, scores_b)]
    mean = statistics.mean(diffs)
    sd = statistics.pstdev(diffs)
    se = sd / (len(diffs) ** 0.5)
    return (mean - 1.96*se, mean + 1.96*se)  # if it spans 0 -> not significant

Guard Against Eval Leakage

The fastest way to make tuning look falsely good is leakage - training examples that overlap the eval set. A leaked eval rewards memorization and inflates the tuned candidate's score.

De-duplicate across the train/eval boundary, check for near-duplicates, and prefer a temporally separated eval (held out by date) so the tuned model cannot have seen it. Leakage is the single most common cause of a tuning decision that fails in production.

Monitor After You Ship

The decision is not final at launch. Production distribution drifts, and a tuned model can silently degrade as inputs move away from its training distribution.

Sample live traffic and score it against the same rubric
Alert on quality drops and on cost-per-call creep
Re-run the frozen eval whenever the base model version changes

Treat the chosen approach as a hypothesis under continuous test, not a closed decision.

Decision Record

Capture the comparison as a written decision record: the frozen eval, each candidate's quality and cost vector, the significance result, the assumed volume, and the chosen point on the frontier with its rationale.

This makes the choice auditable and re-evaluable. When volume or the base model changes, you re-open the record and re-run rather than re-litigating from memory.

End-to-End Decision Function

Tie it together: score each candidate on the frozen eval, attach its cost, drop dominated options, require significance over the cheapest baseline, then select against your binding constraint.

def decide(candidates, quality_bar, cost_ceiling):
    frontier = non_dominated(candidates)
    feasible = [c for c in frontier
                if c['quality'] >= quality_bar and c['cost'] <= cost_ceiling]
    if not feasible:
        return 'NO_CANDIDATE_MEETS_CONSTRAINTS'
    # cheapest option that clears the quality bar
    return min(feasible, key=lambda c: c['cost'])['name']

# Prefer prompt-only on ties: lower maintenance TCO

Quick Check

A tuned model scores 2 points higher than prompting on a 150-example eval, but a paired confidence interval on the difference spans zero. It also costs more per month. What is the right call?

Recap

Decide on a frozen eval and an honest cost vector, not intuition. Quality and cost are two axes; the answer is a point on the quality-cost frontier chosen against your binding constraint.

Freeze one eval set; score every candidate identically on it
Pick task-matched metrics and watch the tail, not just the mean
Model the full cost vector and amortize training over real volume
Require statistical significance; a CI spanning zero is no improvement
Guard against eval leakage - the top cause of false tuning wins
Monitor post-launch and record the decision so it can be re-run

Frequently asked questions

Is the “Evaluating the Decision” lesson free?

Yes — the full text of “Evaluating the Decision” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Evaluating the Decision”?

Measuring quality and cost. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Evaluating the Decision” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering