AI Engineering Academy · Lesson

Evaluating and Deploying Your Fine-Tuned Model

Run quantitative evals comparing the base and fine-tuned model on held-out test cases, convert to GGUF for local inference, and serve via llama.cpp or vLLM.

Evaluating and Deploying Your Fine-Tuned Model is a free AI Engineering Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Evaluation Must Come Before Deployment

A fine-tuned model that performs well on training data may perform worse than the base model on your actual production use case. The only way to know is rigorous evaluation. Fine-tuning can cause catastrophic forgetting (losing capabilities the base model had), over-specialization (performing well on your task but worse on adjacent tasks), or subtle regressions in safety behaviors. Never deploy a fine-tuned model without evaluating it against the base model on a representative test set.

Building a Held-Out Test Set

Your test set must be completely separate from training and validation data — examples the model has never seen during any phase of training. The test set should represent the full distribution of production inputs: common cases, edge cases, and adversarial inputs. For instruction-following tasks, include examples that require following all aspects of the instruction, not just the most common ones. A test set of 100-500 examples is typically sufficient for reliable evaluation.

import json
from typing import TypedDict

class TestCase(TypedDict):
    input: str                 # the user message
    expected_output: str       # the ideal response
    category: str              # e.g., 'format', 'accuracy', 'edge_case'
    evaluation_method: str     # 'exact_match', 'json_schema', 'llm_judge'

# Load test set (never used during training)
def load_test_set(path: str) -> list[TestCase]:
    cases = []
    with open(path) as f:
        for line in f:
            data = json.loads(line.strip())
            cases.append({
                'input': data['messages'][-2]['content'],  # user message
                'expected_output': data['messages'][-1]['content'],  # assistant response
                'category': data.get('metadata', {}).get('category', 'general'),
                'evaluation_method': data.get('metadata', {}).get('eval_method', 'llm_judge')
            })
    return cases

test_set = load_test_set('test.jsonl')
print(f'Test set loaded: {len(test_set)} examples')

Quantitative Metrics for Task-Specific Evaluation

Choose evaluation metrics that match your task. For JSON extraction, measure schema compliance rate and field-level accuracy. For classification, measure accuracy, precision, and recall per class. For text generation, use LLM-as-judge scoring for quality. For format adherence, measure the exact format compliance rate. Run each metric on both the base model and the fine-tuned model so you can measure the improvement delta.

import json

def evaluate_json_extraction(model_output: str, expected: str, schema: dict) -> dict:
    metrics = {'valid_json': False, 'schema_compliant': False, 'field_accuracy': 0.0}
    
    try:
        parsed = json.loads(model_output.strip())
        metrics['valid_json'] = True
        
        # Check schema compliance
        required_fields = schema.get('required', [])
        all_present = all(field in parsed for field in required_fields)
        correct_types = all(
            isinstance(parsed.get(field), schema['properties'][field]['expected_type'])
            for field in required_fields if field in parsed
        )
        metrics['schema_compliant'] = all_present and correct_types
        
        # Field-level accuracy against expected output
        expected_parsed = json.loads(expected)
        correct_fields = sum(1 for k in expected_parsed if parsed.get(k) == expected_parsed[k])
        metrics['field_accuracy'] = correct_fields / len(expected_parsed) if expected_parsed else 0.0
    
    except json.JSONDecodeError:
        pass  # valid_json stays False
    
    return metrics

LLM-as-Judge Evaluation

For open-ended generation tasks, use an LLM-as-judge to score the fine-tuned model's outputs against the expected outputs. Prompt GPT-4o to act as an evaluator, provide the input, expected output, and model output, and ask it to rate correctness, completeness, and format compliance on a 1-5 scale. Average the scores across the test set to get an overall quality rating. Compare the fine-tuned model's score to the base model's score on the same test set.

from openai import OpenAI

client = OpenAI()

def llm_judge_score(instruction: str, expected: str, actual: str) -> dict:
    judge_prompt = f'''Evaluate the quality of an AI assistant response.

Instruction given to assistant:
{instruction}

Expected ideal response:
{expected}

Actual response from model being evaluated:
{actual}

Rate the actual response on these criteria (1=poor, 5=excellent):
1. Correctness: Is the information accurate?
2. Format compliance: Does it follow the expected output format?
3. Completeness: Does it address all parts of the instruction?

Return JSON: {{"correctness": N, "format": N, "completeness": N, "overall": N, "reason": "brief explanation"}}'''
    
    response = client.chat.completions.create(
        model='gpt-4o',
        messages=[{'role': 'user', 'content': judge_prompt}],
        response_format={'type': 'json_object'}
    )
    return json.loads(response.choices[0].message.content)

Running the Comparison Evaluation

Run a head-to-head evaluation comparing base model, fine-tuned model (and optionally, a strong prompting baseline) on all test cases. Generate outputs from each model on every test case, then score all outputs with your evaluation metrics. Produce a comparison table showing metric scores, standard deviations, and examples of where the fine-tuned model wins and loses compared to the baseline.

def run_full_evaluation(test_set: list, models: dict, system_prompt: str) -> dict:
    results = {name: {'scores': [], 'errors': 0} for name in models}
    
    for i, test_case in enumerate(test_set):
        print(f'Evaluating test case {i+1}/{len(test_set)}')
        
        for model_name, model_fn in models.items():
            try:
                output = model_fn(test_case['input'], system_prompt)
                score = llm_judge_score(
                    test_case['input'],
                    test_case['expected_output'],
                    output
                )
                results[model_name]['scores'].append(score['overall'])
            except Exception as e:
                results[model_name]['errors'] += 1
                results[model_name]['scores'].append(0)
    
    # Summarize
    summary = {}
    for name, data in results.items():
        scores = data['scores']
        summary[name] = {
            'mean_score': sum(scores) / len(scores),
            'errors': data['errors']
        }
        print(f'{name}: mean={summary[name]["mean_score"]:.2f}, errors={data["errors"]}')
    return summary

Regression Testing for Catastrophic Forgetting

Fine-tuning can degrade the model's general capabilities, a phenomenon called catastrophic forgetting. Run a regression test suite on both the base and fine-tuned model covering tasks you care about beyond your target task: general question answering, reasoning, code generation, instruction following. If the fine-tuned model scores significantly lower on these tasks, your LoRA rank may be too high or you trained for too many epochs.

REGRESSION_TEST_CASES = [
    # General QA
    {'input': 'What is the capital of France?', 'expected_substring': 'Paris'},
    {'input': 'What is 17 * 23?', 'expected_substring': '391'},
    # Instruction following
    {'input': 'List 3 planets. Format as: 1. Planet Name', 'expected_pattern': r'^1\. '},
    # Reasoning
    {'input': 'If all A are B and all B are C, are all A also C?', 'expected_substring': 'yes'},
]

def run_regression_tests(model_fn, test_cases: list) -> float:
    passed = 0
    for test in test_cases:
        output = model_fn(test['input'], '')
        if 'expected_substring' in test:
            if test['expected_substring'].lower() in output.lower():
                passed += 1
        elif 'expected_pattern' in test:
            import re
            if re.search(test['expected_pattern'], output):
                passed += 1
    
    rate = passed / len(test_cases)
    print(f'Regression test pass rate: {rate:.1%} ({passed}/{len(test_cases)})')
    return rate

Converting to GGUF for Local Inference

For local deployment without expensive GPU infrastructure, convert your merged model to GGUF format and run inference with llama.cpp. GGUF supports various quantization levels: Q4_K_M (4-bit, good quality/speed balance), Q8_0 (8-bit, near-full quality), and Q2_K (2-bit, very fast but lower quality). A Q4 quantized 7B model requires only ~4GB of RAM and can run on CPU at 1-5 tokens per second.

# Step 1: Convert merged HuggingFace model to GGUF
# git clone https://github.com/ggerganov/llama.cpp
# python llama.cpp/convert_hf_to_gguf.py ./merged-model --outtype f16 --outfile model-f16.gguf

# Step 2: Quantize to 4-bit
# ./llama.cpp/llama-quantize model-f16.gguf model-q4.gguf Q4_K_M

# Step 3: Run inference with llama.cpp Python bindings
# pip install llama-cpp-python
from llama_cpp import Llama

llm = Llama(
    model_path='./model-q4.gguf',
    n_ctx=4096,         # context window
    n_threads=8,        # CPU threads
    n_gpu_layers=0      # set > 0 to offload layers to GPU
)

output = llm.create_chat_completion(
    messages=[{'role': 'user', 'content': 'What is the capital of France?'}],
    temperature=0.1
)
print(output['choices'][0]['message']['content'])

Serving with vLLM for Production

For production serving of a fine-tuned model at scale, vLLM is the current standard. vLLM uses PagedAttention to efficiently batch multiple requests together, dramatically increasing GPU throughput. It supports OpenAI-compatible API endpoints, making it a drop-in replacement for the OpenAI API. A single A100 GPU running vLLM with a fine-tuned 7B model can handle hundreds of requests per minute.

# Start vLLM server (run from command line)
# pip install vllm
# python -m vllm.entrypoints.openai.api_server \
#     --model ./merged-model \
#     --host 0.0.0.0 \
#     --port 8000 \
#     --max-model-len 4096 \
#     --tensor-parallel-size 1

# Use with OpenAI client (drop-in replacement)
from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:8000/v1',
    api_key='not-needed'  # vLLM doesn't require auth by default
)

response = client.chat.completions.create(
    model='merged-model',  # model name matches the path you passed to vLLM
    messages=[{'role': 'user', 'content': 'Extract JSON from: "Alice, 30, NYC"'}]
)
print(response.choices[0].message.content)

A/B Testing the Fine-Tuned Model

Before fully switching to the fine-tuned model in production, run an A/B test: route a percentage of production traffic (start with 5-10%) to the fine-tuned model while the majority still uses the base model or existing prompt approach. Monitor quality scores, latency, and user satisfaction metrics for both groups. Only increase the fine-tuned model's traffic share if the A/B test confirms improvement after a statistically significant number of requests.

import random

class ModelRouter:
    def __init__(self, fine_tuned_traffic_fraction=0.1):
        self.ft_fraction = fine_tuned_traffic_fraction
        self.metrics = {'base': {'count': 0, 'quality_sum': 0}, 'fine_tuned': {'count': 0, 'quality_sum': 0}}

    def route(self, user_id: str, request: str) -> dict:
        # Deterministic routing by user_id (same user always goes to same model)
        use_fine_tuned = (hash(user_id) % 100) < (self.ft_fraction * 100)
        model_group = 'fine_tuned' if use_fine_tuned else 'base'
        
        response = call_model(request, use_fine_tuned=use_fine_tuned)
        return {'response': response, 'model_group': model_group}

    def record_quality(self, model_group: str, quality_score: float):
        self.metrics[model_group]['count'] += 1
        self.metrics[model_group]['quality_sum'] += quality_score

    def ab_test_summary(self) -> dict:
        summary = {}
        for group, data in self.metrics.items():
            avg = data['quality_sum'] / data['count'] if data['count'] > 0 else 0
            summary[group] = {'avg_quality': avg, 'n': data['count']}
        return summary

Maintaining Fine-Tuned Models Over Time

Fine-tuned models require ongoing maintenance. When the base model updates (new GPT-4o version, new Mistral release), your adapter weights may not be compatible and you may need to retrain. When your task requirements change, you need to update the training data and retrain. When you discover new failure modes in production, add examples to your training set. Plan for periodic retraining as part of your production ML operations workflow.

Deployment Decision Matrix

Choosing a deployment strategy for your fine-tuned model depends on your scale and infrastructure constraints. For low volume (less than 1000 requests/day), OpenAI's fine-tuning API is simplest. For medium volume (1000-100,000/day), consider vLLM on a single GPU instance. For high volume or latency-critical applications, use vLLM with multiple GPUs, consider tensor parallelism, and add a caching layer in front. For privacy-sensitive data, self-host on your own infrastructure with GGUF/llama.cpp or vLLM.

Quick Check

Test your understanding of evaluating and deploying fine-tuned models from this lesson.

Lesson Recap

In this lesson you learned: rigorous pre-deployment evaluation comparing fine-tuned vs. base model on held-out test cases is non-negotiable, regression testing detects catastrophic forgetting of general capabilities caused by over-specialization, and deployment options range from OpenAI's managed fine-tuning API for simplicity to vLLM for high-throughput production serving and GGUF/llama.cpp for CPU-based local inference. Congratulations on completing the AI Engineering track!

Frequently asked questions

Is the “Evaluating and Deploying Your Fine-Tuned Model” lesson free?

Yes — the full text of “Evaluating and Deploying Your Fine-Tuned Model” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Evaluating and Deploying Your Fine-Tuned Model”?

Run quantitative evals comparing the base and fine-tuned model on held-out test cases, convert to GGUF for local inference, and serve via llama.cpp or vLLM. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Evaluating and Deploying Your Fine-Tuned Model” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

When Fine-Tuning Beats Prompting
Preparing a High-Quality Training Dataset
LoRA Fine-Tuning with Hugging Face PEFT
Evaluating and Deploying Your Fine-Tuned Model

← Back to AI Engineering Academy