AI Engineering Academy · Lesson

Preparing a High-Quality Training Dataset

Collect, clean, and format instruction-following data in the Alpaca and ShareGPT formats, apply data deduplication, and split into train and validation sets.

Preparing a High-Quality Training Dataset is a free AI Engineering Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Data Is the Most Important Fine-Tuning Factor

In fine-tuning, the quality of your training data matters more than any hyperparameter, architecture choice, or training technique. 100 high-quality examples outperform 10,000 mediocre ones. Garbage in, garbage out — the fine-tuned model will faithfully reproduce whatever patterns exist in your data, including mistakes, biases, and format inconsistencies. Investing in data quality is the highest-leverage action in any fine-tuning project.

Instruction-Following Formats

Most fine-tuning for instruction-following tasks uses a conversational message format with system, user, and assistant roles. OpenAI's fine-tuning API uses JSONL files where each line is a complete conversation example. The Alpaca format (instruction/input/output) and ShareGPT format (conversations list) are also widely used. Choose the format that matches the fine-tuning framework you plan to use.

import json

# OpenAI fine-tuning format (JSONL)
# Each line is one training example
openai_example = {
    'messages': [
        {'role': 'system', 'content': 'You are a JSON extraction agent.'},
        {'role': 'user', 'content': 'Extract: "John Smith, age 32, from Seattle, joined 2023-01-15"'},
        {'role': 'assistant', 'content': '{"name": "John Smith", "age": 32, "city": "Seattle", "join_date": "2023-01-15"}'}
    ]
}

# Alpaca format
alpaca_example = {
    'instruction': 'Extract structured data from the following text.',
    'input': 'John Smith, age 32, from Seattle, joined 2023-01-15',
    'output': '{"name": "John Smith", "age": 32, "city": "Seattle", "join_date": "2023-01-15"}'
}

# Write as JSONL
with open('train.jsonl', 'w') as f:
    f.write(json.dumps(openai_example) + '\n')
    # Add more examples here...

Collecting Training Data: Three Strategies

There are three main strategies for building a fine-tuning dataset. Human generation: experts manually write ideal examples — highest quality but slow and expensive. LLM generation: a powerful model (GPT-4o) generates examples that are then validated by humans — much faster and cheaper, good quality if validated. Mining from logs: extract input-output pairs from existing production logs, filtering for high-quality examples using quality signals like user ratings or LLM-as-judge scoring.

from openai import OpenAI

client = OpenAI()

def generate_training_example_with_gpt4o(task_description: str, example_input: str) -> dict:
    '''Use GPT-4o to generate a training example for a smaller model.'''
    prompt = f'''You are creating a training example for fine-tuning a smaller model.
Task: {task_description}

Given this input:
{example_input}

Write the ideal assistant response that demonstrates the correct behavior for this task.
Be specific, accurate, and follow the expected format precisely.'''
    
    response = client.chat.completions.create(
        model='gpt-4o',  # teacher model
        messages=[{'role': 'user', 'content': prompt}]
    )
    
    return {
        'messages': [
            {'role': 'system', 'content': task_description},
            {'role': 'user', 'content': example_input},
            {'role': 'assistant', 'content': response.choices[0].message.content}
        ]
    }

Quality Filtering and Validation

Every training example should pass a quality validation step before inclusion. Validate: the response is in the correct format, the response is accurate (for factual tasks), the response does not contain hallucinations or harmful content, the instruction-response pair is coherent, and the example is representative of the production use case. Use automated validation for format checks and LLM-as-judge for content quality, with human review for a sample.

import json

def validate_training_example(example: dict, schema_validator=None) -> dict:
    issues = []
    
    # Format check
    if 'messages' not in example:
        issues.append('Missing messages field')
        return {'valid': False, 'issues': issues}
    
    messages = example['messages']
    if not any(m['role'] == 'assistant' for m in messages):
        issues.append('No assistant message found')
    
    # Check assistant message quality
    assistant_content = next((m['content'] for m in messages if m['role'] == 'assistant'), '')
    
    if len(assistant_content) < 5:
        issues.append('Assistant response too short')
    
    # Schema validation for JSON output tasks
    if schema_validator:
        try:
            parsed = json.loads(assistant_content)
            schema_validator(parsed)  # raises if invalid
        except json.JSONDecodeError:
            issues.append('Assistant response is not valid JSON')
        except Exception as e:
            issues.append(f'Schema validation failed: {str(e)}')
    
    return {'valid': len(issues) == 0, 'issues': issues}

# Run validation on all examples before training
examples = load_training_examples('raw_dataset.jsonl')
valid_examples = [e for e in examples if validate_training_example(e)['valid']]
print(f'Valid examples: {len(valid_examples)}/{len(examples)}')

Data Deduplication

Duplicate or near-duplicate examples in training data are harmful. They cause the model to over-fit to those specific examples, wasting training capacity that could have learned diverse patterns. Run deduplication before finalizing your dataset. Exact deduplication uses hashing to find identical examples. Near-deduplication uses MinHash or embedding similarity to find examples that differ only in trivial ways.

import hashlib
from datasketch import MinHash, MinHashLSH

def exact_deduplicate(examples: list[dict]) -> list[dict]:
    seen_hashes = set()
    unique = []
    
    for ex in examples:
        # Hash the user and assistant messages
        content = str(ex['messages'])
        h = hashlib.md5(content.encode()).hexdigest()
        if h not in seen_hashes:
            seen_hashes.add(h)
            unique.append(ex)
    
    print(f'Exact dedup: {len(examples)} -> {len(unique)} ({len(examples)-len(unique)} removed)')
    return unique

def near_deduplicate_by_input(examples: list[dict], similarity_threshold=0.85) -> list[dict]:
    # Build index of input texts
    inputs = [next((m['content'] for m in ex['messages'] if m['role'] == 'user'), '') for ex in examples]
    
    lsh = MinHashLSH(threshold=similarity_threshold, num_perm=128)
    unique_indices = set()
    
    for i, text in enumerate(inputs):
        m = MinHash(num_perm=128)
        for word in text.lower().split():
            m.update(word.encode('utf-8'))
        if not lsh.query(m):  # no similar items found
            lsh.insert(str(i), m)
            unique_indices.add(i)
    
    return [examples[i] for i in sorted(unique_indices)]

Train/Validation Split

Split your dataset into training and validation sets before starting any fine-tuning run. The validation set is used to monitor for overfitting during training (if validation loss increases while training loss decreases, the model is overfitting). A typical split is 90% training / 10% validation. Ensure the split is random and stratified if your dataset has meaningful categories (e.g., equal representation of all intent types in both sets).

import random
import json

def split_dataset(examples: list[dict], val_fraction=0.1, seed=42) -> tuple[list, list]:
    random.seed(seed)  # reproducible split
    shuffled = examples.copy()
    random.shuffle(shuffled)
    
    n_val = max(1, int(len(shuffled) * val_fraction))
    val_set = shuffled[:n_val]
    train_set = shuffled[n_val:]
    
    print(f'Train: {len(train_set)} examples, Validation: {len(val_set)} examples')
    return train_set, val_set

def save_jsonl(examples: list[dict], path: str):
    with open(path, 'w') as f:
        for ex in examples:
            f.write(json.dumps(ex) + '\n')

# Split and save
examples = load_training_examples('clean_dataset.jsonl')
train, val = split_dataset(examples, val_fraction=0.1)
save_jsonl(train, 'train.jsonl')
save_jsonl(val, 'validation.jsonl')
print(f'Saved train.jsonl ({len(train)}) and validation.jsonl ({len(val)})')

Balancing the Dataset

Imbalanced datasets cause fine-tuned models to over-specialize in common cases and fail on rare but important ones. If your dataset has 900 examples of category A and 100 examples of category B, the model may learn to always predict A. Balance the dataset by: oversampling minority categories (duplicate rare examples), undersampling majority categories, or generating synthetic examples for underrepresented cases using GPT-4o.

from collections import Counter
import random

def analyze_distribution(examples: list[dict], category_extractor) -> dict:
    categories = [category_extractor(ex) for ex in examples]
    counts = Counter(categories)
    print('Category distribution:')
    for cat, count in counts.most_common():
        print(f'  {cat}: {count} ({100*count/len(examples):.1f}%)')
    return counts

def oversample_minority(examples: list[dict], category_extractor, target_count: int) -> list[dict]:
    by_category = {}
    for ex in examples:
        cat = category_extractor(ex)
        by_category.setdefault(cat, []).append(ex)
    
    balanced = []
    for cat, cat_examples in by_category.items():
        if len(cat_examples) < target_count:
            # Oversample with replacement
            oversampled = random.choices(cat_examples, k=target_count)
            balanced.extend(oversampled)
        else:
            # Undersample to target_count
            balanced.extend(random.sample(cat_examples, target_count))
    
    random.shuffle(balanced)
    return balanced

Data Cleaning and Normalization

Training data often contains inconsistencies that harm fine-tuning: mixed capitalization in output fields, trailing whitespace, inconsistent use of quotes, mixed number formats, or varied JSON key naming conventions. Normalize these before training. The fine-tuned model will learn the exact formatting present in your data — if your data has inconsistencies, the model will reproduce them.

import json
import re

def normalize_json_output_example(example: dict) -> dict:
    '''Normalize JSON output in assistant messages for consistency.'''
    messages = example.get('messages', [])
    normalized = []
    
    for msg in messages:
        if msg['role'] == 'assistant':
            content = msg['content'].strip()
            
            # Try to parse and re-serialize JSON for consistent formatting
            try:
                parsed = json.loads(content)
                # Normalize: sort keys, consistent spacing
                content = json.dumps(parsed, ensure_ascii=False, sort_keys=True)
            except json.JSONDecodeError:
                pass  # Not JSON output - leave as is
            
            normalized.append({'role': 'assistant', 'content': content})
        else:
            normalized.append(msg)
    
    return {'messages': normalized}

def normalize_dataset(examples: list[dict]) -> list[dict]:
    return [normalize_json_output_example(ex) for ex in examples]

Measuring Dataset Quality Metrics

Before submitting data for fine-tuning, compute quality metrics on the dataset as a whole. Check: average and maximum token count per example (long examples cost more to train and may truncate), vocabulary coverage (does the dataset cover the full diversity of production inputs?), and consistency score (do similar inputs get similar outputs?). Most fine-tuning providers have a data validation endpoint that checks for format errors before billing you for a failed training run.

import tiktoken

def analyze_dataset_quality(examples: list[dict], model='gpt-4o-mini') -> dict:
    encoder = tiktoken.encoding_for_model(model)
    
    token_counts = []
    for ex in examples:
        total_tokens = sum(
            len(encoder.encode(m['content']))
            for m in ex['messages']
        )
        token_counts.append(total_tokens)
    
    report = {
        'total_examples': len(examples),
        'avg_tokens_per_example': sum(token_counts) / len(token_counts),
        'max_tokens': max(token_counts),
        'min_tokens': min(token_counts),
        'examples_over_4k_tokens': sum(1 for t in token_counts if t > 4096),
        'estimated_training_tokens': sum(token_counts),
        'estimated_cost': sum(token_counts) / 1_000_000 * 8.0  # ~$8/1M tokens for gpt-4o-mini
    }
    
    for key, value in report.items():
        print(f'{key}: {value}')
    return report

Iterative Dataset Improvement

Dataset preparation is iterative. Fine-tune a small model on your initial dataset, evaluate it on held-out examples, identify the failure modes, and trace them back to dataset gaps or quality issues. Then fix the data and retrain. This error-driven data improvement loop is the standard practice in production fine-tuning and is far more effective than trying to collect a perfect dataset in one pass.

System Prompt Consistency Across Examples

If your fine-tuned model will always use the same system prompt in production, include that exact system prompt in every training example. If you want the model to work without any system prompt, train without one. Mismatches between training and inference conditions are a leading cause of fine-tuning disappointing expectations. The model learns behaviors conditioned on the exact prompt structure it sees during training.

Quick Check

Test your understanding of preparing a fine-tuning training dataset from this lesson.

Lesson Recap

In this lesson you learned: data quality beats data quantity in fine-tuning — 100 excellent examples outperform 10,000 mediocre ones, deduplication, validation, balancing, and normalization are the four key data cleaning steps before any training run, and the train/validation split is essential to detect overfitting during training before it degrades real-world performance. Next up we run LoRA fine-tuning with Hugging Face PEFT.

Frequently asked questions

Is the “Preparing a High-Quality Training Dataset” lesson free?

Yes — the full text of “Preparing a High-Quality Training Dataset” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Preparing a High-Quality Training Dataset”?

Collect, clean, and format instruction-following data in the Alpaca and ShareGPT formats, apply data deduplication, and split into train and validation sets. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Preparing a High-Quality Training Dataset” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

When Fine-Tuning Beats Prompting
Preparing a High-Quality Training Dataset
LoRA Fine-Tuning with Hugging Face PEFT
Evaluating and Deploying Your Fine-Tuned Model

← Back to AI Engineering Academy