AI Prompt Engineering · Lesson

Reading and Evaluating AI Outputs

Criteria for judging relevance, accuracy, and completeness of AI responses.

Reading and Evaluating AI Outputs is a free AI Prompt Engineering lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Evaluation Matters

Getting an AI response is only half the job. The second half is judging whether that response is actually good.

Without a clear evaluation framework, you might accept a response that looks polished but answers the wrong question, or reject a response that is genuinely useful because it isn't formatted the way you expected.

Developing a reliable eye for AI output quality is one of the most practical skills in prompt engineering.

The Four Evaluation Criteria

When you read an AI response, judge it on four dimensions:

Relevance — Did it answer the actual question you asked?
Accuracy — Is the information factually correct?
Completeness — Did it cover all parts of the request?
Format — Is it structured the way you need it to be?

A response can score high on three and fail on the fourth. Each criterion matters independently.

Criterion 1: Relevance

Relevance asks: did the model answer the question you actually asked, or did it answer a related but different question?

Example: You ask "What are the risks of using LLMs in healthcare?" and the model responds with a general overview of how LLMs work. That response may be accurate, but it is not relevant — it missed the point.

To check relevance, re-read your original prompt and ask: does the response directly address what I asked?

Criterion 2: Accuracy

Accuracy asks: are the facts, figures, and claims in the response correct?

AI models can hallucinate — they may state things confidently that are simply wrong. Common accuracy problems include:

Incorrect statistics or dates
Misattributed quotes
Outdated information presented as current
Made-up references or citations

For factual topics, always verify key claims with a trusted source before using the output.

Criterion 3: Completeness

Completeness asks: did the response cover all the parts of your request?

If your prompt had multiple parts — "Explain X, list three examples, and suggest a next step" — check that the model addressed all three, not just the first.

Models sometimes drop the last part of a multi-part request, especially if the prompt is long. Reading the response against your prompt point by point is the most reliable way to check completeness.

Criterion 4: Format

Format asks: is the response structured in the way you need?

Even a perfectly accurate and complete response can be hard to use if the format is wrong. Common format mismatches:

Prose when you needed bullet points
A long essay when you needed a summary
Missing headers that would help navigation
Code without explanations, or explanations without code

Format problems are often the easiest to fix with a follow-up prompt.

A Simple Evaluation Checklist

After receiving any AI response, run through this mental checklist:

Relevance: Does it answer my actual question?
Accuracy: Can I trust the facts? What would I need to verify?
Completeness: Did it cover all parts of my request?
Format: Is it structured in a way I can use?

If any criterion fails, that tells you exactly what kind of follow-up prompt to write. Each criterion points to a specific type of correction.

Evaluating in Code: Scoring a Response

You can build a lightweight evaluation wrapper in Python that scores a response on each criterion using a second LLM call:

import openai

client = openai.OpenAI(api_key='sk-...')

def evaluate_response(original_prompt, response_text):
    eval_prompt = f'''You are an evaluator. Score the following AI response on a scale of 1-5 for each criterion.

Original prompt: {original_prompt}

AI response: {response_text}

Score each:
- Relevance (1-5): Did it answer the actual question?
- Accuracy (1-5): Are the facts correct? (flag any concerns)
- Completeness (1-5): Did it cover all parts of the request?
- Format (1-5): Is it structured appropriately?

Return your scores as: Relevance: X, Accuracy: X, Completeness: X, Format: X
Then a one-sentence note for any score below 4.'''

    result = client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[{'role': 'user', 'content': eval_prompt}]
    )
    return result.choices[0].message.content

Relevance Failures: Common Patterns

Relevance failures tend to follow predictable patterns. Recognizing them speeds up your evaluation:

Topic drift — Model starts correctly then wanders to a related topic
Question swap — Model answers a simpler or easier version of your question
Over-generalization — You asked about a specific case, model answers the general case
Constraint ignore — You specified a context (e.g., "for beginners"), model ignored it

Each pattern suggests a specific fix in your follow-up prompt.

Accuracy Red Flags to Watch For

Even when you cannot immediately verify a claim, certain signals suggest lower accuracy:

Very specific numbers cited without a source
Claims that seem too convenient or perfectly on-point
References to studies, papers, or books by title and author
Dates and version numbers (these change frequently)
Legal, medical, or financial specifics

These are not proof of error, but they are the items worth double-checking before you use the output in a high-stakes context.

Using Evaluation to Write Better Follow-Ups

The real value of evaluation is that each failing criterion maps directly to a follow-up prompt type:

Relevance fail → "You answered X but I was asking about Y. Please focus on Y."
Accuracy concern → "Please double-check the claim about Z and cite your basis."
Completeness fail → "You didn't address the third part of my question. Please add that."
Format fail → "Rewrite this as bullet points with a one-line summary at the top."

Evaluation and follow-up prompting work as a feedback loop.

Knowledge Check: Evaluation Criteria

You ask the model: "List the top 5 Python web frameworks with a one-sentence description of each." The model responds with a well-written essay about Python's history and its rise in web development, mentioning Flask and Django briefly but not listing 5 frameworks.

Which criterion does this response fail most critically?

Recap: Reading and Evaluating AI Outputs

Great prompting is a two-step process: crafting the prompt and evaluating the response.

The four criteria — Relevance, Accuracy, Completeness, Format — give you a systematic way to assess any AI output. Each failing criterion tells you exactly what kind of follow-up to write.

In the next lesson, you will practice writing those follow-up prompts to correct specific types of failures.

Frequently asked questions

Is the “Reading and Evaluating AI Outputs” lesson free?

Yes — the full text of “Reading and Evaluating AI Outputs” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Reading and Evaluating AI Outputs”?

Criteria for judging relevance, accuracy, and completeness of AI responses. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Reading and Evaluating AI Outputs” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering