Prompt Engineering & LLM Optimization for Developers · Lesson

Self-Consistency & Generated Knowledge

Implement techniques where LLMs generate multiple reasoning paths and choose the most consistent answer, or generate knowledge to aid reasoning.

Self-Consistency & Generated Knowledge is a free Prompt Engineering & LLM Optimization for Developers lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Prompt Engineering & LLM Optimization for Developers learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Intro: Consistency & Knowledge

Welcome to Lesson 2! In complex problem-solving, Large Language Models (LLMs) can sometimes struggle, leading to incorrect or inconsistent answers.

This lesson introduces two powerful techniques to boost their reliability: Self-Consistency and Generated Knowledge. These methods help LLMs 'think' more deeply and systematically.

Why Advanced Reasoning?

LLMs are great at generating text, but they can sometimes make logical errors or 'hallucinate' (produce factually incorrect information), especially with multi-step reasoning.

Advanced prompting strategies like Self-Consistency and Generated Knowledge aim to:

Improve accuracy for complex tasks.
Reduce the likelihood of factual errors.
Make LLM responses more robust and reliable.

Understanding Self-Consistency

Self-Consistency is a technique where you prompt an LLM to generate multiple distinct reasoning paths or answers for the same question.

Instead of relying on a single output, you then aggregate these different outputs and select the most consistent (e.g., the most frequent) answer. It's like asking several experts and taking the majority opinion.

Self-Consistency in Action

Imagine asking an LLM: "If a train leaves station A at 8 AM traveling at 60 mph, and another leaves station B (300 miles away) at 9 AM traveling at 70 mph, when do they meet?"

A single prompt might give a wrong answer. With self-consistency, you'd ask this multiple times, perhaps with slightly varied phrasing, then compare the results to find the most common meeting time.

Code: Simple Self-Consistency

This Python example simulates calling an LLM multiple times for a math problem. It then picks the most frequent answer, enhancing reliability.

import collections
import random

def call_llm(prompt):
    # Simulate LLM responses for a math problem
    # In a real app, this would be an actual LLM API call
    if "What is (15 * 3) - 7?" in prompt:
        return random.choice(["38", "The answer is 38.", "40 (Oops!)"])
    return "Simulated response."

def main():
    print("--- Self-Consistency Example ---")
    question_prompt = "What is (15 * 3) - 7? Give your final numeric answer only."
    answers = []
    num_attempts = 5

    for i in range(num_attempts):
        raw_output = call_llm(question_prompt)
        # Simple extraction of numeric part
        numeric_answer = ''.join(filter(str.isdigit, raw_output))
        if numeric_answer:
            answers.append(int(numeric_answer))
            print(f"Attempt {i+1}: {numeric_answer}")
        else:
            print(f"Attempt {i+1}: Could not parse '{raw_output}'")

    # Find the most common answer (voting)
    if answers:
        most_common = collections.Counter(answers).most_common(1)
        print(f"\nMost consistent answer: {most_common[0][0]}")
    else:
        print("\nNo valid answers generated.")

if __name__ == "__main__":
    main()

Introducing Generated Knowledge

Generated Knowledge is a technique where the LLM first generates relevant facts, context, or intermediate thoughts, and then uses this self-generated information to answer the main query.

This is akin to doing research before writing an essay. The LLM essentially 'pre-computes' or 'recalls' relevant knowledge to build a stronger foundation for its final answer.

Generated Knowledge in Practice

Consider a question like: "Describe the main differences between a black hole and a wormhole."

Instead of directly answering, you could first prompt the LLM to:

"List key characteristics of a black hole."
"List key characteristics of a wormhole."

Then, use these generated lists as context for the original question, leading to a more informed and accurate comparison.

Code: Pre-computation with LLM

This Python example demonstrates a two-step process: first, asking the LLM to generate knowledge, and then using that knowledge in a subsequent prompt to answer a complex question.

def call_llm(prompt):
    # Simulate LLM responses for knowledge generation
    if "What are the key components of prompt injection?" in prompt:
        return "Malicious user input, LLM vulnerability to new instructions, attempt to bypass security or extract data."
    elif "Using the information about Malicious user input, LLM vulnerability to new instructions, attempt to bypass security or extract data., explain the concept of 'prompt injection' in cybersecurity." in prompt:
        return "Prompt injection is an attack where crafted malicious user input manipulates an LLM to override its original instructions, potentially leading to unauthorized actions, data exposure, or harmful content generation. It exploits the LLM's tendency to follow new directions even if they contradict its safety guidelines."
    return "Simulated response."

def main():
    print("--- Generated Knowledge Example ---")

    # Step 1: Generate knowledge
    knowledge_prompt = "What are the key components of prompt injection?"
    print(f"LLM generating knowledge...")
    generated_knowledge = call_llm(knowledge_prompt)
    print(f"Generated Knowledge:\n{generated_knowledge}\n")

    # Step 2: Use generated knowledge to answer the main question
    main_question = "explain the concept of 'prompt injection' in cybersecurity."
    final_prompt = f"Using the information about {generated_knowledge}, {main_question}"
    print(f"LLM answering main question using generated knowledge...")
    final_answer = call_llm(final_prompt)
    print(f"Final Answer:\n{final_answer}")

if __name__ == "__main__":
    main()

Synergies: Combining Techniques

While powerful individually, Self-Consistency and Generated Knowledge can also be combined for even greater robustness.

For instance, you could first use Generated Knowledge to create a robust set of facts, and then apply Self-Consistency to the final answer generation phase, ensuring both factual grounding and reliable output.

Check Your Understanding

Test your knowledge about Self-Consistency and Generated Knowledge.

Summary: Powering LLMs

In this lesson, you learned about two advanced prompting strategies:

Self-Consistency: Generating multiple answers and selecting the most common one to improve reliability.
Generated Knowledge: Having the LLM create relevant context first, then using that context to answer the main question.

These techniques empower LLMs to tackle complex problems with greater accuracy and less risk of errors. Continue experimenting with them to unlock more robust LLM applications!

Frequently asked questions

Is the “Self-Consistency & Generated Knowledge” lesson free?

Yes — the full text of “Self-Consistency & Generated Knowledge” is free to read here on the web, and the Prompt Engineering & LLM Optimization for Developers course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Prompt Engineering & LLM Optimization for Developers course, upgrade to CoddyKit PRO.

What will I learn in “Self-Consistency & Generated Knowledge”?

Implement techniques where LLMs generate multiple reasoning paths and choose the most consistent answer, or generate knowledge to aid reasoning. You practise Prompt Engineering & LLM Optimization for Developers with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Prompt Engineering & LLM Optimization for Developers?

No prior experience is required. Prompt Engineering & LLM Optimization for Developers on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Self-Consistency & Generated Knowledge” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Prompt Engineering & LLM Optimization for Developers lesson?

Yes. Every Prompt Engineering & LLM Optimization for Developers lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Prompt Engineering & LLM Optimization for Developers