AI Prompt Engineering · Lesson

How AI Generates Responses

Token prediction, probability, and why AI doesn't 'think' like humans.

How AI Generates Responses is a free AI Prompt Engineering lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Text as a Probability Problem

At its core, a language model does one thing: predict the next token given all the tokens before it.

A token is a small unit of text — roughly a word or word fragment. The model assigns a probability to every token in its vocabulary and picks one. Then it repeats the process, token by token, until it generates a complete response.

What Is a Token?

Tokens are the atoms of LLM text processing. English text tokenizes roughly as:

Common words → 1 token each (the, run)
Less common words → split into 2-3 tokens (running → run + ning)
Punctuation and spaces → often their own tokens

Models like GPT-4o and Claude use tokenizers that handle 100k+ vocabulary entries including subwords across many languages.

import tiktoken

encoding = tiktoken.encoding_for_model('gpt-4o')

sentence = 'The temperature parameter controls randomness in token selection.'
tokens = encoding.encode(sentence)
token_strings = [encoding.decode([t]) for t in tokens]

print(f'Sentence: {sentence}')
print(f'Token count: {len(tokens)}')
print(f'Tokens: {token_strings}')

Autoregressive Generation

Generation is autoregressive: each new token is appended to the input before predicting the next one.

So when generating 'The sky is blue', the model:

Sees 'The' → predicts 'sky'
Sees 'The sky' → predicts 'is'
Sees 'The sky is' → predicts 'blue'
Sees 'The sky is blue' → predicts end-of-sequence

This is why generation slows down for very long outputs — each token requires a full forward pass.

# Simulated autoregressive token-by-token output via streaming
import openai

client = openai.OpenAI(api_key='sk-your-key-here')

stream = client.chat.completions.create(
    model='gpt-4o',
    stream=True,     # receive tokens as they are generated
    messages=[{'role': 'user', 'content': 'Name five planets in our solar system.'}]
)

print('Tokens arriving one by one:')
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end='', flush=True)
print()  # newline at end

Temperature: Controlling Randomness

Temperature is a number between 0 and 2 that scales the probability distribution before sampling:

Temperature 0: always pick the most probable token — deterministic, repetitive
Temperature 1: sample according to the raw probabilities — balanced
Temperature 2: flatten probabilities — very random, sometimes incoherent

Use low temperature for factual tasks; higher temperature for creative work.

import openai

client = openai.OpenAI(api_key='sk-your-key-here')

prompt = 'Continue this sentence with one word: The ocean is'

for temp in [0.0, 0.7, 1.5]:
    response = client.chat.completions.create(
        model='gpt-4o',
        temperature=temp,
        max_tokens=5,
        messages=[{'role': 'user', 'content': prompt}]
    )
    word = response.choices[0].message.content.strip()
    print(f'Temperature {temp}: "{word}"')

Why Responses Vary Between Runs

Even with the same prompt, two runs of the same model can produce different outputs. This happens because:

Sampling is probabilistic — the model draws from a distribution, not a lookup table
Small numerical differences in floating-point math can cascade
Hardware parallelism introduces non-determinism

Set temperature=0 and seed (if supported) to maximize reproducibility.

import openai

client = openai.OpenAI(api_key='sk-your-key-here')

prompt = 'Give me a one-word color that feels calm.'

for run in range(3):
    response = client.chat.completions.create(
        model='gpt-4o',
        temperature=1.0,   # randomness ON
        max_tokens=5,
        messages=[{'role': 'user', 'content': prompt}]
    )
    print(f'Run {run + 1}: {response.choices[0].message.content.strip()}')

# For reproducible outputs use seed + temperature=0
response = client.chat.completions.create(
    model='gpt-4o',
    temperature=0,
    seed=42,
    max_tokens=5,
    messages=[{'role': 'user', 'content': prompt}]
)
print(f'Deterministic: {response.choices[0].message.content.strip()}')

Top-p Sampling

Top-p (nucleus sampling) is another way to control randomness. Instead of scaling all probabilities, it restricts sampling to the smallest set of tokens whose cumulative probability exceeds p.

top_p=0.1: only the very top tokens (conservative)
top_p=0.9: broader pool (creative)
top_p=1.0: all tokens (full distribution)

In practice, most teams tune temperature and leave top-p at 1.0.

import openai

client = openai.OpenAI(api_key='sk-your-key-here')

# Conservative generation: top_p=0.1 restricts to high-confidence tokens
response_conservative = client.chat.completions.create(
    model='gpt-4o',
    top_p=0.1,
    max_tokens=30,
    messages=[{'role': 'user', 'content': 'What is the capital of Japan?'}]
)

# Creative generation: top_p=0.95 allows broader token pool
response_creative = client.chat.completions.create(
    model='gpt-4o',
    top_p=0.95,
    max_tokens=30,
    messages=[{'role': 'user', 'content': 'Write a poetic one-liner about the moon.'}]
)

print('Conservative:', response_conservative.choices[0].message.content)
print('Creative:    ', response_creative.choices[0].message.content)

No True Understanding — Pattern Matching

LLMs do not understand language the way humans do. They are extremely sophisticated pattern matchers trained on massive text corpora.

When the model answers 'What is photosynthesis?', it is not retrieving a stored fact — it is generating the sequence of tokens that statistically follows the question pattern, based on having seen millions of similar documents during training.

# Demonstration: the model can produce plausible-sounding but wrong answers
import anthropic

client = anthropic.Anthropic(api_key='sk-ant-your-key-here')

# Asking about something nonsensical — model may still try to answer
response = client.messages.create(
    model='claude-opus-4-5',
    max_tokens=128,
    messages=[{
        'role': 'user',
        'content': (
            'What is the boiling point of happiness in degrees Celsius? '
            'Please just say "I cannot answer this" if the question makes no sense.'
        )
    }]
)
print(response.content[0].text)

Training vs Inference

The model's 'knowledge' was frozen during training on a large text corpus. At inference (when you send a prompt), the model generates text based on that frozen knowledge — it is not learning or searching the internet.

This means it cannot know about events after its training cutoff, cannot access URLs, and cannot verify whether a fact has changed since training.

import anthropic

client = anthropic.Anthropic(api_key='sk-ant-your-key-here')

# Asking the model about its own knowledge cutoff
response = client.messages.create(
    model='claude-opus-4-5',
    max_tokens=128,
    messages=[{
        'role': 'user',
        'content': (
            'What is your knowledge cutoff date? '
            'And can you access the internet right now to look up today\'s news?'
        )
    }]
)
print(response.content[0].text)

What Happens Inside a Forward Pass

At a high level, each token prediction involves:

Converting all input tokens to numerical embeddings
Passing them through many transformer layers (attention + feed-forward)
Producing a probability distribution over the entire vocabulary
Sampling one token from that distribution

Modern models have billions of parameters that shape this transformation — all learned during training from human text.

# You can inspect logprobs (token probabilities) to see the model's confidence
import openai

client = openai.OpenAI(api_key='sk-your-key-here')

response = client.chat.completions.create(
    model='gpt-4o',
    max_tokens=5,
    logprobs=True,
    top_logprobs=3,
    messages=[{'role': 'user', 'content': 'The opposite of hot is'}]
)

for token_info in response.choices[0].logprobs.content:
    print(f'Chosen token: "{token_info.token}"')
    for alt in token_info.top_logprobs:
        import math
        prob = round(math.exp(alt.logprob) * 100, 1)
        print(f'  Option "{alt.token}": {prob}% probability')

Stop Sequences

Stop sequences tell the model to halt generation when it produces a specific string. This is useful for:

Preventing the model from generating more than one answer in a list
Stopping at a delimiter like ### or ---END---
Enforcing single-line outputs

Without stop sequences, the model generates until it hits max_tokens or predicts an end-of-sequence token.

import openai

client = openai.OpenAI(api_key='sk-your-key-here')

# Stop after the first item in a numbered list
response = client.chat.completions.create(
    model='gpt-4o',
    max_tokens=64,
    stop=['2.'],   # halt as soon as '2.' appears
    messages=[{
        'role': 'user',
        'content': 'List 5 programming languages, numbered 1 through 5.'
    }]
)
print(response.choices[0].message.content)
print('Stop reason:', response.choices[0].finish_reason)

Practical Implications for Prompt Writers

Understanding generation mechanics helps you write better prompts:

Use temperature 0 for tasks requiring consistent, factual output
Use temperature 0.7-1.0 for creative writing
Verify facts — the model generates plausible text, not guaranteed truth
Use stop sequences to control output length precisely
Short prompts → more creative but less controlled output
Detailed prompts → more constrained, focused output

import anthropic

client = anthropic.Anthropic(api_key='sk-ant-your-key-here')

# Low temperature for factual classification
fact_response = client.messages.create(
    model='claude-opus-4-5',
    max_tokens=8,
    temperature=0,   # deterministic
    messages=[{'role': 'user', 'content': 'Is Paris the capital of France? Answer YES or NO only.'}]
)
print('Factual (temp=0):', fact_response.content[0].text.strip())

# Higher temperature for creative tasks
creative_response = client.messages.create(
    model='claude-opus-4-5',
    max_tokens=64,
    temperature=1.0,
    messages=[{'role': 'user', 'content': 'Write a surprising one-line poem about code.'}]
)
print('Creative (temp=1):', creative_response.content[0].text.strip())

Knowledge Check

You have learned how LLMs generate text token by token. Let's check your understanding of temperature.

A developer is building a customer support chatbot that must give consistent, accurate answers to billing questions. Which temperature setting is most appropriate?

How AI Generates Responses — Recap

You now understand the mechanics behind every AI response:

Models predict the next token one at a time — autoregressive generation
Temperature controls how much randomness is injected into token selection
Top-p restricts sampling to a nucleus of high-probability tokens
The model does not understand — it pattern-matches at massive scale
Knowledge is frozen at training — no real-time data, no internet access
Stop sequences let you control exactly where output ends

Frequently asked questions

Is the “How AI Generates Responses” lesson free?

Yes — the full text of “How AI Generates Responses” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “How AI Generates Responses”?

Token prediction, probability, and why AI doesn't 'think' like humans. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “How AI Generates Responses” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering