Confidence and Uncertainty in Classification
Asking the model to score confidence and handle ambiguous classifications.
Confidence and Uncertainty in Classification is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Problem of Overconfident Models
By default, LLMs answer classification tasks with apparent certainty even when the input is genuinely ambiguous. A model told to return positive, negative, or neutral will always pick one — never saying I am not sure.
In production systems, acting on uncertain classifications as if they were certain causes costly errors: misrouted support tickets, wrong recommendations, inaccurate reports.
Uncertainty quantification in classification prompts solves this problem.
Confidence Scores 1-10
Asking the model to rate its confidence on a numeric scale gives a granular signal that downstream systems can threshold:
import anthropic, json
client = anthropic.Anthropic(api_key='YOUR_API_KEY')
def classify_with_confidence(text):
prompt = f'''
Classify the sentiment of the text below.
Return JSON:
{{
"sentiment": "positive|negative|neutral",
"confidence": 1-10,
"reason": "brief explanation of confidence level"
}}
Confidence scale: 10=completely certain, 1=total guess, 5=genuinely ambiguous
Text: {text}
'''
r = client.messages.create(
model='claude-opus-4-5', max_tokens=100,
messages=[{'role': 'user', 'content': prompt}]
)
return json.loads(r.content[0].text)
print(classify_with_confidence('I sort of liked it but the wait was too long.'))
print(classify_with_confidence('This product is absolutely outstanding!'))The UNCERTAIN Response
Instructing the model to explicitly respond with UNCERTAIN when classification confidence is below a threshold creates a three-way output: positive, negative, or UNCERTAIN:
def classify_or_uncertain(text, uncertainty_threshold=4):
prompt = f'''
Classify the sentiment of the text: positive, negative, or neutral.
If the sentiment is genuinely ambiguous or you are not confident (confidence below {uncertainty_threshold}/10),
return UNCERTAIN instead of guessing.
Return JSON: {{"sentiment": "positive|negative|neutral|UNCERTAIN", "confidence": 1-10}}
Text: {text}
'''
r = client.messages.create(
model='claude-opus-4-5', max_tokens=80,
messages=[{'role': 'user', 'content': prompt}]
)
result = json.loads(r.content[0].text)
if result['sentiment'] == 'UNCERTAIN' or result['confidence'] < uncertainty_threshold:
print(f'Routing to human review: confidence={result["confidence"]}')
return result
print(classify_or_uncertain('It was fine, I guess. Not bad, not great.'))
print(classify_or_uncertain('Absolutely terrible product. Never buying again.'))Ranked Category Probabilities
Instead of forcing a single category, ask the model to rank all possible categories by likelihood. This reveals how close the top two categories are:
def classify_ranked(text, categories):
cats = ', '.join(categories)
prompt = f'''
Classify this text into one of these categories: {cats}
Return ALL categories ranked by likelihood, highest first.
Return JSON: {{"ranked": [{{"category": str, "probability": 0.0-1.0}}]}}
Probabilities must sum to 1.0.
Text: {text}
'''
r = client.messages.create(
model='claude-opus-4-5', max_tokens=150,
messages=[{'role': 'user', 'content': prompt}]
)
result = json.loads(r.content[0].text)
return result['ranked']
cats = ['billing', 'technical', 'general', 'cancellation']
ranked = classify_ranked('I was charged twice and now my account is locked.', cats)
for item in ranked:
print(f'{item["category"]}: {item["probability"]:.0%}')Using Probability Spread to Detect Ambiguity
The spread between the top two ranked probabilities is a reliable ambiguity signal. A small gap means the model is uncertain; a large gap means confident:
def classify_with_ambiguity_detection(text, categories, ambiguity_threshold=0.15):
ranked = classify_ranked(text, categories)
top1_prob = ranked[0]['probability']
top2_prob = ranked[1]['probability'] if len(ranked) > 1 else 0
spread = top1_prob - top2_prob
is_ambiguous = spread < ambiguity_threshold
return {
'primary': ranked[0]['category'],
'secondary': ranked[1]['category'] if len(ranked) > 1 else None,
'confidence_spread': round(spread, 3),
'is_ambiguous': is_ambiguous,
'action': 'human_review' if is_ambiguous else 'auto_classify'
}
result = classify_with_ambiguity_detection(
'My upgrade did not apply and I think I was still charged.', ['billing', 'technical', 'general', 'cancellation']
)
print(result)Conditional Uncertainty: If Unsure, Ask
For conversational applications, instead of returning UNCERTAIN, the model can ask for clarification:
SYSTEM_CLARIFY = '''
You are a support ticket classifier.
If the customer message is clear, classify it and respond with JSON:
{"action": "classify", "category": str, "confidence": 1-10}
If the message is ambiguous or you are not sure which category applies, respond with:
{"action": "clarify", "question": "A single clarifying question to ask the customer"}
Categories: billing, technical, account, cancellation
Only ask for clarification when genuinely needed. Prefer classification when possible.
'''
def classify_or_ask(message):
r = client.messages.create(
model='claude-opus-4-5', max_tokens=100,
system=SYSTEM_CLARIFY,
messages=[{'role': 'user', 'content': message}]
)
return json.loads(r.content[0].text)
print(classify_or_ask('It is not working anymore.'))
print(classify_or_ask('Cancel my subscription immediately.'))Calibrating Confidence: Temperature and Consistency
Running the same classification multiple times at different temperatures reveals true model uncertainty. High variance = truly ambiguous input:
from collections import Counter
def calibrated_classify(text, n_samples=5):
results = []
for _ in range(n_samples):
r = client.messages.create(
model='claude-opus-4-5', max_tokens=50,
messages=[{'role': 'user', 'content': f'Classify as positive/negative/neutral. Return JSON: {{"sentiment": str}}\n\n{text}'}]
)
results.append(json.loads(r.content[0].text)['sentiment'])
counts = Counter(results)
dominant = counts.most_common(1)[0]
agreement_rate = dominant[1] / n_samples
return {
'classification': dominant[0],
'agreement_rate': agreement_rate,
'is_uncertain': agreement_rate < 0.7,
'all_results': dict(counts)
}
result = calibrated_classify('The product is okay, nothing special.')
print(result)Routing Based on Confidence
A production routing system uses confidence levels to route to different handlers:
def route_by_confidence(text, classify_fn, auto_threshold=8, human_threshold=4):
result = classify_fn(text)
confidence = result.get('confidence', 5)
category = result.get('category') or result.get('sentiment', 'unknown')
if confidence >= auto_threshold:
return {'route': 'auto_process', 'category': category, 'confidence': confidence}
elif confidence >= human_threshold:
return {'route': 'auto_process_with_flag', 'category': category, 'confidence': confidence,
'flag': 'Low confidence — monitor output'}
else:
return {'route': 'human_review', 'category': category, 'confidence': confidence,
'flag': 'Very low confidence — human classification required'}
print(route_by_confidence('Hate this product.', classify_with_confidence))
print(route_by_confidence('It is kind of okay but also not really.', classify_with_confidence))Structured Uncertainty Fields
A comprehensive uncertainty schema for classification outputs:
UNCERTAINTY_SCHEMA = '''
Return JSON:
{
"primary_category": "string",
"confidence": 1-10,
"uncertainty_type": "none | ambiguous_input | insufficient_context | boundary_case | none",
"alternative_categories": ["string"] or [],
"uncertainty_explanation": "string or null",
"recommended_action": "auto_classify | human_review | request_more_info"
}
Uncertainty types:
- ambiguous_input: The text could clearly mean multiple things
- insufficient_context: Need more information to classify correctly
- boundary_case: The text sits on the border between two categories
- none: Clear classification, no uncertainty
'''
print(UNCERTAINTY_SCHEMA)
print('Use this schema for any classification task requiring uncertainty quantification.')Tracking Uncertainty in Production
Monitor uncertainty rates in production to detect prompt degradation or category drift:
class ClassificationMonitor:
def __init__(self, human_review_threshold=0.15):
self.total = 0
self.uncertain = 0
self.threshold = human_review_threshold
self.category_counts = {}
def record(self, result):
self.total += 1
cat = result.get('category', 'unknown')
self.category_counts[cat] = self.category_counts.get(cat, 0) + 1
if result.get('confidence', 10) < 5 or result.get('sentiment') == 'UNCERTAIN':
self.uncertain += 1
def report(self):
uncertain_rate = self.uncertain / self.total if self.total else 0
alert = uncertain_rate > self.threshold
return {
'total': self.total,
'uncertain_rate': round(uncertain_rate, 3),
'alert': alert,
'category_distribution': self.category_counts
}
monitor = ClassificationMonitor()
print('Production monitoring system defined.')When to Trust High Confidence
High model confidence does not always mean correct classification. Common failure modes even at high confidence:
- Systematic bias: Model consistently mislabels a specific pattern as high-confidence wrong answer
- Domain shift: Model is confident but the input style is very different from what it was trained on
- Sycophancy: Model calibrates confidence to what sounds good, not to actual certainty
Always evaluate confidence calibration against a labeled test set — not just accuracy, but whether high-confidence predictions are actually more accurate than low-confidence ones.
Quick Check
What does a small spread between the top two ranked category probabilities indicate in a classification result?
Uncertainty in Classification — Key Takeaways
Uncertainty quantification transforms classification from a black box into a manageable system:
- Ask for confidence scores (1-10) with every classification — never treat outputs as equally reliable
- Use the UNCERTAIN response for genuinely ambiguous inputs rather than forcing a category pick
- Rank all categories by probability — the spread between top two is the best ambiguity signal
- Route to human review when confidence is below threshold; auto-process when above
- For conversational apps, ask clarifying questions rather than returning UNCERTAIN
- Monitor uncertainty rates in production — rising rates signal prompt degradation or category drift
- Always evaluate confidence calibration against labeled data — not just accuracy
Frequently asked questions
Is the “Confidence and Uncertainty in Classification” lesson free?
Yes — the full text of “Confidence and Uncertainty in Classification” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.
What will I learn in “Confidence and Uncertainty in Classification”?
Asking the model to score confidence and handle ambiguous classifications. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Prompt Engineering?
No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Confidence and Uncertainty in Classification” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Prompt Engineering lesson?
Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Named Entity Extraction Prompts
- Schema-Driven Data Extraction
- LLM as Text Classifier
- Confidence and Uncertainty in Classification