Confidence and Uncertainty in Classification
Asking the model to score confidence and handle ambiguous classifications.
The Problem of Overconfident Models
By default, LLMs answer classification tasks with apparent certainty even when the input is genuinely ambiguous. A model told to return positive, negative, or neutral will always pick one — never saying I am not sure.
In production systems, acting on uncertain classifications as if they were certain causes costly errors: misrouted support tickets, wrong recommendations, inaccurate reports.
Uncertainty quantification in classification prompts solves this problem.
Confidence Scores 1-10
Asking the model to rate its confidence on a numeric scale gives a granular signal that downstream systems can threshold:
import anthropic, json
client = anthropic.Anthropic(api_key='YOUR_API_KEY')
def classify_with_confidence(text):
prompt = f'''
Classify the sentiment of the text below.
Return JSON:
{{
"sentiment": "positive|negative|neutral",
"confidence": 1-10,
"reason": "brief explanation of confidence level"
}}
Confidence scale: 10=completely certain, 1=total guess, 5=genuinely ambiguous
Text: {text}
'''
r = client.messages.create(
model='claude-opus-4-5', max_tokens=100,
messages=[{'role': 'user', 'content': prompt}]
)
return json.loads(r.content[0].text)
print(classify_with_confidence('I sort of liked it but the wait was too long.'))
print(classify_with_confidence('This product is absolutely outstanding!'))All lessons in this course
- Named Entity Extraction Prompts
- Schema-Driven Data Extraction
- LLM as Text Classifier
- Confidence and Uncertainty in Classification