升级与指标反模式
基于情绪升级、置信度评分以及仅使用汇总指标。
升级与指标反模式 是 CoddyKit 上的免费 Claude Architect 课时。 这是第 4 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Claude Architect 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Claude Architect 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
Two Silent Failure Modes
Production agents rarely fail loudly. They fail by handing off to humans for the wrong reasons and by reporting health metrics that hide real damage.
This lesson dissects two exam-favorite anti-patterns: escalation driven by sentiment or self-rated confidence, and aggregate-only accuracy metrics. Both feel reasonable, both pass a demo, and both quietly erode trust at scale.
Architect-grade systems escalate on defensible triggers and measure performance stratified by what actually matters.
What a GOOD Escalation Trigger Looks Like
Escalation to a human is a privileged action. It should fire on triggers you can defend in an audit:
- Explicit human request — the customer asks for a person, you escalate immediately.
- Policy gaps — no rule covers this case.
- No progress — repeated attempts have not resolved the issue.
- Threshold violations — e.g. a refund exceeds an allowed limit.
Each of these maps to a concrete, observable fact in the conversation or the tool results — not to a guess about how the user feels.
def should_escalate(turn):
if turn.customer_requested_human:
return True # explicit request -> escalate now
if turn.no_matching_policy:
return True # policy gap
if turn.attempts >= turn.max_attempts and not turn.resolved:
return True # no progress after real attempts
if turn.refund_amount > REFUND_LIMIT:
return True # threshold violation
return FalseThe Sentiment Escalation Trap
It is tempting to wire escalation to a sentiment classifier: "if the user sounds angry, hand off to a human."
This is a bad trigger. Sentiment is noisy, easy to misread, and uncorrelated with whether the agent can actually solve the problem. A calm user with an impossible request still needs escalation; a frustrated user with a one-step fix does not.
Escalating on emotion floods your human queue with solvable tickets and trains the agent to give up instead of resolving.
Self-Rated Confidence Is Not a Trigger Either
The second seductive anti-pattern: ask the model to rate its own confidence 1-10 and escalate when it is "low."
A model's self-rated confidence is not calibrated. It can be supremely confident while wrong and hesitant while correct. The same applies to untrained classifiers bolted onto the pipeline.
Confidence theater gives you a number that looks like a signal but carries no reliable information about correctness.
# Anti-pattern: escalate on the model's own confidence score
ESCALATE_IF = """Rate your confidence 1-10. If <= 4, escalate."""
# Problem: that 1-10 number is uncalibrated. The model may rate
# a hallucinated answer 9/10. This signal is not trustworthy.The Right Pattern for Emotional Conversations
Frustration is real and worth handling — just not as an escalation trigger. The correct pattern is a sequence:
- Acknowledge the emotion — show the user they were heard.
- Propose a concrete solution — actually attempt to resolve the issue.
- Escalate only if the request is reiterated — if the user still insists on a human after a genuine attempt.
This resolves most cases in-agent and reserves human time for situations that truly need it.
SYSTEM = """When a customer is upset:
1. Acknowledge their frustration briefly and sincerely.
2. Propose a concrete next step using your tools.
3. Escalate to a human ONLY if they reiterate the request
for a person after you have attempted a solution.
Never escalate based on tone alone."""Ambiguous Identity: Ask, Don't Guess
A related reliability rule shows up in the Customer Support scenario. When a lookup returns multiple matching customers, the agent must ask for more identifiers — never guess which record is correct.
Guessing risks acting on the wrong account: refunding the wrong order, leaking another person's data. "Most likely match" is not good enough when the action is irreversible.
Like escalation, identity resolution should rest on observable facts, not probabilistic hunches.
result = get_customer(email=email)
if len(result.matches) > 1:
# Do NOT pick the first / 'most likely' match.
return ask_user(
"I found multiple accounts. Can you share your order "
"number or postal code so I can find the right one?"
)Deterministic Enforcement for Hard Limits
Some escalation triggers are really business rules — and business rules with financial, legal, or safety stakes must not rely on the prompt.
Prompts are roughly 90% probabilistic; hooks are 100% deterministic. A PostToolUse or outgoing-call hook can block a policy-violating action (e.g. a refund over $500) before it ever executes, regardless of what the model decided.
If a threshold violation has real consequences, enforce it with a hook, not a hopeful instruction.
{
"hooks": {
"PostToolUse": [{
"matcher": "process_refund",
"command": "./guards/block_refund_over_500.sh"
}]
}
}
// Hook rejects refund_amount > 500 deterministically,
// forcing escalation instead of trusting the prompt.Why Aggregate Accuracy Lies
Switching to metrics: a single headline number like "97% accuracy" is one of the most dangerous things on an architect's dashboard.
An aggregate average can hide poor performance on a specific document type or field. Your extractor might be 99% accurate on invoices and 60% accurate on handwritten receipts — and the blended 97% looks great while the receipt path is quietly broken.
Aggregate-only metrics give false confidence and delay the discovery of localized failures.
Stratify by What Matters
The fix is stratified random sampling plus field-level confidence. Instead of one global score, you break performance down across the dimensions that carry risk: document type, field, customer segment, language.
Stratification surfaces the 60%-accurate receipt path that the average concealed. It turns "the system works" into "the system works here and fails there" — which is the only statement you can act on.
# Don't report one number. Report per-stratum accuracy.
for doc_type in ("invoice", "receipt", "handwritten"):
sample = stratified_sample(labeled_set, doc_type, n=200)
for field in REQUIRED_FIELDS:
acc = field_accuracy(sample, field)
print(doc_type, field, acc) # exposes hidden weak spotsCalibrate Before You Automate
Field-level confidence is only useful if it is calibrated on a labeled validation set before you let it gate automation. Calibration tells you what a confidence of 0.8 actually means in terms of real-world correctness.
Without calibration you are back to confidence theater — the same flaw as a model self-rating 1-10. The discipline is identical for escalation and for metrics: trust a number only after you have shown it tracks reality.
Detecting Discrepancies, Not Just Reporting Scores
Good measurement also builds in self-checks. For extraction, have the model surface both a calculated_total and a stated_total so a downstream validator can flag mismatches — a concrete, verifiable signal instead of a vibe.
This pairs naturally with stratified metrics: discrepancies cluster in exactly the strata your aggregate number was hiding. Measure where it breaks, then enforce the limits with deterministic guards.
schema = {
"type": "object",
"properties": {
"calculated_total": {"type": "number"},
"stated_total": {"type": "number"}
},
"required": ["calculated_total", "stated_total"]
}
# Validator compares the two; a gap is a hard, actionable signal.Quick Check: Escalation Trigger
Apply the rule to a real design decision.
Recap: Defensible Triggers, Honest Metrics
Key takeaways:
- Good escalation triggers: explicit human request, policy gaps, no progress, threshold violations.
- Bad triggers: sentiment, model self-rated confidence, untrained classifiers — none are calibrated signals.
- Emotional cases: acknowledge, propose a solution, escalate only if reiterated. With multiple identity matches, ask for more identifiers — never guess.
- Hard limits (refund > $500) belong in deterministic hooks, not prompts.
- Aggregate accuracy hides weak document types and fields. Use stratified sampling and field-level confidence calibrated on a labeled set before automating.
Escalate on facts; measure where it breaks.
常见问题解答
「升级与指标反模式」课时是免费的吗？
是的 — 「升级与指标反模式」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Claude Architect 课程的其余内容，请升级到 CoddyKit PRO。 Claude Architect 课程共包含 4 节课。
「升级与指标反模式」这节课中我会学到什么？
基于情绪升级、置信度评分以及仅使用汇总指标。 你通过在浏览器中直接运行的动手代码来练习 Claude Architect，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 Claude Architect 需要有经验吗？
无需任何先前经验。CoddyKit 上的 Claude Architect 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 4 节课，共 4 节。
「升级与指标反模式」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 Claude Architect 课中编写并运行代码吗？
能。每节 Claude Architect 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。