Claude Architect · レッスン

エスカレーションとメトリクスのアンチパターン

感情によるエスカレーション、信頼度スコア、集約だけのメトリクスについて学びます

「エスカレーションとメトリクスのアンチパターン」はCoddyKit上の無料Claude Architectレッスンです。これはレッスン4/4です。下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。これはClaude Architect学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 Claude Architectコースには全4レッスンが含まれています。

このレッスンの一部はまだ翻訳されておらず、英語で表示されています。

Two Silent Failure Modes

Production agents rarely fail loudly. They fail by handing off to humans for the wrong reasons and by reporting health metrics that hide real damage.

This lesson dissects two exam-favorite anti-patterns: escalation driven by sentiment or self-rated confidence, and aggregate-only accuracy metrics. Both feel reasonable, both pass a demo, and both quietly erode trust at scale.

Architect-grade systems escalate on defensible triggers and measure performance stratified by what actually matters.

What a GOOD Escalation Trigger Looks Like

Escalation to a human is a privileged action. It should fire on triggers you can defend in an audit:

Explicit human request — the customer asks for a person, you escalate immediately.
Policy gaps — no rule covers this case.
No progress — repeated attempts have not resolved the issue.
Threshold violations — e.g. a refund exceeds an allowed limit.

Each of these maps to a concrete, observable fact in the conversation or the tool results — not to a guess about how the user feels.

def should_escalate(turn):
    if turn.customer_requested_human:
        return True            # explicit request -> escalate now
    if turn.no_matching_policy:
        return True            # policy gap
    if turn.attempts >= turn.max_attempts and not turn.resolved:
        return True            # no progress after real attempts
    if turn.refund_amount > REFUND_LIMIT:
        return True            # threshold violation
    return False

The Sentiment Escalation Trap

It is tempting to wire escalation to a sentiment classifier: "if the user sounds angry, hand off to a human."

This is a bad trigger. Sentiment is noisy, easy to misread, and uncorrelated with whether the agent can actually solve the problem. A calm user with an impossible request still needs escalation; a frustrated user with a one-step fix does not.

Escalating on emotion floods your human queue with solvable tickets and trains the agent to give up instead of resolving.

Self-Rated Confidence Is Not a Trigger Either

The second seductive anti-pattern: ask the model to rate its own confidence 1-10 and escalate when it is "low."

A model's self-rated confidence is not calibrated. It can be supremely confident while wrong and hesitant while correct. The same applies to untrained classifiers bolted onto the pipeline.

Confidence theater gives you a number that looks like a signal but carries no reliable information about correctness.

# Anti-pattern: escalate on the model's own confidence score
ESCALATE_IF = """Rate your confidence 1-10. If <= 4, escalate."""
# Problem: that 1-10 number is uncalibrated. The model may rate
# a hallucinated answer 9/10. This signal is not trustworthy.

The Right Pattern for Emotional Conversations

Frustration is real and worth handling — just not as an escalation trigger. The correct pattern is a sequence:

Acknowledge the emotion — show the user they were heard.
Propose a concrete solution — actually attempt to resolve the issue.
Escalate only if the request is reiterated — if the user still insists on a human after a genuine attempt.

This resolves most cases in-agent and reserves human time for situations that truly need it.

SYSTEM = """When a customer is upset:
1. Acknowledge their frustration briefly and sincerely.
2. Propose a concrete next step using your tools.
3. Escalate to a human ONLY if they reiterate the request
   for a person after you have attempted a solution.
Never escalate based on tone alone."""

Ambiguous Identity: Ask, Don't Guess

A related reliability rule shows up in the Customer Support scenario. When a lookup returns multiple matching customers, the agent must ask for more identifiers — never guess which record is correct.

Guessing risks acting on the wrong account: refunding the wrong order, leaking another person's data. "Most likely match" is not good enough when the action is irreversible.

Like escalation, identity resolution should rest on observable facts, not probabilistic hunches.

result = get_customer(email=email)
if len(result.matches) > 1:
    # Do NOT pick the first / 'most likely' match.
    return ask_user(
        "I found multiple accounts. Can you share your order "
        "number or postal code so I can find the right one?"
    )

Deterministic Enforcement for Hard Limits

Some escalation triggers are really business rules — and business rules with financial, legal, or safety stakes must not rely on the prompt.

Prompts are roughly 90% probabilistic; hooks are 100% deterministic. A PostToolUse or outgoing-call hook can block a policy-violating action (e.g. a refund over $500) before it ever executes, regardless of what the model decided.

If a threshold violation has real consequences, enforce it with a hook, not a hopeful instruction.

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "process_refund",
      "command": "./guards/block_refund_over_500.sh"
    }]
  }
}
// Hook rejects refund_amount > 500 deterministically,
// forcing escalation instead of trusting the prompt.

Why Aggregate Accuracy Lies

Switching to metrics: a single headline number like "97% accuracy" is one of the most dangerous things on an architect's dashboard.

An aggregate average can hide poor performance on a specific document type or field. Your extractor might be 99% accurate on invoices and 60% accurate on handwritten receipts — and the blended 97% looks great while the receipt path is quietly broken.

Aggregate-only metrics give false confidence and delay the discovery of localized failures.

Stratify by What Matters

The fix is stratified random sampling plus field-level confidence. Instead of one global score, you break performance down across the dimensions that carry risk: document type, field, customer segment, language.

Stratification surfaces the 60%-accurate receipt path that the average concealed. It turns "the system works" into "the system works here and fails there" — which is the only statement you can act on.

# Don't report one number. Report per-stratum accuracy.
for doc_type in ("invoice", "receipt", "handwritten"):
    sample = stratified_sample(labeled_set, doc_type, n=200)
    for field in REQUIRED_FIELDS:
        acc = field_accuracy(sample, field)
        print(doc_type, field, acc)   # exposes hidden weak spots

Calibrate Before You Automate

Field-level confidence is only useful if it is calibrated on a labeled validation set before you let it gate automation. Calibration tells you what a confidence of 0.8 actually means in terms of real-world correctness.

Without calibration you are back to confidence theater — the same flaw as a model self-rating 1-10. The discipline is identical for escalation and for metrics: trust a number only after you have shown it tracks reality.

Detecting Discrepancies, Not Just Reporting Scores

Good measurement also builds in self-checks. For extraction, have the model surface both a calculated_total and a stated_total so a downstream validator can flag mismatches — a concrete, verifiable signal instead of a vibe.

This pairs naturally with stratified metrics: discrepancies cluster in exactly the strata your aggregate number was hiding. Measure where it breaks, then enforce the limits with deterministic guards.

schema = {
    "type": "object",
    "properties": {
        "calculated_total": {"type": "number"},
        "stated_total": {"type": "number"}
    },
    "required": ["calculated_total", "stated_total"]
}
# Validator compares the two; a gap is a hard, actionable signal.

Quick Check: Escalation Trigger

Apply the rule to a real design decision.

Recap: Defensible Triggers, Honest Metrics

Key takeaways:

Good escalation triggers: explicit human request, policy gaps, no progress, threshold violations.
Bad triggers: sentiment, model self-rated confidence, untrained classifiers — none are calibrated signals.
Emotional cases: acknowledge, propose a solution, escalate only if reiterated. With multiple identity matches, ask for more identifiers — never guess.
Hard limits (refund > $500) belong in deterministic hooks, not prompts.
Aggregate accuracy hides weak document types and fields. Use stratified sampling and field-level confidence calibrated on a labeled set before automating.

Escalate on facts; measure where it breaks.

よくある質問

「エスカレーションとメトリクスのアンチパターン」レッスンは無料ですか？

はい。「エスカレーションとメトリクスのアンチパターン」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、Claude Architectコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 Claude Architectコースには全4レッスンが含まれています。

「エスカレーションとメトリクスのアンチパターン」で何を学びますか？

感情によるエスカレーション、信頼度スコア、集約だけのメトリクスについて学びますブラウザで直接実行するハンズオンコードでClaude Architectを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。

Claude Architectを始めるのに経験は必要ですか？

事前経験は必要ありません。CoddyKitのClaude Architectは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。これはレッスン4/4です。

「エスカレーションとメトリクスのアンチパターン」レッスンにはどのくらい時間がかかりますか？

ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。

このClaude Architectレッスンでコードを書いて実行できますか？

はい。すべてのClaude Architectレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。

このコースのすべてのレッスン

← Claude Architectに戻る