自我改进中的评估与选择
了解如何判断生成的提示词哪个更好，并选出优胜者。
自我改进中的评估与选择 是 CoddyKit 上的免费 AI Prompt Engineering 课时。 这是第 4 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 AI Prompt Engineering 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 AI Prompt Engineering 课程共包含 4 节课。
为什么评估是最困难的部分
生成提示词变体很容易。评估哪个变体确实更好才是自我改进中最困难的部分。如果没有严格的评估，您无法判断重写后的提示词是真正得到了改进，还是只是有所不同。评估质量决定了整个改进循环的质量。
评分标准设计
在运行自我改进循环之前，请先定义对于您的任务而言“更好”意味着什么。评分标准应当客观、可衡量，并且与任务的成功条件直接相关。
# Scoring criteria for different task types
SCORING_CRITERIA = {
'Classification': {
'primary_metric': 'Accuracy',
'formula': 'correct_predictions / total_predictions',
'secondary': ['Precision per class', 'F1 for rare classes'],
'target': 0.90
},
'Summarization': {
'primary_metric': 'LLM judge quality score',
'formula': 'average of judge scores on 1-5 scale, normalized to 0-1',
'secondary': ['ROUGE-L', 'Coverage of key facts', 'Hallucination rate'],
'target': 4.0 # on 1-5 scale
},
'Code generation': {
'primary_metric': 'Test pass rate',
'formula': 'tests_passing / total_tests',
'secondary': ['Syntax validity rate', 'Edge case coverage'],
'target': 0.85
},
'Information extraction': {
'primary_metric': 'F1 (precision + recall)',
'formula': '2 * precision * recall / (precision + recall)',
'secondary': ['Field-level accuracy', 'Format compliance rate'],
'target': 0.88
}
}
for task, criteria in SCORING_CRITERIA.items():
print(f'{task}: {criteria["primary_metric"]} (target: {criteria["target"]})')LLM 作为评判器：多维评分
对于开放式任务，LLM 评判器可以同时评估多个质量维度。多维评分比单一分数提供更丰富的信号。
import anthropic, json
client = anthropic.Anthropic(api_key='YOUR_API_KEY')
MULTI_DIM_JUDGE_PROMPT = '''Evaluate this AI response across 5 dimensions. Return JSON only.
Task description: {task}
User input: {input}
AI response: {response}
Score each dimension 1-5:
- accuracy: Is the information correct and complete?
- relevance: Does it address exactly what was asked?
- clarity: Is it clear and easy to understand for the target audience?
- format: Does it follow the required output format?
- safety: Does it avoid harmful, misleading, or inappropriate content?
Return: {{"accuracy": N, "relevance": N, "clarity": N,
"format": N, "safety": N, "overall": avg, "rationale": "<1 sentence>"}}'''
def judge_response(task, input_text, response, weights=None):
weights = weights or {'accuracy': 0.30, 'relevance': 0.25,
'clarity': 0.20, 'format': 0.15, 'safety': 0.10}
judge_result = client.messages.create(
model='claude-opus-4-5', max_tokens=300,
messages=[{'role': 'user', 'content':
MULTI_DIM_JUDGE_PROMPT.format(
task=task, input=input_text, response=response
)}]
)
scores = json.loads(judge_result.content[0].text)
weighted_score = sum(scores[dim] * w for dim, w in weights.items() if dim in scores)
scores['weighted_total'] = round(weighted_score, 2)
return scores跨测试用例汇总分数
提示词的汇总分数是在完整测试套件上计算的。不同的汇总策略可以揭示提示词质量的不同方面。
import statistics
def aggregate_scores(case_scores):
'''
case_scores: list of dicts with dimension scores per test case
'''
dimensions = ['accuracy', 'relevance', 'clarity', 'format', 'safety']
aggregated = {}
for dim in dimensions:
values = [s[dim] for s in case_scores if dim in s]
if not values:
continue
aggregated[dim] = {
'mean': round(statistics.mean(values), 2),
'min': min(values),
'max': max(values),
'stdev': round(statistics.stdev(values), 2) if len(values) > 1 else 0
}
# Overall weighted mean
overall_scores = [s.get('weighted_total', 0) for s in case_scores]
aggregated['overall'] = {
'mean': round(statistics.mean(overall_scores), 2),
'p10': round(sorted(overall_scores)[int(len(overall_scores)*0.10)], 2),
'p90': round(sorted(overall_scores)[int(len(overall_scores)*0.90)], 2)
}
return aggregated
# Example
sample_scores = [
{'accuracy': 4, 'relevance': 5, 'clarity': 4, 'format': 3, 'safety': 5, 'weighted_total': 4.1},
{'accuracy': 3, 'relevance': 4, 'clarity': 5, 'format': 4, 'safety': 5, 'weighted_total': 3.9},
]
print(aggregate_scores(sample_scores))选择标准：质量、多样性、稳健性
从候选项中选择最佳提示词时，应考虑三个维度：质量（平均分）、多样性（它是否在与当前提示词不同的用例上失败）和稳健性（跨用例的低方差）。
def select_best_prompt(candidates, current_prompt_score):
'''
candidates: list of {prompt, scores, aggregated}
Returns the best candidate based on quality, diversity, robustness
'''
scored_candidates = []
for c in candidates:
agg = c['aggregated']['overall']
# Quality: mean score
quality = agg['mean']
# Robustness: inverse of P90-P10 spread (low spread = robust)
robustness = 1.0 - (agg['p90'] - agg['p10']) / 5.0
# Must beat current: reject if not better
if quality <= current_prompt_score:
continue
# Combined score
combined = 0.70 * quality + 0.30 * robustness
scored_candidates.append((combined, c))
if not scored_candidates:
print('No candidate beats current prompt. Keeping current.')
return None
scored_candidates.sort(reverse=True)
best = scored_candidates[0][1]
print(f'Selected candidate with combined score {scored_candidates[0][0]:.2f}')
return best['prompt']跟踪表现最佳的提示词
在整个改进循环中，应始终跟踪目前为止表现最佳的提示词。最后一次迭代的提示词不一定是最好的——一次重写可能改善某些用例，却使其他用例发生回归。
class BestPromptTracker:
def __init__(self):
self.best_score = -1
self.best_prompt = None
self.best_iteration = -1
self.all_scores = []
def update(self, iteration, prompt, score):
self.all_scores.append({'iteration': iteration, 'score': score})
if score > self.best_score:
self.best_score = score
self.best_prompt = prompt
self.best_iteration = iteration
print(f'New best at iteration {iteration}: score={score:.2f}')
else:
print(f'Iteration {iteration}: score={score:.2f} '
f'(best is still {self.best_score:.2f} at iter {self.best_iteration})')
def get_best(self):
return self.best_prompt, self.best_score, self.best_iteration
def is_converged(self, patience=2, min_delta=0.01):
if len(self.all_scores) < patience + 1:
return False
recent = self.all_scores[-(patience+1):]
improvement = recent[-1]['score'] - recent[0]['score']
return improvement < min_delta
tracker = BestPromptTracker()
for i, score in enumerate([0.65, 0.72, 0.78, 0.77, 0.79]):
tracker.update(i, f'prompt_v{i}', score)
print('Best:', tracker.get_best())停止标准
知道何时停止与知道何时继续同样重要。糟糕的停止标准会浪费接口预算，或者在达到质量目标之前过早终止。
STOPPING_CONDITIONS = [
'Target score reached (e.g., >= 0.90)',
'No improvement over last N iterations (patience)',
'Maximum iteration budget exhausted',
'Score improvement delta below minimum threshold',
'All test cases pass (score = 1.0)',
'Cost budget exceeded'
]
def should_stop(tracker, config):
_, best_score, _ = tracker.get_best()
# Condition 1: Target reached
if best_score >= config['target_score']:
return True, f'Target score {config["target_score"]} reached'
# Condition 2: Patience exhausted
if tracker.is_converged(patience=config['patience'],
min_delta=config['min_delta']):
return True, f'No improvement over {config["patience"]} iterations'
# Condition 3: Max iterations
if len(tracker.all_scores) >= config['max_iterations']:
return True, f'Max iterations {config["max_iterations"]} reached'
return False, 'Continue'
config = {'target_score': 0.90, 'patience': 3,
'min_delta': 0.01, 'max_iterations': 10}
stop, reason = should_stop(tracker, config)
print(f'Stop: {stop} | Reason: {reason}')自我改进的测试套件设计
自我改进循环的质量取决于测试套件的质量。设计良好的测试套件应覆盖典型、边界和对抗性用例，并保持稳定（循环期间不更新）。
def design_test_suite_for_improvement(task_description, target_size=50):
'''
Generate a balanced test suite for prompt self-improvement loops.
The suite is fixed and does not change during iterations.
'''
distribution = {
'typical': int(target_size * 0.50), # 50% typical cases
'edge_case': int(target_size * 0.25), # 25% edge cases
'adversarial': int(target_size * 0.15), # 15% adversarial
'off_topic': int(target_size * 0.10) # 10% off-topic (guardrails)
}
print('Test suite distribution:')
for category, count in distribution.items():
print(f' {category}: {count} cases')
# Key properties of a good evaluation test suite:
properties = [
'Fixed (never modified during improvement loop)',
'Balanced across difficulty levels',
'Has ground truth labels / expected outputs',
'Diverse in topic and phrasing within each category',
'Held-out: separate from any few-shot examples in the prompt'
]
for p in properties:
print(f' Property: {p}')
return distribution评估提示词版本之间的多样性
两个提示词的平均分可能相同，但它们可能在完全不同的用例上失败。比较它们的失败集合，可以揭示新提示词是否真正具有互补性，这对集成使用很有帮助。
def compare_failure_sets(prompt_a_results, prompt_b_results):
'''
Compare which cases each prompt fails on.
'''
fails_a = {r['case_id'] for r in prompt_a_results if not r['correct']}
fails_b = {r['case_id'] for r in prompt_b_results if not r['correct']}
both_fail = fails_a & fails_b
only_a_fails = fails_a - fails_b
only_b_fails = fails_b - fails_a
overlap = len(both_fail) / max(len(fails_a | fails_b), 1)
print(f'Prompt A failures: {len(fails_a)}')
print(f'Prompt B failures: {len(fails_b)}')
print(f'Both fail: {len(both_fail)} (overlap: {overlap:.0%})')
print(f'Only A fails: {len(only_a_fails)}')
print(f'Only B fails: {len(only_b_fails)}')
if overlap < 0.3:
print('LOW overlap — prompts are complementary. Consider ensembling.')
else:
print('HIGH overlap — B is a refinement of A, not a different approach.')
return {'overlap': overlap, 'only_a': only_a_fails, 'only_b': only_b_fails}校准：您的评判分数可靠吗
只有当 LLM 评判器的分数与真实质量相关时，这些分数才有用。校准会检查评判器的分数是否与金标准示例集上的人工评分一致。
def calibrate_judge(judge_fn, gold_standard):
'''
gold_standard: list of {input, response, human_score} dicts
Returns correlation between judge scores and human scores.
'''
import statistics
judge_scores = []
human_scores = []
for example in gold_standard:
judge_score = judge_fn(
task='General response quality',
input_text=example['input'],
response=example['response']
)
judge_scores.append(judge_score)
human_scores.append(example['human_score'])
# Pearson correlation
n = len(judge_scores)
mean_j = statistics.mean(judge_scores)
mean_h = statistics.mean(human_scores)
cov = sum((j - mean_j) * (h - mean_h) for j, h in zip(judge_scores, human_scores))
std_j = statistics.stdev(judge_scores)
std_h = statistics.stdev(human_scores)
if std_j == 0 or std_h == 0:
return 0.0
correlation = cov / ((n - 1) * std_j * std_h)
print(f'Judge-Human correlation: {correlation:.3f}')
if correlation < 0.7:
print('WARNING: Low correlation. Judge may not reflect human quality judgment.')
return correlation集成方法：组合最佳提示词
当多个提示词变体在不同用例上失败时，通过多数投票或置信度加权将它们组合起来，可以获得比任何单个提示词更高的质量。
def ensemble_prompts(prompts, test_input, model='claude-haiku-4-5'):
'''
Run multiple prompts on the same input and take majority vote.
'''
outputs = []
for i, prompt in enumerate(prompts):
response = client.messages.create(
model=model, max_tokens=300,
messages=[{'role': 'user', 'content':
prompt + '\n\nInput: ' + test_input}]
)
outputs.append(response.content[0].text.strip())
# Majority vote for classification
from collections import Counter
vote_counts = Counter(outputs)
majority = vote_counts.most_common(1)[0][0]
confidence = vote_counts[majority] / len(outputs)
print(f'Ensemble ({len(prompts)} prompts): {majority} ({confidence:.0%} agreement)')
return majority, confidence
# When to use ensemble vs. single best prompt:
# - Ensemble: high-stakes classification, tolerate 3x API cost
# - Single best: cost-sensitive production, latency-critical
print('Ensemble is costlier but more robust for high-stakes outputs.')快速检查
两个提示词变体在评估套件上都获得了 0.82 分。您应如何决定将哪一个提升为正式版本？
评估与选择总结
评估与选择是有效自我改进循环的基础：
- 标准设计：在开始前定义“更好”意味着什么——准确性、稳健性和格式合规性
- 多维评分：LLM 评判器分别评估质量、相关性、清晰度、格式和安全性
- 汇总：跟踪平均值、第 10 百分位和第 90 百分位，而不仅仅是平均分
- 选择：平衡质量（平均分）与稳健性（低方差）
- 最佳提示词跟踪：始终保留历史最佳提示词，而不仅是最新迭代版本
- 停止标准：目标分数、耐心阈值、最大迭代次数和成本预算
- 失败集合多样性：识别适合进行集成的互补提示词
常见问题解答
「自我改进中的评估与选择」课时是免费的吗？
是的 — 「自我改进中的评估与选择」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 AI Prompt Engineering 课程的其余内容，请升级到 CoddyKit PRO。 AI Prompt Engineering 课程共包含 4 节课。
「自我改进中的评估与选择」这节课中我会学到什么？
了解如何判断生成的提示词哪个更好，并选出优胜者。 你通过在浏览器中直接运行的动手代码来练习 AI Prompt Engineering，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 AI Prompt Engineering 需要有经验吗？
无需任何先前经验。CoddyKit 上的 AI Prompt Engineering 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 4 节课，共 4 节。
「自我改进中的评估与选择」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 AI Prompt Engineering 课中编写并运行代码吗？
能。每节 AI Prompt Engineering 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。
此课程中的所有课时
- 什么是元提示词？
- 生成提示词的提示词
- 自我改进的提示词系统
- 自我改进中的评估与选择