0Pricing
AI Prompt Engineering · 课时

自我改进的提示词系统

通过反馈 → 批评 → 重写循环自动优化提示词。

自我改进的提示词系统 是 CoddyKit 上的免费 AI Prompt Engineering 课时。 这是第 3 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 AI Prompt Engineering 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 AI Prompt Engineering 课程共包含 4 节课。

自我改进循环

自我改进提示词系统会创建一个反馈循环：使用提示词处理测试用例，评估输出，批评提示词，重写提示词，然后重复这一过程。每次迭代都应产生可量化的更好结果。这是在每轮循环中无需人工干预的自动化提示词优化。

循环架构概览

自我改进循环包含五个组件：执行器（运行提示词）、评估器（为输出评分）、批评器（找出提示词弱点）、重写器（改进提示词）和历史记录（跟踪所有迭代）。每个组件都通过一次 LLM 调用实现。

import anthropic

client = anthropic.Anthropic(api_key='YOUR_API_KEY')

class SelfImprovingPromptSystem:
    def __init__(self, initial_prompt, test_cases, eval_fn, max_iterations=5):
        self.current_prompt = initial_prompt
        self.test_cases = test_cases
        self.eval_fn = eval_fn
        self.max_iterations = max_iterations
        self.history = []  # [(iteration, prompt, score, critique)]

    def run(self):
        for i in range(self.max_iterations):
            print(f'=== Iteration {i+1}/{self.max_iterations} ===')
            score = self._evaluate_prompt()
            print(f'Score: {score:.2f}')
            self.history.append((i, self.current_prompt, score))
            if score >= 0.95:
                print('Target score reached. Stopping.')
                break
            critique = self._critique_prompt(score)
            self.current_prompt = self._rewrite_prompt(critique)
        return self.best_prompt()

    def best_prompt(self):
        return max(self.history, key=lambda x: x[2])[1]

执行器：运行提示词

执行器会将当前提示词应用于每个测试用例，并收集输出。这是标准的 LLM 调用循环——这里没有特别之处，但记录每个输出对应的测试用例至关重要。

def _execute_prompt(self, prompt):
    outputs = []
    for case in self.test_cases:
        response = client.messages.create(
            model='claude-haiku-4-5',  # use cheaper model for execution
            max_tokens=500,
            messages=[
                {'role': 'user', 'content': prompt + '\n\nInput: ' + case['input']}
            ]
        )
        outputs.append({
            'case_id': case['id'],
            'input': case['input'],
            'expected': case['expected'],
            'actual': response.content[0].text
        })
    return outputs

# Bind method to class (demonstration)
SelfImprovingPromptSystem._execute = _execute_prompt

# Sample test cases
test_cases = [
    {'id': 1, 'input': 'The meeting was cancelled.', 'expected': 'negative'},
    {'id': 2, 'input': 'Great product, love it!', 'expected': 'positive'},
    {'id': 3, 'input': 'It arrived on time.', 'expected': 'neutral'},
]
print(f'Test suite: {len(test_cases)} cases')

评估器：为输出评分

评估器会将执行器的输出与预期答案进行比较并评分。对于开放式任务，可以使用完全匹配、模糊匹配或单独的 LLM 评判器。

def _evaluate_prompt(self):
    outputs = self._execute(self.current_prompt)
    correct = 0
    failed_cases = []
    for out in outputs:
        # Exact match for classification tasks
        if out['expected'].lower() in out['actual'].lower():
            correct += 1
        else:
            failed_cases.append(out)
    score = correct / len(outputs)
    self._last_failed_cases = failed_cases
    return score

# For open-ended tasks: LLM-as-judge evaluator
JUDGE_PROMPT = '''Rate the quality of this AI response (1-5).
Task: {task_description}
Input: {input}
Expected approach: {expected}
Actual response: {actual}

Return only the integer score (1-5). No explanation.'''

def llm_judge_score(task_desc, input_text, expected, actual):
    response = client.messages.create(
        model='claude-haiku-4-5', max_tokens=5,
        messages=[{'role': 'user', 'content':
            JUDGE_PROMPT.format(
                task_description=task_desc, input=input_text,
                expected=expected, actual=actual
            )}]
    )
    try:
        return int(response.content[0].text.strip()) / 5.0
    except ValueError:
        return 0.5

批评器：识别提示词弱点

批评器会分析失败用例和当前提示词，找出具体弱点。这是元提示词的关键步骤——模型会批评自己的提示词。

CRITIC_PROMPT = '''You are a prompt engineering expert analyzing why a prompt fails.

Current prompt:
{current_prompt}

Failed test cases (where the prompt gave wrong outputs):
{failed_cases}

For each failure, explain:
1. What went wrong in the output
2. Which part of the prompt caused or failed to prevent this
3. A specific, actionable fix

End with a prioritized list of the top 3 prompt improvements to make.
Be specific — quote the relevant prompt section and suggest the exact replacement.'''

def _critique_prompt(self, score):
    failed_json = '\n'.join(
        f'Input: {c["input"]}\nExpected: {c["expected"]}\nActual: {c["actual"]}'
        for c in self._last_failed_cases[:5]
    )
    response = client.messages.create(
        model='claude-opus-4-5', max_tokens=1000,
        messages=[{'role': 'user', 'content':
            CRITIC_PROMPT.format(
                current_prompt=self.current_prompt,
                failed_cases=failed_json
            )}]
    )
    return response.content[0].text

重写器：改进提示词

重写器会接收批评器的分析，并生成改进版本的提示词。关键约束是：改进应针对具体问题，而不是进行全面重写。

REWRITER_PROMPT = '''You are a prompt engineer. Improve the prompt based on the critique below.

Current prompt:
{current_prompt}

Critique and suggested improvements:
{critique}

Rules for rewriting:
1. Make TARGETED changes based on the critique — do not rewrite everything
2. Keep all parts of the prompt that were working well
3. Apply all prioritized fixes from the critique
4. Do not add unnecessary verbosity — conciseness is a quality
5. Output ONLY the improved prompt, no explanation

Improved prompt:'''

def _rewrite_prompt(self, critique):
    response = client.messages.create(
        model='claude-opus-4-5', max_tokens=1000,
        messages=[{'role': 'user', 'content':
            REWRITER_PROMPT.format(
                current_prompt=self.current_prompt,
                critique=critique
            )}]
    )
    new_prompt = response.content[0].text.strip()
    print(f'Prompt updated. Length: {len(new_prompt)} chars '
          f'(was {len(self.current_prompt)} chars)')
    return new_prompt

完整循环：整合各部分

下面是将所有组件连接起来并包含基本收敛检测的完整自我改进循环。

def run_improvement_loop(initial_prompt, test_cases, max_iterations=5,
                          target_score=0.90):
    history = []
    current_prompt = initial_prompt
    last_failed_cases = []

    for i in range(max_iterations):
        print(f'\n--- Iteration {i+1} ---')

        # Execute
        outputs = execute_prompt(current_prompt, test_cases)

        # Evaluate
        score, failed_cases = evaluate_outputs(outputs)
        last_failed_cases = failed_cases
        print(f'Score: {score:.2f} ({len(failed_cases)} failures)')
        history.append({'iteration': i, 'prompt': current_prompt, 'score': score})

        if score >= target_score:
            print(f'Target score {target_score} reached!')
            break

        if not failed_cases:
            print('No failures to learn from. Stopping.')
            break

        # Critique and rewrite
        critique = critique_prompt(current_prompt, failed_cases)
        current_prompt = rewrite_prompt(current_prompt, critique)

    best = max(history, key=lambda x: x['score'])
    print(f'\nBest prompt at iteration {best["iteration"]+1} with score {best["score"]:.2f}')
    return best['prompt'], history

自我改进循环中的成本管理

自我改进循环可能成本高昂——每次迭代都会发起多次接口调用。成本管理策略可以控制循环的成本。

# Cost optimization strategies

# 1. Use cheap model for execution, expensive model for critique/rewrite
def execute_prompt(prompt, test_cases):
    # Use cheapest capable model
    model = 'claude-haiku-4-5'
    # ... execute ...
    pass

def critique_prompt(prompt, failed_cases):
    # Use best model for reasoning about why the prompt fails
    model = 'claude-opus-4-5'
    # ... critique ...
    pass

# 2. Limit test suite size (sample from larger set)
import random

def get_evaluation_sample(full_test_suite, sample_size=20):
    if len(full_test_suite) <= sample_size:
        return full_test_suite
    return random.sample(full_test_suite, sample_size)

# 3. Early stopping: stop if score doesn't improve
def has_converged(history, patience=2, min_delta=0.02):
    if len(history) < patience + 1:
        return False
    recent_scores = [h['score'] for h in history[-patience:]]
    best_recent = max(recent_scores)
    baseline = history[-(patience+1)]['score']
    return (best_recent - baseline) < min_delta

print('Cost optimization: cheap model for execution, expensive for critique')

跟踪提示词溯源

在自我改进循环中，每个提示词版本都应能够追溯到触发它的失败分析。这种溯源关系有助于调试意外回归，也支持人工审查。

class PromptLineage:
    def __init__(self):
        self.lineage = []

    def record(self, iteration, prompt, score, critique=None,
                failed_case_ids=None):
        self.lineage.append({
            'iteration': iteration,
            'prompt': prompt,
            'score': score,
            'critique_summary': critique[:100] if critique else None,
            'failed_case_ids': failed_case_ids or [],
            'prompt_length': len(prompt.split())
        })

    def print_history(self):
        print('Prompt improvement history:')
        for entry in self.lineage:
            print(
                f'  Iter {entry["iteration"]}: '
                f'score={entry["score"]:.2f} '
                f'words={entry["prompt_length"]} '
                f'failures={len(entry["failed_case_ids"])}'
            )

    def get_best(self):
        return max(self.lineage, key=lambda x: x['score'])

lineage = PromptLineage()
lineage.record(0, 'Initial simple prompt', 0.60)
lineage.record(1, 'Improved with examples', 0.75, 'Missing edge cases')
lineage.record(2, 'Added edge case handling', 0.92, 'Minor format issue')
lineage.print_history()

提示词版本历史可视化

将各次迭代中的分数变化可视化，有助于确定循环是否正在收敛以及收敛速度。一张简单的文本图表无需图形库即可让趋势一目了然。

def visualize_improvement_history(history):
    '''
    history: list of (iteration, prompt, score)
    Prints an ASCII chart of score progression.
    '''
    if not history:
        print('No history to visualize.')
        return

    max_score = 1.0
    bar_width = 40
    print('\nScore Progression:')
    print('-' * (bar_width + 20))

    for iteration, prompt, score in history:
        filled = int(score * bar_width)
        bar = '#' * filled + '-' * (bar_width - filled)
        marker = ' <-- BEST' if score == max(h[2] for h in history) else ''
        print(f'Iter {iteration:2d}: [{bar}] {score:.3f}{marker}')

    final_score = history[-1][2]
    best_score = max(h[2] for h in history)
    gain = best_score - history[0][2]
    print('-' * (bar_width + 20))
    print(f'Initial: {history[0][2]:.3f} -> Best: {best_score:.3f} (gain: +{gain:.3f})')

# Example
sample_history = [
    (0, 'v0', 0.60),
    (1, 'v1', 0.72),
    (2, 'v2', 0.78),
    (3, 'v3', 0.77),
    (4, 'v4', 0.85)
]
visualize_improvement_history(sample_history)

回归检测

重写后的提示词可能修复某些失败用例，却导致之前通过的用例失败，这就是回归。每次迭代后都应通过比较具体测试用例的状态变化来检查回归。

def detect_regressions(old_outputs, new_outputs):
    old_results = {o['case_id']: o['correct'] for o in old_outputs}
    new_results = {o['case_id']: o['correct'] for o in new_outputs}

    regressions = []
    improvements = []

    for case_id in old_results:
        was_correct = old_results[case_id]
        is_correct = new_results.get(case_id, False)
        if was_correct and not is_correct:
            regressions.append(case_id)
        elif not was_correct and is_correct:
            improvements.append(case_id)

    print(f'Fixed: {len(improvements)} cases. Regressed: {len(regressions)} cases.')
    if regressions:
        print(f'WARNING: Regression on cases: {regressions}')
        print('Consider reverting to previous prompt version.')
    return regressions, improvements

# Example
old = [{'case_id': 1, 'correct': True}, {'case_id': 2, 'correct': False}]
new = [{'case_id': 1, 'correct': False}, {'case_id': 2, 'correct': True}]
detect_regressions(old, new)  # Fixed: 1, Regressed: 1

快速检查

在自我改进提示词循环中，批评器组件的作用是什么？

自我改进提示词系统总结

自我改进提示词系统会自动完成提示词优化循环：

  • 循环结构：执行 → 评估 → 批评 → 重写 → 重复
  • 执行器：在测试用例上运行提示词（使用低成本模型）
  • 评估器：使用完全匹配或 LLM 评判器为输出评分
  • 批评器：从失败用例中识别具体提示词弱点的元提示词
  • 重写器：利用批评结果进行针对性改进（使用最佳模型）
  • 收敛：达到目标分数或分数不再提升时停止
  • 回归检测：始终检查改进是否导致原本通过的用例失败

常见问题解答

「自我改进的提示词系统」课时是免费的吗？

是的 — 「自我改进的提示词系统」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 AI Prompt Engineering 课程的其余内容，请升级到 CoddyKit PRO。 AI Prompt Engineering 课程共包含 4 节课。

「自我改进的提示词系统」这节课中我会学到什么？

通过反馈 → 批评 → 重写循环自动优化提示词。 你通过在浏览器中直接运行的动手代码来练习 AI Prompt Engineering，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 AI Prompt Engineering 需要有经验吗？

无需任何先前经验。CoddyKit 上的 AI Prompt Engineering 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 3 节课，共 4 节。

「自我改进的提示词系统」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 AI Prompt Engineering 课中编写并运行代码吗？

能。每节 AI Prompt Engineering 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. 什么是元提示词？
  2. 生成提示词的提示词
  3. 自我改进的提示词系统
  4. 自我改进中的评估与选择
← 返回 AI Prompt Engineering