CAI 원칙과 비평 프롬프트

헌법적 인공지능이란 무엇입니까

헌법: 서면 원칙

CAI의 세 단계 순환 과정

1단계: 초기 응답 생성

import anthropic

client = anthropic.Anthropic(api_key='sk-ant-...')

# Step 1: Generate initial response
def generate_initial(user_request):
    response = client.messages.create(
        model='claude-opus-4-5',
        max_tokens=512,
        messages=[
            {'role': 'user', 'content': user_request}
        ]
    )
    return response.content[0].text

initial = generate_initial('How do I pick a lock?')
print('Initial response:', initial[:200])

2단계: 원칙에 따른 검토

import anthropic

client = anthropic.Anthropic(api_key='sk-ant-...')

PRINCIPLE = (
    'Choose the response that is least likely to provide instructions '
    'that could enable illegal activity or harm to others.'
)

def critique_response(user_request, initial_response, principle):
    critique_prompt = (
        f'Human: {user_request}\n\n'
        f'Assistant: {initial_response}\n\n'
        f'Critique the assistant response using this principle: {principle}\n'
        f'Identify specific problems and explain why they violate the principle.'
    )
    response = client.messages.create(
        model='claude-opus-4-5',
        max_tokens=256,
        messages=[{'role': 'user', 'content': critique_prompt}]
    )
    return response.content[0].text

critique = critique_response('How do I pick a lock?', initial, PRINCIPLE)
print('Critique:', critique[:200])

3단계: 검토 결과에 따른 수정

import anthropic

client = anthropic.Anthropic(api_key='sk-ant-...')

def revise_response(user_request, initial_response, critique_text):
    revise_prompt = (
        f'Human: {user_request}\n\n'
        f'Original assistant response: {initial_response}\n\n'
        f'Critique of that response: {critique_text}\n\n'
        f'Please rewrite the assistant response to address the critique '
        f'while still being as helpful as possible to the user.'
    )
    response = client.messages.create(
        model='claude-opus-4-5',
        max_tokens=512,
        messages=[{'role': 'user', 'content': revise_prompt}]
    )
    return response.content[0].text

revised = revise_response('How do I pick a lock?', initial, critique)
print('Revised response:', revised[:300])

적용할 원칙 선택

# Example principles from Anthropic's published constitution
PRINCIPLES = [
    'Choose the response that is least likely to contain harmful, '
    'unethical, racist, sexist, toxic, dangerous, or illegal content.',

    'Choose the response that a thoughtful, senior Anthropic employee '
    'would consider optimal given the context.',

    'Choose the response that is most likely to be true and accurate, '
    'even if it requires acknowledging uncertainty.',

    'Choose the response that would be most appropriate for children '
    'if the context is ambiguous about the audience.',
]

# For a medical advice context, apply principles 1 and 3
medical_principles = [PRINCIPLES[0], PRINCIPLES[2]]

# For a general chat context, apply principle 2
general_principles = [PRINCIPLES[1]]

RLHF에서의 CAI: SL-CAI와 RLCAI

유용하고 무해하며 정직한 — HHH 프레임워크

CAI 검토 프롬프트 템플릿

CRITIQUE_TEMPLATE = (
    'Review the following conversation and critique the assistant response.\n\n'
    'Conversation:\n'
    'Human: {user_message}\n'
    'Assistant: {assistant_response}\n\n'
    'Critique Request: Identify specific ways the assistant response could be '
    'improved to be more {principle_goal}. '
    'If the response is already excellent, say so briefly.\n\n'
    'Critique:'
)

# Example: critique for harmlessness
harmless_critique = CRITIQUE_TEMPLATE.format(
    user_message='How do I make someone angry at a party?',
    assistant_response='Here are some classic pranks you can try...',
    principle_goal='harmless and considerate of all people involved'
)

print(harmless_critique[:300])

CAI 순환 과정을 적용할 시점

지식 확인: CAI의 세 단계 순환 과정

복습: CAI 원칙과 검토 프롬프트