Zero-Shot、One-Shot、Few-Shot

ショット数のスペクトラム

from dataclasses import dataclass

@dataclass
class ICLConfig:
    k: int           # number of demonstrations
    selection: str   # 'static' | 'dynamic'
    order: str       # 'random' | 'similarity' | 'curriculum'

# Zero-shot is simply k=0
cfg = ICLConfig(k=0, selection='static', order='random')

ゼロショットが有効な場合

# Zero-shot with explicit schema beats vague few-shot
PROMPT = (
    'Classify sentiment as POSITIVE, NEGATIVE, or NEUTRAL.\n'
    'Respond with only the label.\n\n'
    'Text: ' + user_text + '\nLabel:'
)

フォーマットのアンカーとしてのワンショット

ONE_SHOT = (
    'Extract entities as JSON.\n\n'
    'Input: Apple released the iPhone in Cupertino.\n'
    'Output: {"org": ["Apple"], "product": ["iPhone"], "loc": ["Cupertino"]}\n\n'
    'Input: ' + query + '\nOutput:'
)

Few-shotとk曲線

def sweep_k(eval_set, candidates, ks=(1,2,4,8,16)):
    results = {}
    for k in ks:
        acc = evaluate(build_prompt(candidates[:k]), eval_set)
        results[k] = acc
    return max(results, key=results.get)

ICLが機能する理由：暗黙的推論

# Min, Lyu et al. (2022): label correctness matters less than
#   - the input distribution
#   - the label space (which classes exist)
#   - the format / structure
# Implication: invest in representative inputs + valid label set

トークン予算とコストのトレードオフ

def cost_efficiency(acc_by_k, tokens_by_k, price_per_1k):
    return {
        k: acc_by_k[k] / (tokens_by_k[k] / 1000 * price_per_1k)
        for k in acc_by_k
    }
# Pick the k maximizing accuracy per dollar, not raw accuracy

多数派ラベルバイアスと位置バイアス

# Contextual calibration (Zhao et al. 2021)
p_cf = model_probs(prompt_with_input('N/A'))  # content-free prior
W = 1.0 / p_cf                                  # diagonal correction
def calibrated(probs):
    return normalize(W * probs)

デモンストレーションセットのバランス調整

import random

def balanced_demos(pool, k, label_fn):
    by_label = {}
    for ex in pool:
        by_label.setdefault(label_fn(ex), []).append(ex)
    per = k // len(by_label)
    picks = [e for lst in by_label.values() for e in random.sample(lst, per)]
    random.shuffle(picks)
    return picks

Few-shotとファインチューニング

# Decision heuristic
if num_labeled < 500 or task_volatility == 'high':
    strategy = 'few-shot ICL'
elif latency_budget_ms < 200 or prompt_token_cost_dominant:
    strategy = 'fine-tune + zero-shot'
else:
    strategy = 'few-shot now, distill later'

推論タスクに必要なのはショット数だけではない

COT_SHOT = (
    'Q: A shop had 23 apples, used 20, bought 6 more. How many now?\n'
    'A: Start 23, minus 20 leaves 3, plus 6 is 9. Answer: 9\n\n'
    'Q: ' + question + '\nA:'
)

kの評価ハーネス

def harness(pool, val, ks, seeds=5):
    report = {}
    for k in ks:
        accs = []
        for s in range(seeds):
            demos = balanced_demos(pool, k, label_fn)
            accs.append(evaluate(build_prompt(demos), val))
        report[k] = (mean(accs), stdev(accs))
    return report

理解度チェック

まとめ