AI Agents · Lesson

Source Verification and Citation

Cross-referencing claims across sources and attaching source URLs to facts.

Source Verification and Citation is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Source Verification Matters

An agent can retrieve a false claim from a low-quality source and present it as fact. Without verification, the research agent is just a sophisticated search engine with no quality filter.

Verification adds a layer that checks whether claims are corroborated by multiple independent sources.

The Two-Source Rule

A claim is considered verified when it appears in at least two independent sources. 'Independent' means different domains — finding the same claim on two sites that both cite the same original article does not count.

from urllib.parse import urlparse

def extract_domain(url: str) -> str:
    return urlparse(url).netloc.replace('www.', '')

def is_verified(fact: str, all_facts: list[dict]) -> bool:
    matching_domains = set()
    for f in all_facts:
        if fact.lower() in f['fact'].lower() or f['fact'].lower() in fact.lower():
            matching_domains.add(extract_domain(f['source']))
    return len(matching_domains) >= 2

# Example
facts = [
    {'fact': 'Inflation peaked at 9.1% in June 2022', 'source': 'https://bls.gov/cpi'},
    {'fact': 'US inflation hit 9.1% peak in mid-2022',  'source': 'https://reuters.com/article/a'}
]
print(is_verified('inflation peaked at 9.1%', facts))  # True

LLM-Based Semantic Matching

String matching misses paraphrased claims. Use an LLM to judge whether two facts are semantically equivalent before counting them as corroborating sources.

import openai, json

client = openai.OpenAI(api_key='YOUR_OPENAI_KEY')

def claims_are_equivalent(claim_a: str, claim_b: str) -> bool:
    prompt = (
        f'Are these two statements making the same factual claim?\n'
        f'A: "{claim_a}"\n'
        f'B: "{claim_b}"\n'
        f'Answer JSON: {{"equivalent": true/false}}'
    )
    resp = client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[{'role': 'user', 'content': prompt}],
        response_format={'type': 'json_object'}
    )
    return json.loads(resp.choices[0].message.content).get('equivalent', False)

print(claims_are_equivalent(
    'CPI hit 9.1% in June 2022',
    'US consumer prices rose 9.1 percent year-over-year in June 2022'
))  # True

Source Quality Scoring

Not all sources are equal. Score each source by domain type, publication date, and known reliability. Tier 1 sources (government stats, peer-reviewed journals) should carry more weight than blogs or social media.

from datetime import datetime, timezone

HIGH_AUTHORITY_DOMAINS = [
    'gov', 'edu', 'nature.com', 'science.org', 'pubmed.ncbi.nlm.nih.gov',
    'reuters.com', 'bbc.com', 'apnews.com', 'who.int'
]

def score_source(url: str, publication_date: str | None) -> float:
    score = 0.5   # baseline
    domain = extract_domain(url)
    tld = domain.split('.')[-1]

    # Domain authority
    if any(ha in domain for ha in HIGH_AUTHORITY_DOMAINS):
        score += 0.3
    elif tld in ('gov', 'edu', 'org'):
        score += 0.1

    # Recency
    if publication_date:
        try:
            pub = datetime.fromisoformat(publication_date)
            days_old = (datetime.now(timezone.utc) - pub.replace(tzinfo=timezone.utc)).days
            if days_old < 365:
                score += 0.1
            elif days_old > 1825:  # 5 years
                score -= 0.1
        except ValueError:
            pass

    return min(1.0, max(0.0, score))

Cross-Referencing Claims Against Sources

For each claim the agent intends to include in the final report, cross-reference it against all retrieved documents to find the best corroborating sources.

def cross_reference(claim: str, all_facts: list[dict]) -> list[dict]:
    supporting = []
    seen_domains = set()

    for fact in all_facts:
        if claims_are_equivalent(claim, fact['fact']):
            domain = extract_domain(fact['source'])
            if domain not in seen_domains:  # only one per domain
                seen_domains.add(domain)
                supporting.append({
                    'source': fact['source'],
                    'domain': domain,
                    'fact':   fact['fact'],
                    'score':  score_source(fact['source'], fact.get('publication_date'))
                })

    return sorted(supporting, key=lambda x: x['score'], reverse=True)

Author Credentials Check

For scientific or medical claims, author credentials matter. A study with named authors at accredited institutions outweighs an anonymous blog post. Extract author information and flag claims from anonymous sources.

def assess_authorship(url: str, page_text: str) -> dict:
    prompt = (
        f'From this webpage text, extract:\n'
        f'1. Author name(s) (or null)\n'
        f'2. Author affiliation/credentials (or null)\n'
        f'3. Publication name (or null)\n'
        f'Return JSON: {{"authors": [], "affiliation": null, "publication": null}}\n\n'
        f'TEXT:\n{page_text[:2000]}'
    )
    resp = client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[{'role': 'user', 'content': prompt}],
        response_format={'type': 'json_object'}
    )
    return json.loads(resp.choices[0].message.content)

Flagging Unverified Claims

Claims that cannot be cross-referenced from two independent sources should be flagged as unverified in the output. Depending on the report's risk level, unverified claims may be excluded or presented with a disclaimer.

def classify_claims(claims: list[str], all_facts: list[dict]) -> list[dict]:
    classified = []
    for claim in claims:
        support = cross_reference(claim, all_facts)
        classified.append({
            'claim':       claim,
            'verified':    len(support) >= 2,
            'sources':     support[:3],  # top 3 supporting sources
            'confidence':  'high' if len(support) >= 3 else
                           'medium' if len(support) == 2 else
                           'low'
        })
    return classified

Citation Format: Inline and Reference List

Reports should include inline citations (numbered superscripts) and a reference list at the end. Generate both from the verified source metadata.

def format_citations(classified_claims: list[dict]) -> tuple[list[str], list[str]]:
    ref_list = []
    ref_map = {}   # url -> citation number
    inline_claims = []

    for item in classified_claims:
        nums = []
        for src in item['sources']:
            url = src['source']
            if url not in ref_map:
                ref_map[url] = len(ref_list) + 1
                ref_list.append(f'[{ref_map[url]}] {url}')
            nums.append(f'[{ref_map[url]}]')

        citation_str = ''.join(nums)
        prefix = '' if item['verified'] else '[UNVERIFIED] '
        inline_claims.append(f'{prefix}{item["claim"]} {citation_str}')

    return inline_claims, ref_list

if __name__ == '__main__':
    demo_claims = [
        {'claim': 'Revenue grew 12% year over year', 'verified': True, 'sources': [{'source': 'https://example.com/report'}]},
        {'claim': 'The CEO plans to step down', 'verified': False, 'sources': [{'source': 'https://example.com/rumor'}]},
    ]
    inline, refs = format_citations(demo_claims)
    for line in inline:
        print(line)
    print('References:')
    for r in refs:
        print(' ', r)

Detecting Contradictions Between Sources

Sometimes two credible sources contradict each other. Detect contradictions and present both sides rather than silently picking one. This is especially important for contested empirical claims.

def find_contradictions(claim: str, all_facts: list[dict]) -> list[dict]:
    contradictions = []
    for fact in all_facts:
        if extract_domain(fact['source']) == extract_domain(claim):
            continue
        prompt = (
            f'Does fact B contradict fact A?\n'
            f'A: "{claim}"\nB: "{fact["fact"]}"\n'
            f'JSON: {{"contradicts": true/false}}'
        )
        resp = client.chat.completions.create(
            model='gpt-4o-mini',
            messages=[{'role': 'user', 'content': prompt}],
            response_format={'type': 'json_object'}
        )
        result = json.loads(resp.choices[0].message.content)
        if result.get('contradicts'):
            contradictions.append(fact)
    return contradictions

Verification Report Card

Before generating the final report, produce a verification summary: how many claims verified, how many unverified, top sources by authority score, any contradictions detected.

def verification_report_card(classified: list[dict]) -> dict:
    total     = len(classified)
    verified  = sum(1 for c in classified if c['verified'])
    all_sources = [
        src for c in classified for src in c['sources']
    ]
    top_domains = sorted(
        set(s['domain'] for s in all_sources),
        key=lambda d: max(s['score'] for s in all_sources if s['domain'] == d),
        reverse=True
    )[:5]
    return {
        'total_claims':    total,
        'verified':        verified,
        'unverified':      total - verified,
        'verification_rate': f'{verified/max(total,1)*100:.0f}%',
        'top_sources':     top_domains
    }

if __name__ == '__main__':
    demo_classified = [
        {'verified': True, 'sources': [{'domain': 'docs.python.org', 'score': 1.3}]},
        {'verified': False, 'sources': [{'domain': 'quora.com', 'score': 0.5}]},
    ]
    card = verification_report_card(demo_classified)
    for k, v in card.items():
        print(f'{k}: {v}')

Confidence-Gated Output

Set a minimum verification threshold before publishing. If fewer than 70% of claims are verified, prompt the research loop to run another iteration targeting the unverified claims specifically.

MIN_VERIFICATION_RATE = 0.70

def should_run_more_research(classified: list[dict]) -> list[str]:
    unverified = [c for c in classified if not c['verified']]
    total = len(classified)
    if total == 0:
        return []
    rate = (total - len(unverified)) / total
    if rate >= MIN_VERIFICATION_RATE:
        return []  # Sufficient confidence — stop
    # Return search queries to verify remaining claims
    return [f'verify: {c["claim"]}' for c in unverified[:3]]

if __name__ == '__main__':
    demo_classified = [
        {'claim': 'A', 'verified': True},
        {'claim': 'B', 'verified': False},
        {'claim': 'C', 'verified': False},
    ]
    followups = should_run_more_research(demo_classified)
    print('Follow-up queries needed:', followups)

What defines two 'independent' sources for the two-source verification rule?

The independence criterion prevents echo-chamber verification where the same claim is traced back to a single original source cited by many outlets.

Source Verification Recap

Verified research requires: two-source corroboration across different domains, LLM semantic matching to detect paraphrased equivalents, source quality scoring (domain authority + recency), contradiction detection, and confidence-gated output that triggers more research if the verification rate is too low.

Frequently asked questions

Is the “Source Verification and Citation” lesson free?

Yes — the full text of “Source Verification and Citation” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Source Verification and Citation”?

Cross-referencing claims across sources and attaching source URLs to facts. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Source Verification and Citation” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Multi-Step Research Loop Design
Source Verification and Citation
Structured Report Generation
Fact-Checking and Hallucination Prevention

← Back to AI Agents