AI Agents · Lesson

A Web-Browsing Research Agent

Combine a search tool, a fetch tool, and an LLM to research a topic and write a sourced summary.

A Web-Browsing Research Agent is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Project Goal

Build an agent that researches a topic by searching the web, fetching pages, and writing a sourced summary — like a tiny version of Perplexity.

Tools You Will Need

search — query a search API (Tavily, Serper, Brave) and return results
fetch_url — download a page and return cleaned text
answer — the agent writes the final summary with citations

Step 1: Search Tool

import requests

def search(query: str, k: int = 5) -> list[dict]:
    r = requests.post(
        'https://api.tavily.com/search',
        json={'api_key': TAVILY_KEY, 'query': query, 'max_results': k}
    )
    return r.json()['results']   # list of {url, title, snippet}

Step 2: Fetch Tool

from bs4 import BeautifulSoup

def fetch_url(url: str) -> str:
    r = requests.get(url, timeout=10)
    soup = BeautifulSoup(r.text, 'html.parser')
    for tag in soup(['script', 'style', 'nav', 'footer']):
        tag.decompose()
    return soup.get_text(separator='\n', strip=True)[:5000]   # truncate

Step 3: Tool Definitions

tools = [
    {'type': 'function', 'function': {
        'name': 'search',
        'description': 'Search the web for a query, return top results with URLs',
        'parameters': {'type': 'object', 'properties': {'query': {'type': 'string'}}, 'required': ['query']}
    }},
    {'type': 'function', 'function': {
        'name': 'fetch_url',
        'description': 'Download the text of a URL',
        'parameters': {'type': 'object', 'properties': {'url': {'type': 'string'}}, 'required': ['url']}
    }}
]
import json
print(json.dumps(tools, indent=2))

Step 4: The Agent Loop

def research(question, max_steps=8):
    messages = [
        {'role': 'system', 'content': 'You are a research agent. Use search and fetch_url to gather facts. Cite each claim with a URL.'},
        {'role': 'user', 'content': question}
    ]
    for _ in range(max_steps):
        r = oai.chat.completions.create(model='gpt-4o-mini', messages=messages, tools=tools)
        msg = r.choices[0].message
        messages.append(msg)
        if not msg.tool_calls:
            return msg.content
        for tc in msg.tool_calls:
            result = dispatch(tc)
            messages.append({'role': 'tool', 'tool_call_id': tc.id, 'content': json.dumps(result)})
    return 'Step limit reached'

Tool Dispatcher

def dispatch(tc):
    args = json.loads(tc.function.arguments)
    if tc.function.name == 'search':
        return search(**args)
    if tc.function.name == 'fetch_url':
        return fetch_url(**args)
    return {'error': 'unknown tool'}

Add a Step Cap

Without a cap, the agent can spin forever. max_steps=8 is a sane default.

Truncate Fetched Pages

Most pages are 50KB+ — way too big. Truncate to 5000 chars (~1300 tokens). The model rarely needs more.

Respect robots.txt

Production browsing must respect robots.txt and the site's Terms of Service. Use a managed search/fetch service when in doubt.

Cache Results

The same query and URL will be requested often. Cache results in Redis for an hour:

@lru_cache(maxsize=1000)
def cached_fetch(url):
    return fetch_url(url)

Handle Failures

Network errors are routine. Return the error to the model so it can try another source:

try:
    return fetch_url(url)
except Exception as e:
    return {'error': f'Could not fetch {url}: {e}'}

Force Citations

End the system prompt with: "Your final answer must include URLs for every claim, like [1] https://..." — and consider an output schema check.

Watch the Cost

Research agents are expensive — 5-15 tool calls per question, each fetching kilobytes. Set a per-query budget cap.

Step Cap

Why set a max_steps limit on the agent loop?

Recap

Search + fetch + LLM + a step cap = a working research agent in 50 lines. Excellent project 3.

Frequently asked questions

Is the “A Web-Browsing Research Agent” lesson free?

Yes — the full text of “A Web-Browsing Research Agent” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “A Web-Browsing Research Agent”?

Combine a search tool, a fetch tool, and an LLM to research a topic and write a sourced summary. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “A Web-Browsing Research Agent” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Agents