AI Agents · Lesson

A Web-Browsing Research Agent

Combine a search tool, a fetch tool, and an LLM to research a topic and write a sourced summary.

A Web-Browsing Research Agent is a free AI Agents lesson on CoddyKit. This is lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, and your progress syncs across the web and the CoddyKit app. The AI Agents course includes 4 lessons in total.

Project Goal

Build an agent that researches a topic by searching the web, fetching pages, and writing a sourced summary — like a tiny version of Perplexity.

Tools You Will Need

search — query a search API (Tavily, Serper, Brave) and return results
fetch_url — download a page and return cleaned text
answer — the agent writes the final summary with citations

Step 1: Search Tool

import requests

def search(query: str, k: int = 5) -> list[dict]:
    r = requests.post(
        'https://api.tavily.com/search',
        json={'api_key': TAVILY_KEY, 'query': query, 'max_results': k}
    )
    return r.json()['results']   # list of {url, title, snippet}

Step 2: Fetch Tool

from bs4 import BeautifulSoup

def fetch_url(url: str) -> str:
    r = requests.get(url, timeout=10)
    soup = BeautifulSoup(r.text, 'html.parser')
    for tag in soup(['script', 'style', 'nav', 'footer']):
        tag.decompose()
    return soup.get_text(separator='\n', strip=True)[:5000]   # truncate

Step 3: Tool Definitions

tools = [
    {'type': 'function', 'function': {
        'name': 'search',
        'description': 'Search the web for a query, return top results with URLs',
        'parameters': {'type': 'object', 'properties': {'query': {'type': 'string'}}, 'required': ['query']}
    }},
    {'type': 'function', 'function': {
        'name': 'fetch_url',
        'description': 'Download the text of a URL',
        'parameters': {'type': 'object', 'properties': {'url': {'type': 'string'}}, 'required': ['url']}
    }}
]
import json
print(json.dumps(tools, indent=2))

Step 4: The Agent Loop

def research(question, max_steps=8):
    messages = [
        {'role': 'system', 'content': 'You are a research agent. Use search and fetch_url to gather facts. Cite each claim with a URL.'},
        {'role': 'user', 'content': question}
    ]
    for _ in range(max_steps):
        r = oai.chat.completions.create(model='gpt-4o-mini', messages=messages, tools=tools)
        msg = r.choices[0].message
        messages.append(msg)
        if not msg.tool_calls:
            return msg.content
        for tc in msg.tool_calls:
            result = dispatch(tc)
            messages.append({'role': 'tool', 'tool_call_id': tc.id, 'content': json.dumps(result)})
    return 'Step limit reached'

Tool Dispatcher

def dispatch(tc):
    args = json.loads(tc.function.arguments)
    if tc.function.name == 'search':
        return search(**args)
    if tc.function.name == 'fetch_url':
        return fetch_url(**args)
    return {'error': 'unknown tool'}

Add a Step Cap

Without a cap, the agent can spin forever. max_steps=8 is a sane default.

Truncate Fetched Pages

Most pages are 50KB+ — way too big. Truncate to 5000 chars (~1300 tokens). The model rarely needs more.

Respect robots.txt

Production browsing must respect robots.txt and the site's Terms of Service. Use a managed search/fetch service when in doubt.

Cache Results

The same query and URL will be requested often. Cache results in Redis for an hour:

@lru_cache(maxsize=1000)
def cached_fetch(url):
    return fetch_url(url)

Handle Failures

Network errors are routine. Return the error to the model so it can try another source:

try:
    return fetch_url(url)
except Exception as e:
    return {'error': f'Could not fetch {url}: {e}'}

Force Citations

End the system prompt with: "Your final answer must include URLs for every claim, like [1] https://..." — and consider an output schema check.

Watch the Cost

Research agents are expensive — 5-15 tool calls per question, each fetching kilobytes. Set a per-query budget cap.

Step Cap

Why set a max_steps limit on the agent loop?

Recap

Search + fetch + LLM + a step cap = a working research agent in 50 lines. Excellent project 3.

Frequently Asked Questions

Is the “A Web-Browsing Research Agent” lesson free?

Yes — the full text of “A Web-Browsing Research Agent” is free to read here on the web. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO. The AI Agents course includes 4 lessons in total.

What will I learn in “A Web-Browsing Research Agent”?

Combine a search tool, a fetch tool, and an LLM to research a topic and write a sourced summary. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners, so you can start here or from the beginning and move at your own pace. This is lesson 3 of 4.

How long does the “A Web-Browsing Research Agent” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Agents