AI Agents · Lesson

Tavily and SerpAPI for Agent Search

API setup, query construction, and result parsing for web-search tools.

Tavily and SerpAPI for Agent Search is a free AI Agents lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Agents Need Web Search

LLMs have a knowledge cutoff date. For questions about current events, live prices, recent news, or fast-changing topics, the model's training data is stale.

Web search tools like Tavily and SerpApi give agents real-time access to the internet, bridging the gap between static training data and current reality.

Tavily: Purpose-Built for AI Agents

Tavily is a search API designed specifically for AI agents. Unlike general web scraping, it returns clean, structured results with pre-extracted content — no HTML parsing needed.

Install with pip install tavily-python. Get an API key at tavily.com.

from tavily import TavilyClient
import os

client = TavilyClient(api_key=os.getenv('TAVILY_API_KEY'))

results = client.search(
    query='latest OpenAI GPT-5 release date',
    max_results=5
)

for r in results['results']:
    print(r['title'])
    print(r['url'])
    print(r['content'][:200])
    print('---')

Tavily Search Parameters

Tavily offers several parameters to control search behavior. The most useful for agents are max_results, search_depth, and include_domains / exclude_domains.

# Basic search
results = client.search(
    query='Python async best practices 2024',
    max_results=5,
    search_depth='basic'  # 'basic' (fast) or 'advanced' (slower, deeper)
)

# Domain-filtered search
results = client.search(
    query='machine learning papers',
    max_results=5,
    include_domains=['arxiv.org', 'papers.nips.cc', 'openreview.net']
)

# Exclude low-quality domains
results = client.search(
    query='stock market today',
    max_results=5,
    exclude_domains=['pinterest.com', 'quora.com']
)

Tavily Result Structure

Each Tavily result contains: title, url, content (extracted text), score (relevance), and optionally published_date. The content field is clean text — no HTML tags.

results = client.search('Claude AI pricing 2024', max_results=3)

for r in results['results']:
    print(f"Title:   {r['title']}")
    print(f"URL:     {r['url']}")
    print(f"Score:   {r.get('score', 'N/A')}")
    print(f"Date:    {r.get('published_date', 'unknown')}")
    print(f"Content: {r['content'][:300]}")
    print()

# Tavily also returns 'answer' — a direct answer synthesized from results
if results.get('answer'):
    print('Direct answer:', results['answer'])

SerpApi: Google Search Programmatic Access

SerpApi provides structured access to Google, Bing, YouTube, and other search engines. It returns rich data including organic results, knowledge panels, featured snippets, and shopping results.

Install with pip install google-search-results.

from serpapi import GoogleSearch
import os

params = {
    'q': 'best Python web frameworks 2024',
    'api_key': os.getenv('SERPAPI_KEY'),
    'num': 5,              # number of results
    'hl': 'en',            # language
    'gl': 'us',            # country
    'safe': 'active'       # safe search
}

search = GoogleSearch(params)
results = search.get_dict()

for r in results.get('organic_results', []):
    print(r['title'])
    print(r['link'])
    print(r.get('snippet', ''))
    print()

SerpApi Result Structure

SerpApi returns richer structured data than Tavily. Key fields in organic_results: title, link, snippet, position. Featured snippets appear separately under answer_box.

def extract_serp_results(serp_data, max_results=5):
    results = []

    # Featured snippet / answer box (highest priority)
    if 'answer_box' in serp_data:
        box = serp_data['answer_box']
        results.append({
            'title': box.get('title', 'Featured Snippet'),
            'url':   box.get('link', ''),
            'snippet': box.get('answer') or box.get('snippet', ''),
            'is_featured': True
        })

    # Organic results
    for r in serp_data.get('organic_results', [])[:max_results]:
        results.append({
            'title':   r.get('title', ''),
            'url':     r.get('link', ''),
            'snippet': r.get('snippet', ''),
            'position': r.get('position', 0),
            'is_featured': False
        })

    return results

if __name__ == '__main__':
    demo_serp = {
        'answer_box': {'title': 'Python version', 'link': 'https://python.org', 'answer': '3.13'},
        'organic_results': [
            {'title': 'Python Docs', 'link': 'https://docs.python.org', 'snippet': 'Official docs', 'position': 1},
        ],
    }
    for r in extract_serp_results(demo_serp):
        print(f"{'[FEATURED] ' if r['is_featured'] else ''}{r['title']} - {r['url']}")

Freshness Options and Date Filtering

For time-sensitive queries, filter results by publication date. Tavily supports days parameter; SerpApi supports tbs (time-based search) parameter.

# Tavily: results from last 7 days
recent_results = client.search(
    query='AI news',
    max_results=5,
    days=7  # only results published in last 7 days
)

# SerpApi: results from last month
params = {
    'q': 'AI news',
    'api_key': os.getenv('SERPAPI_KEY'),
    'tbs': 'qdr:m',   # qdr:d=day, qdr:w=week, qdr:m=month, qdr:y=year
    'num': 5
}
search = GoogleSearch(params)
results = search.get_dict()

Building a Search Tool for LangChain

To use Tavily or SerpApi inside a LangChain agent, wrap the search function as a Tool. The agent can then call it like any other tool based on the description you provide.

from langchain.tools import Tool
from tavily import TavilyClient
import os

tavily = TavilyClient(api_key=os.getenv('TAVILY_API_KEY'))

def tavily_search(query: str) -> str:
    results = tavily.search(query=query, max_results=3, search_depth='basic')
    output = []
    for r in results['results']:
        output.append(f"Title: {r['title']}\nURL: {r['url']}\nContent: {r['content'][:500]}")
    return '\n\n'.join(output)

web_search_tool = Tool(
    name='web_search',
    func=tavily_search,
    description='Search the web for current information. Input: a search query string. Returns top results with title, URL, and content.'
)

Domain Filtering for Quality Control

Not all search results are reliable. Domain filtering lets you restrict searches to trusted sources (for research) or exclude spam-heavy domains (for general queries).

TRUSTED_DOMAINS = {
    'medical':  ['nih.gov', 'who.int', 'mayoclinic.org', 'pubmed.ncbi.nlm.nih.gov'],
    'legal':    ['law.cornell.edu', 'supreme.justia.com', 'congress.gov'],
    'tech':     ['docs.python.org', 'developer.mozilla.org', 'stackoverflow.com'],
    'finance':  ['sec.gov', 'federalreserve.gov', 'bloomberg.com']
}

SPAM_DOMAINS = ['pinterest.com', 'quora.com', 'answers.yahoo.com', 'wikihow.com']

def domain_aware_search(query, domain_category=None):
    kwargs = {'query': query, 'max_results': 5, 'exclude_domains': SPAM_DOMAINS}
    if domain_category and domain_category in TRUSTED_DOMAINS:
        kwargs['include_domains'] = TRUSTED_DOMAINS[domain_category]
    return client.search(**kwargs)

Handling Search Errors and Rate Limits

Both Tavily and SerpApi have rate limits. Implement retry logic with exponential backoff, and gracefully handle the case where search returns no results.

import time

def robust_search(query, max_retries=3):
    for attempt in range(max_retries):
        try:
            results = client.search(query=query, max_results=5)
            if results and results.get('results'):
                return results['results']
            print(f'No results for: {query}')
            return []
        except Exception as e:
            if '429' in str(e) or 'rate' in str(e).lower():
                wait = 2 ** attempt  # 1s, 2s, 4s
                print(f'Rate limited. Waiting {wait}s...')
                time.sleep(wait)
            else:
                print(f'Search error: {e}')
                return []
    return []

Snippet Length Normalization

Result snippets vary wildly in length: some are 50 characters, others are 2000 characters. Normalizing snippet length ensures each source gets equal representation in the LLM context.

MAX_SNIPPET_CHARS = 600
MIN_SNIPPET_CHARS = 100

def normalize_snippets(results):
    normalized = []
    for r in results:
        content = r.get('content') or r.get('snippet', '')
        # Truncate long snippets at sentence boundary
        if len(content) > MAX_SNIPPET_CHARS:
            # Find last sentence end before limit
            cutoff = content.rfind('. ', 0, MAX_SNIPPET_CHARS)
            content = content[:cutoff + 1] if cutoff > 0 else content[:MAX_SNIPPET_CHARS]
        # Skip very short snippets
        if len(content) < MIN_SNIPPET_CHARS:
            continue
        normalized.append({**r, 'content': content})
    return normalized

if __name__ == '__main__':
    demo_results = [
        {'content': 'Short.'},
        {'content': 'A' * 200 + '. ' + 'B' * 500},
    ]
    for r in normalize_snippets(demo_results):
        print(f"Length {len(r['content'])}: {r['content'][:60]}...")

Knowledge Check

What is the key advantage of Tavily over a general web scraping approach for AI agents?

Recap: Tavily and SerpApi for Agent Search

Web search APIs give agents access to current information beyond their training cutoff. Tavily is purpose-built for AI agents with clean content extraction; SerpApi provides richer structured Google data including featured snippets and knowledge panels.

Key practices: use domain filtering for quality control, apply freshness filters for time-sensitive queries, normalize snippet lengths, and implement retry logic with backoff for rate limit handling.

Frequently asked questions

Is the “Tavily and SerpAPI for Agent Search” lesson free?

Yes — the full text of “Tavily and SerpAPI for Agent Search” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Tavily and SerpAPI for Agent Search”?

API setup, query construction, and result parsing for web-search tools. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Tavily and SerpAPI for Agent Search” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Agents