0Pricing
AI Agents · Lesson

HTTP Clients for Agents: httpx and requests

Synchronous and async HTTP requests, session management, headers.

HTTP Clients for Agents: httpx and requests is a free AI Agents lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why HTTP Clients Matter for Agents

AI agents frequently need to fetch data from external sources: APIs, websites, and services. A reliable HTTP client is a core tool in any agent's toolkit.

Python has two popular HTTP libraries: requests (synchronous, simple) and httpx (supports both sync and async). Understanding when to use each is essential for building efficient agents.

Basic GET Request with requests

The requests library makes simple HTTP calls easy. Use requests.get(url) to fetch a resource and inspect the response.

Always check the status code before using the response body to avoid silent failures.

import requests

url = 'https://api.example.com/data'
response = requests.get(url)

print(response.status_code)  # 200
print(response.text)         # raw string body
print(response.json())       # parsed JSON dict

Adding Headers and Timeouts

Most APIs require authentication headers. The headers= parameter lets you pass a dictionary of headers. Always set a timeout= to prevent your agent from hanging indefinitely on a slow server.

import requests

headers = {
    'Authorization': 'Bearer YOUR_API_KEY',
    'Accept': 'application/json'
}

response = requests.get(
    'https://api.example.com/items',
    headers=headers,
    timeout=10  # seconds
)

data = response.json()
print(data)

raise_for_status() — Fail Fast on Errors

response.raise_for_status() raises an HTTPError for 4xx and 5xx status codes. Without it, a 404 or 500 response is silently treated as success.

This is a best practice in agents: fail loudly so the agent knows to retry or report an error.

import requests

try:
    response = requests.get('https://api.example.com/missing', timeout=10)
    response.raise_for_status()  # raises if status >= 400
    data = response.json()
except requests.HTTPError as e:
    print(f'HTTP error: {e}')
except requests.RequestException as e:
    print(f'Network error: {e}')

Introduction to httpx

httpx is a modern HTTP client with the same API as requests but with added async support. It also enforces timeouts by default, making it safer for production agents.

Install it with pip install httpx. Synchronous usage looks almost identical to requests.

import httpx

response = httpx.get(
    'https://api.example.com/data',
    headers={'Authorization': 'Bearer YOUR_KEY'},
    timeout=10.0
)

response.raise_for_status()
data = response.json()
print(data)

Async HTTP with httpx.AsyncClient

When your agent runs in an async context (e.g., with FastAPI or asyncio), use httpx.AsyncClient to avoid blocking the event loop. Wrap it in async with to ensure the connection is properly closed.

import httpx
import asyncio

async def fetch_data(url: str) -> dict:
    async with httpx.AsyncClient(timeout=10.0) as client:
        response = await client.get(
            url,
            headers={'Authorization': 'Bearer YOUR_KEY'}
        )
        response.raise_for_status()
        return response.json()

# result = asyncio.run(fetch_data('https://api.example.com/data'))

Making Multiple Async Requests Concurrently

One major advantage of async HTTP is fetching multiple URLs at once with asyncio.gather(). This can dramatically speed up agents that need data from several endpoints before responding.

import httpx
import asyncio

async def fetch_all(urls: list) -> list:
    async with httpx.AsyncClient(timeout=10.0) as client:
        tasks = [client.get(url) for url in urls]
        responses = await asyncio.gather(*tasks)
        return [r.json() for r in responses]

urls = [
    'https://api.example.com/item/1',
    'https://api.example.com/item/2',
    'https://api.example.com/item/3'
]
# results = asyncio.run(fetch_all(urls))

Session Reuse and Connection Pooling

Creating a new HTTP connection for every request is slow. Both libraries support connection pooling: requests.Session and httpx.Client reuse TCP connections and share headers/cookies across requests.

This is especially useful in agents that make many calls to the same API.

import httpx

# Create once, reuse for many requests
client = httpx.Client(
    base_url='https://api.example.com',
    headers={'Authorization': 'Bearer YOUR_KEY'},
    timeout=10.0
)

response1 = client.get('/users')
response2 = client.get('/items')
response3 = client.get('/orders')

client.close()  # always close when done

Sending POST Requests with JSON Body

Agents often need to send data, not just read it. Use the json= parameter to automatically serialize a Python dict and set the correct Content-Type header.

import httpx

payload = {
    'query': 'latest AI news',
    'max_results': 5,
    'language': 'en'
}

response = httpx.post(
    'https://api.example.com/search',
    json=payload,
    headers={'Authorization': 'Bearer YOUR_KEY'},
    timeout=15.0
)

response.raise_for_status()
results = response.json()
print(results['items'])

Parsing the Response: text, json, and content

The response object has three main body properties:

  • .text — decoded string (HTML, XML, plain text)
  • .json() — parses JSON into a Python dict/list
  • .content — raw bytes (for images or binary files)

Use the appropriate one based on the API's content type.

import httpx

response = httpx.get('https://api.example.com/report', timeout=10.0)

# For JSON APIs
data = response.json()            # dict or list

# For HTML or plain text
html = response.text              # str

# For binary files
image_bytes = response.content    # bytes

print(type(data), type(html), type(image_bytes))

Putting It Together: An Agent HTTP Fetch Tool

Here is a complete, reusable fetch function that an agent can call as a tool. It handles errors gracefully, logs the request, and returns structured data.

This pattern is a solid foundation for any web-fetching agent tool.

import httpx
import logging

logger = logging.getLogger(__name__)

def agent_fetch(url: str, headers: dict = None) -> dict:
    try:
        response = httpx.get(
            url,
            headers=headers or {},
            timeout=10.0
        )
        response.raise_for_status()
        logger.info(f'Fetched {url} -> {response.status_code}')
        return {'success': True, 'data': response.json()}
    except httpx.HTTPStatusError as e:
        return {'success': False, 'error': str(e)}
    except httpx.RequestError as e:
        return {'success': False, 'error': f'Network error: {e}'}

Knowledge Check: HTTP Clients

Test your understanding of HTTP clients for agents.

Recap: HTTP Clients for Agents

In this lesson you learned how to equip agents with reliable HTTP fetching capabilities:

  • Use requests for simple synchronous fetching
  • Use httpx.AsyncClient for non-blocking async requests
  • Always set timeout= and call raise_for_status()
  • Reuse sessions/clients for multiple calls to the same host
  • Use .json(), .text, or .content based on the response type

A well-written fetch tool is the gateway between your agent and the web.

Frequently asked questions

Is the “HTTP Clients for Agents: httpx and requests” lesson free?

Yes — the full text of “HTTP Clients for Agents: httpx and requests” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “HTTP Clients for Agents: httpx and requests”?

Synchronous and async HTTP requests, session management, headers. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “HTTP Clients for Agents: httpx and requests” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. HTTP Clients for Agents: httpx and requests
  2. Parsing HTML with BeautifulSoup
  3. Handling Pagination and Dynamic Content
  4. Respectful Scraping Practices
← Back to AI Agents