0Pricing
AI Agents · Lesson

Parsing HTML with BeautifulSoup

find(), select(), CSS selectors, and extracting structured content from HTML.

Parsing HTML with BeautifulSoup is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Parse HTML in Agents?

Many data sources are not APIs — they are web pages. When an agent needs to extract structured information from HTML, it must parse the raw markup into a navigable tree.

BeautifulSoup (bs4) is the standard Python library for this. It turns messy HTML into a Python object you can query with ease.

Creating a BeautifulSoup Object

Pass raw HTML and a parser name to BeautifulSoup(). The 'html.parser' is built into Python and requires no extra install. For faster parsing of large pages, 'lxml' is available via pip.

from bs4 import BeautifulSoup
import httpx

# Fetch HTML
response = httpx.get('https://example.com', timeout=10.0)
html = response.text

# Parse it
soup = BeautifulSoup(html, 'html.parser')

# Get the page title
print(soup.title.text)  # 'Example Domain'

find() — Find a Single Element

soup.find(tag, attrs) returns the first matching element, or None if not found. You can match by tag name, class, id, or any attribute.

Always check for None before calling methods on the result.

from bs4 import BeautifulSoup

html = '<div class="content"><h1>Title</h1><p>Body text here.</p></div>'
soup = BeautifulSoup(html, 'html.parser')

# Find by tag + class
content_div = soup.find('div', class_='content')
if content_div:
    heading = content_div.find('h1')
    print(heading.text)  # 'Title'

# Find by id
sidebar = soup.find('div', id='sidebar')  # None if absent

find_all() — Find Multiple Elements

soup.find_all(tag) returns a list of all matching elements. Iterate over it to extract data from repeated structures like list items, table rows, or article cards.

from bs4 import BeautifulSoup

html = '<ul><li>Apple</li><li>Banana</li><li>Cherry</li></ul>'
soup = BeautifulSoup(html, 'html.parser')

items = soup.find_all('li')
for item in items:
    print(item.text)  # Apple, Banana, Cherry

# Limit results
first_two = soup.find_all('li', limit=2)

CSS Selectors with select()

soup.select('css selector') lets you use familiar CSS syntax. This is often more concise than chained find() calls, especially for nested elements.

from bs4 import BeautifulSoup

html = '''
<table>
  <tr><td class="name">Alice</td><td class="score">95</td></tr>
  <tr><td class="name">Bob</td><td class="score">87</td></tr>
</table>
'''
soup = BeautifulSoup(html, 'html.parser')

# Select all td elements inside tr inside table
cells = soup.select('table tr td')
for cell in cells:
    print(cell.text)

# Select only name cells
names = soup.select('td.name')
for n in names:
    print(n.text)  # Alice, Bob

Extracting Text with .text and .strip()

.text (or .get_text()) returns all text content inside an element, including nested tags. Use .strip() to remove leading and trailing whitespace, which HTML often contains.

from bs4 import BeautifulSoup

html = '<p>  \n  Price: <strong>$29.99</strong>  \n  </p>'
soup = BeautifulSoup(html, 'html.parser')

paragraph = soup.find('p')

# .text includes nested tag content
print(paragraph.text)          # '  \n  Price: $29.99  \n  '
print(paragraph.text.strip())  # 'Price: $29.99'

# get_text with separator
print(paragraph.get_text(separator=' ', strip=True))  # 'Price: $29.99'

Extracting Attributes with .get()

Tag attributes like href, src, and data-* are accessed like a dictionary using .get('attr_name'). This safely returns None if the attribute is missing.

from bs4 import BeautifulSoup

html = '''
<a href="https://example.com/page" data-id="42">Click here</a>
<img src="/images/logo.png" alt="Logo">
'''
soup = BeautifulSoup(html, 'html.parser')

link = soup.find('a')
print(link.get('href'))     # 'https://example.com/page'
print(link.get('data-id'))  # '42'
print(link.get('class'))    # None (no class attribute)

img = soup.find('img')
print(img.get('src'))       # '/images/logo.png'

Navigating the Parse Tree

BeautifulSoup elements have parent/child/sibling relationships you can traverse. Use .parent, .children, .next_sibling, and .previous_sibling to navigate around a found element.

from bs4 import BeautifulSoup

html = '''
<div class="article">
  <h2>Headline</h2>
  <p>First paragraph.</p>
  <p>Second paragraph.</p>
</div>
'''
soup = BeautifulSoup(html, 'html.parser')

h2 = soup.find('h2')
print(h2.text)                    # 'Headline'
print(h2.parent['class'])         # ['article']
print(h2.next_sibling.next_sibling.text)  # 'First paragraph.'

Extracting All Links from a Page

A common agent task is collecting all links from a page for further crawling. Find all <a> tags and extract their href attributes, filtering out empty ones.

from bs4 import BeautifulSoup
import httpx

def extract_links(url: str) -> list:
    response = httpx.get(url, timeout=10.0)
    soup = BeautifulSoup(response.text, 'html.parser')

    links = []
    for tag in soup.find_all('a'):
        href = tag.get('href')
        if href and href.startswith('http'):
            links.append(href)
    return links

# urls = extract_links('https://news.ycombinator.com')
# print(urls[:5])

Handling Malformed HTML Gracefully

Real-world HTML is often broken: unclosed tags, mismatched elements, encoding issues. BeautifulSoup's parser is forgiving and will attempt to fix errors automatically. But always guard against None when accessing nested elements.

from bs4 import BeautifulSoup

# Broken HTML — missing closing tags
html = '<div><p>Hello <strong>World</div>'
soup = BeautifulSoup(html, 'html.parser')

# BS4 repairs the tree automatically
print(soup.prettify())
# <div><p>Hello <strong>World</strong></p></div>

# Safe chained access
price = soup.find('span', class_='price')
price_text = price.text.strip() if price else 'N/A'
print(price_text)  # 'N/A'

Complete Agent Scraper Tool

Putting it all together: a complete scraper function an agent can call as a tool. It fetches a page, parses it, and returns structured data in a format the agent can reason about.

from bs4 import BeautifulSoup
import httpx

def scrape_article(url: str) -> dict:
    response = httpx.get(url, headers={'User-Agent': 'MyAgent/1.0'}, timeout=10.0)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, 'html.parser')

    title = soup.find('h1')
    paragraphs = soup.find_all('p')

    return {
        'url': url,
        'title': title.text.strip() if title else '',
        'text': ' '.join(p.text.strip() for p in paragraphs[:5]),
        'links': [a.get('href') for a in soup.find_all('a', href=True)][:10]
    }

Knowledge Check: BeautifulSoup

Test your understanding of HTML parsing with BeautifulSoup.

Recap: Parsing HTML with BeautifulSoup

You can now extract structured data from HTML pages inside your agents:

  • Create a soup object with BeautifulSoup(html, 'html.parser')
  • Use find() for a single element, find_all() for multiple
  • Use select() for CSS selector queries
  • Get text with .text.strip() and attributes with .get('attr')
  • Navigate the tree with .parent, .children, and siblings

Combined with an HTTP client, BeautifulSoup gives your agent the ability to read any web page.

Frequently asked questions

Is the “Parsing HTML with BeautifulSoup” lesson free?

Yes — the full text of “Parsing HTML with BeautifulSoup” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Parsing HTML with BeautifulSoup”?

find(), select(), CSS selectors, and extracting structured content from HTML. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Parsing HTML with BeautifulSoup” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. HTTP Clients for Agents: httpx and requests
  2. Parsing HTML with BeautifulSoup
  3. Handling Pagination and Dynamic Content
  4. Respectful Scraping Practices
← Back to AI Agents