0Pricing
Python Academy · Lesson

Parsing HTML with BeautifulSoup

Navigate the parse tree and extract data with selectors.

Parsing HTML with BeautifulSoup is a free Python Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Python Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Installing BeautifulSoup

beautifulsoup4 parses HTML and XML. Use the lxml parser for speed or html.parser for zero-dependency use.

# pip install beautifulsoup4 lxml
from bs4 import BeautifulSoup

html = "<h1>Hello</h1><p class='intro'>World</p>"
soup = BeautifulSoup(html, "lxml")
print(soup.h1.text)   # Hello

Finding Elements

find(tag) returns the first matching element. find_all(tag) returns a list of all matching elements.

from bs4 import BeautifulSoup

html = "<ul><li>A</li><li>B</li><li>C</li></ul>"
soup = BeautifulSoup(html, "lxml")
print(soup.find("li").text)       # A
print([li.text for li in soup.find_all("li")])  # [A,B,C]

CSS Selectors

soup.select("css selector") uses CSS syntax. soup.select_one() returns the first match.

from bs4 import BeautifulSoup

html = '<div class="card"><a href="/p/1">Link 1</a></div>'
soup = BeautifulSoup(html, "lxml")
print(soup.select("div.card a")[0]["href"])   # /p/1
print(soup.select_one("a")["href"])          # /p/1

Accessing Attributes

Access tag attributes like a dictionary: tag["href"], tag.get("class", []).

from bs4 import BeautifulSoup

html = '<a href="https://example.com" class="link external">Click</a>'
soup = BeautifulSoup(html, "lxml")
a = soup.find("a")
print(a["href"])           # https://example.com
print(a.get("class", [])) # ['link', 'external']

Navigating the Parse Tree

Use .parent, .children, .next_sibling, .previous_sibling to traverse the tree.

from bs4 import BeautifulSoup

html = "<div><h2>Title</h2><p>Para</p></div>"
soup = BeautifulSoup(html, "lxml")
h2 = soup.find("h2")
print(h2.parent.name)         # div
print(h2.next_sibling.text)   # Para (next sibling)

Extracting Text

tag.get_text(separator=" ", strip=True) returns all text within a tag, with whitespace cleaned up.

from bs4 import BeautifulSoup

html = "<div>  <p>Hello</p>  <p> World </p>  </div>"
soup = BeautifulSoup(html, "lxml")
print(soup.div.get_text(separator=" ", strip=True))
# Hello World

Scraping with requests + BeautifulSoup

Fetch a page with requests and parse it with BeautifulSoup — the classic scraping pair.

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com"
r = requests.get(url, timeout=10)
soup = BeautifulSoup(r.text, "lxml")

titles = [h3.find("a")["title"] for h3 in soup.select("article.product_pod h3")]
print(titles[:3])

Handling Relative URLs

Use urllib.parse.urljoin(base, href) to convert relative links to absolute URLs.

from urllib.parse import urljoin
base = "https://example.com/products/"
href = "../about"
print(urljoin(base, href))   # https://example.com/about

Handling Encodings

BeautifulSoup auto-detects encoding, but you can specify it or use r.encoding from requests.

import requests
from bs4 import BeautifulSoup

r = requests.get("https://example.com")
print(r.encoding)   # e.g. utf-8
soup = BeautifulSoup(r.content, "lxml", from_encoding=r.encoding)

Modifying and Serialising

Modify the parse tree and convert it back to HTML with str(tag) or soup.prettify().

from bs4 import BeautifulSoup

html = "<p>Hello <b>world</b></p>"
soup = BeautifulSoup(html, "lxml")
soup.b.decompose()   # remove the <b> tag
print(str(soup.p))   # <p>Hello </p>

Limits of BeautifulSoup

BS4 parses static HTML. It cannot execute JavaScript. For JS-rendered pages use Playwright or Selenium.

# BeautifulSoup: good for static HTML
# Playwright: for JavaScript-rendered SPAs
# Scrapy: for large-scale crawls with pipelines

# pip install playwright
# playwright install
# from playwright.async_api import async_playwright

Quick Check

What method returns all elements matching a CSS selector in BeautifulSoup?

Recap

BeautifulSoup parses HTML into a navigable tree. Use find()/find_all() for tag search, select() for CSS selectors, and get_text() for clean text. Pair with requests for static pages; use Playwright for JS-rendered content.

Frequently asked questions

Is the “Parsing HTML with BeautifulSoup” lesson free?

Yes — the full text of “Parsing HTML with BeautifulSoup” is free to read here on the web, and the Python Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Python Academy course, upgrade to CoddyKit PRO.

What will I learn in “Parsing HTML with BeautifulSoup”?

Navigate the parse tree and extract data with selectors. You practise Python Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Python Academy?

No prior experience is required. Python Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Parsing HTML with BeautifulSoup” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Python Academy lesson?

Yes. Every Python Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. HTTP Requests with requests and httpx
  2. Parsing HTML with BeautifulSoup
  3. Building a Scrapy Spider
  4. Handling JavaScript and Anti-scraping Measures
← Back to Python Academy