Web Scraping & Bots · Ders

Karmaşık HTML Yapılarında Gezinme

Derin biçimde iç içe geçmiş veya düzensiz yapılandırılmış HTML belgelerinde etkili şekilde gezinme tekniklerini öğrenin.

1. ders / 411 adım

Karmaşık HTML Yapılarında Gezinme, CoddyKit'te ücretsiz bir Web Scraping & Bots dersidir. Bu, 4 dersinin 1. dersidir. Aşağıdan dersin tamamını ücretsiz okuyabilir, sonra tarayıcıda yerleşik kod editörü ve 7/24 yapay zeka koçu ile uygulamalı olarak pratik yapabilirsin. Bu, Web Scraping & Bots öğrenme yolunun bir parçasıdır ve ilerlemeniz web ve CoddyKit uygulaması arasında senkronize olur. Web Scraping & Bots kursu toplamda 4 dersten oluşur.

Bu dersin bazı bölümleri henüz çevrilmemiş olup İngilizce olarak gösterilmektedir.

Beyond Simple Extraction

What happens when the data you need isn't neatly tucked into an element with a unique ID or class? Sometimes, web pages have complex layouts that require a more sophisticated approach.

In this lesson, we'll learn how to "walk" through the HTML structure, finding elements based on their relationships to others. This technique is called HTML tree traversal.

The HTML Tree Structure

Think of an HTML document like a family tree. Every element (like a <div>, <p>, or <h1>) is a "node."

  • Parent: An element that contains other elements.
  • Child: An element directly inside another element.
  • Sibling: Elements at the same level, sharing the same parent.
  • Descendant: Any element inside a parent, directly or indirectly.

Understanding these relationships is key to navigating complex pages.

Children: Direct Descendants

To get the direct children of an element, BeautifulSoup offers a couple of ways. The .contents property returns a list of all children, including NavigableStrings (text nodes).

The .children property returns an iterator, which is often more memory-efficient for large documents. Let's see an example.

from bs4 import BeautifulSoup

html_doc = """
<div class="parent">
  <p>Child 1</p>
  <span>Child 2</span>
  Text node
</div>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
parent_div = soup.find('div', class_='parent')

print("Using .contents:")
for child in parent_div.contents:
  print(f"- {child.name or 'Text'}: {repr(child)[:20]}...")

print("\nUsing .children:")
for child in parent_div.children:
  print(f"- {child.name or 'Text'}: {repr(child)[:20]}...")

Moving Up: Parents

Sometimes you find an element, but need information from its containing element. BeautifulSoup allows you to easily move "up" the tree.

  • The .parent property gives you the direct parent of an element.
  • The .parents property gives you an iterator for all ancestor elements, all the way up to the document root.
from bs4 import BeautifulSoup

html_doc = """
<div class="grandparent">
  <div class="parent">
    <p class="child">Hello</p>
  </div>
</div>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
child_p = soup.find('p', class_='child')

print(f"Child: {child_p.name}")
print(f"Direct Parent: {child_p.parent.name}")

print("All Ancestors:")
for p in child_p.parents:
  if p.name: # Filter out [document] and other non-tag elements
    print(f"- {p.name}")

Side-by-Side: Single Siblings

Elements at the same level are called siblings. You can move between them using .next_sibling and .previous_sibling.

Be aware that these properties will also include "NavigableString" objects if there's whitespace or text directly between your tags. You might need to check if the result is a tag.

from bs4 import BeautifulSoup

html_doc = """
<div class="items">
  <p>Item 1</p>
  <span>Item 2</span>
  <p>Item 3</p>
</div>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
item_one = soup.find('p', string='Item 1')

# The next sibling after <p>Item 1</p> is usually a newline character (NavigableString)
# We need to find the *next tag* sibling
next_tag = item_one.next_sibling
while next_tag and not next_tag.name:
    next_tag = next_tag.next_sibling

print(f"Item 1: {item_one.text}")
if next_tag:
  print(f"Next tag sibling: {next_tag.text}")
else:
  print("No next tag sibling found.")

All Siblings: next_siblings

To get all the siblings that come after or before a particular tag, you can use the .next_siblings and .previous_siblings iterators.

This is extremely useful when you've found one element and need to extract data from all related elements at the same level.

from bs4 import BeautifulSoup

html_doc = """
<div class="menu">
  <a href="#home">Home</a>
  <a href="#about">About</a>
  <a href="#contact">Contact</a>
</div>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
home_link = soup.find('a', href='#home')

print("All next siblings after Home:")
for sibling in home_link.next_siblings:
  if sibling.name == 'a': # Only interested in <a> tags
    print(f"- {sibling.text}: {sibling['href']}")

Beyond Siblings: find_next

Sometimes, the data you need isn't a direct child or sibling, but appears somewhere later in the HTML document relative to your current element. This is where .find_next() and .find_all_next() come in.

These methods search the rest of the document after the current tag, regardless of parent-child-sibling relationships. Similarly, .find_previous() and .find_all_previous() search before.

from bs4 import BeautifulSoup

html_doc = """
<div class="header">
  <h2>Section Title</h2>
</div>
<p>Some introductory text.</p>
<div class="content">
  <p>First paragraph.</p>
  <span>Important data!</span>
  <p>Second paragraph.</p>
</div>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
section_title = soup.find('h2')

# Find the next <span> tag anywhere after the h2
important_data_span = section_title.find_next('span')

print(f"Section Title: {section_title.text}")
if important_data_span:
  print(f"Data found after title: {important_data_span.text}")

# Find all <p> tags after the h2
all_next_paragraphs = section_title.find_all_next('p')
print("All paragraphs after title:")
for p in all_next_paragraphs:
    print(f"- {p.text}")

Chaining Traversal Methods

The real power of traversal comes from chaining these methods. You can start at one point, move to a parent, then find a sibling, and then extract data from its children.

This allows you to navigate very specific, complex pathways to reach exactly the data you need, even if it's not directly addressable by a simple selector.

from bs4 import BeautifulSoup

html_doc = """
<div class="product-listing">
  <div class="product">
    <h3>Laptop X</h3>
    <p class="price">$1200</p>
  </div>
  <div class="product">
    <h3>Mouse Y</h3>
    <p class="price">$50</p>
  </div>
</div>
<div class="summary">
  <p>Total items: 2</p>
</div>
"""
soup = BeautifulSoup(html_doc, 'html.parser')

# Find "Laptop X" and then get its price
laptop_h3 = soup.find('h3', string='Laptop X')
if laptop_h3:
  # The price is a sibling of the h3
  price_tag = laptop_h3.find_next_sibling('p', class_='price')
  if price_tag:
    print(f"{laptop_h3.text} price: {price_tag.text}")

# Find the product listing, then get the text of the summary's paragraph
product_listing_div = soup.find('div', class_='product-listing')
if product_listing_div:
  summary_div = product_listing_div.find_next_sibling('div', class_='summary')
  if summary_div:
    summary_text = summary_div.find('p').text
    print(f"Summary: {summary_text}")

Handling Text Nodes

When traversing, you'll often encounter NavigableString objects. These represent the text content that isn't wrapped in its own HTML tag, like the whitespace (newlines and spaces) between tags.

BeautifulSoup treats these as distinct nodes in the tree. When using traversal methods like .next_sibling, you might get a NavigableString before you get the next actual tag. Always check .name or type() if you only want tags.

from bs4 import BeautifulSoup

html_doc = """
<div>
  Hello
  <p>World</p>
  !
</div>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
div_tag = soup.find('div')

print("Children of the div:")
for child in div_tag.children:
  if child.name:
    print(f"- Tag: {child.name}, Text: {child.text}")
  else:
    print(f"- Text Node: {repr(child)}")

# Accessing a specific NavigableString
hello_text_node = list(div_tag.children)[0]
print(f"\nFirst child (text node): '{hello_text_node.strip()}'")

Traversal Challenge

Consider the following HTML snippet. You want to extract the "Availability" status for "Product B". Which sequence of BeautifulSoup traversal methods would correctly get you the text "In Stock" starting from the <h3> tag for "Product B"?

<html>
<body>
<div class="catalog">
  <div class="item">
    <h3>Product A</h3>
    <p>Price: $10</p>
    <span>Status: Out of Stock</span>
  </div>
  <div class="item">
    <h3>Product B</h3>
    <p>Price: $20</p>
    <span>Availability: In Stock</span>
  </div>
  <div class="item">
    <h3>Product C</h3>
    <p>Price: $30</p>
    <span>Status: Low Stock</span>
  </div>
</div>
</body>
</html>

Recap: Mastering Traversal

Congratulations! You've learned how to navigate complex HTML structures using BeautifulSoup's powerful traversal methods.

  • We explored moving up (.parent, .parents), down (.contents, .children), and sideways (.next_sibling, .previous_sibling, .next_siblings, .previous_siblings) in the HTML tree.
  • You also saw how to find elements anywhere after or before a current tag using .find_next() and .find_previous().
  • Mastering these techniques allows you to extract data from even the most challenging and irregularly structured web pages.

Next, we'll dive into using CSS selectors for even more precise data extraction!

Başlamak ücretsiz

Yapay zeka eğitmeniyle Python öğren — ücretsiz

Tarayıcında gerçek kod yaz ve çalıştır, 7/24 yapay zeka eğitmeninden anında yardım al; web'de ya da uygulamada kaldığın yerden devam et.

Kurslar
12
Dersler
48

Sıkça Sorulan Sorular

“Karmaşık HTML Yapılarında Gezinme” dersi ücretsiz mi?

Evet — “Karmaşık HTML Yapılarında Gezinme” dersin tüm metni burada web'de ücretsiz olarak okunabilir. Etkileşimli olarak pratik yapmak (yerleşik kod editörü ve 7/24 yapay zeka koçu) ve Web Scraping & Bots kursunun geri kalanını açmak için CoddyKit PRO'ya yükselt. Web Scraping & Bots kursu toplamda 4 dersten oluşur.

“Karmaşık HTML Yapılarında Gezinme” dersinde ne öğreneceğim?

Derin biçimde iç içe geçmiş veya düzensiz yapılandırılmış HTML belgelerinde etkili şekilde gezinme tekniklerini öğrenin. Web Scraping & Bots ile uygulamalı kodu tarayıcıda doğrudan çalıştırarak pratik yaparsın ve 7/24 yapay zeka koçu dersi çalışırken sorularını yanıtlar.

Web Scraping & Bots öğrenmeye başlamak için deneyim gerekli mi?

Önceden deneyim gerekmez. CoddyKit'te Web Scraping & Bots, başlangıçtan ileri seviyeye kadar yapılandırıldığı için buradan başlayabilir veya başından başlayıp kendi hızında ilerleme yapabilirsin. Bu, 4 dersinin 1. dersidir.

“Karmaşık HTML Yapılarında Gezinme” dersi ne kadar sürer?

Çoğu CoddyKit dersi yaklaşık 5–10 dakika sürer. Her biri kısa ve etkileşimli olduğu için sabit ilerleme yaparsın ve web ile uygulama arasında tam olarak bıraktığın yerden devam edebilirsin.

Bu Web Scraping & Bots dersinde kod yazıp çalıştırabilir miyim?

Evet. Her Web Scraping & Bots dersi yerleşik bir kod editörü içerir, bu sayede tarayıcıda gerçek kod yazıp çalıştırabilir ve anlık yapay zeka geri bildirimi alırsın — yerel kurulum gerekli değildir.

Bu kursun tüm dersleri

  1. Karmaşık HTML Yapılarında Gezinme
  2. Hassasiyet İçin CSS Seçicileri
  3. Güçlü Seçim İçin XPath
  4. HTML Tablolarından Veri Çıkarma
← Web Scraping & Bots Sayfasına Dön