0Pricing
AI Agents with LangChain & Autonomous Workflows · レッスン

ウェブスクレイピングとデータ拡張

エージェントがウェブサイトから情報を抽出し、知識ベースを動的に拡充する手法を実装します。

「ウェブスクレイピングとデータ拡張」はCoddyKit上の無料AI Agents with LangChain & Autonomous Workflowsレッスンです。 これはレッスン3/4です。 下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。 これはAI Agents with LangChain & Autonomous Workflows学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 AI Agents with LangChain & Autonomous Workflowsコースには全4レッスンが含まれています。

このレッスンの一部はまだ翻訳されておらず、英語で表示されています。

Web Scraping for Agents

AI agents often need up-to-date, specific information that isn't part of their pre-trained knowledge or available via simple API calls.

This is where web scraping comes in! It allows agents to "read" and extract data directly from web pages, just like a human would.

What is Web Scraping?

Web scraping is the automated process of extracting data from websites. Instead of manually copying information, a program does it for you.

For AI agents, this means they can dynamically gather facts, prices, news, or any other publicly available information from the internet to inform their decisions or responses.

Essential Python Libraries

Python has powerful libraries that make web scraping straightforward:

  • requests: Used to make HTTP requests (like visiting a website) and get the raw HTML content.
  • BeautifulSoup (often imported as bs4): A library for parsing HTML and XML documents, making it easy to extract data.

Together, they form a common duo for scraping tasks.

Getting Web Page Content

First, we use the requests library to fetch the content of a web page. This sends a request to the server and gets the HTML response.

Try running this example to see how we fetch the content from example.com:

import requests

def main():
    try:
        # Send a GET request to the URL
        response = requests.get("https://www.example.com")
        
        # Check if the request was successful (status code 200)
        if response.status_code == 200:
            print(f"Successfully fetched example.com!")
            print(f"Content Length: {len(response.text)} characters")
        else:
            print(f"Failed to fetch. Status Code: {response.status_code}")
    except requests.exceptions.RequestException as e:
        print(f"Error fetching URL: {e}")

if __name__ == "__main__":
    main()

HTML Basics for Scraping

Web pages are built with HTML (HyperText Markup Language). HTML uses tags like <h1>, <p>, <a> to structure content.

BeautifulSoup helps us navigate this structure. To extract data, we need to know what tags, classes, or IDs the information is wrapped in.

Parsing with BeautifulSoup

Once you have the raw HTML, BeautifulSoup turns it into a parse tree, allowing you to search for elements easily.

Here's how to create a BeautifulSoup object and extract a simple title from a string:

from bs4 import BeautifulSoup

def main():
    html_doc = """
    <html><head><title>CoddyKit Lesson</title></head>
    <body>
    <p class="intro"><b>Hello Learners!</b></p>
    </body></html>
    """
    
    # Create a BeautifulSoup object
    soup = BeautifulSoup(html_doc, 'html.parser')
    
    # Access the title tag and its string content
    print(f"Page Title: {soup.title.string}")

if __name__ == "__main__":
    main()

Finding Specific Data

BeautifulSoup offers powerful methods like find() (for the first match) and find_all() (for all matches) to locate elements by tag name, attributes, or CSS selectors.

Let's find the text inside an <h1> tag:

from bs4 import BeautifulSoup

def main():
    html_doc = """
    <html>
    <body>
        <h1>Welcome to Our Course</h1>
        <p>This is an example paragraph.</p>
        <div id="footer">Contact Us</div>
    </body>
    </html>
    """
    soup = BeautifulSoup(html_doc, 'html.parser')
    
    # Find the first h1 tag
    first_h1 = soup.find('h1')
    if first_h1:
        print(f"Extracted H1 Text: {first_h1.get_text()}")
    else:
        print("H1 tag not found.")

if __name__ == "__main__":
    main()

Data Augmentation for Agents

Data augmentation, in this context, refers to the process of enhancing an agent's internal knowledge base or current context with newly acquired information.

When an agent scrapes a website, it's not just "reading" for itself; it's gathering data that can be used to improve its understanding, answer questions, or make more informed decisions.

Enhancing Agent Capabilities

Imagine an agent that needs to provide real-time stock prices or the latest news headlines. Its pre-trained model won't have this info.

  • Real-time data: Scrape current stock prices or news.
  • Specific facts: Extract product details from an e-commerce site.
  • Contextual understanding: Get background info on a topic not covered in its training.

This scraped data can then be passed to the LLM as part of the prompt, augmenting its knowledge.

Ethical & Legal Considerations

When scraping, always be mindful of:

  • robots.txt: A file on websites that tells bots which parts of the site they shouldn't access. Respect it!
  • Terms of Service: Many sites prohibit scraping in their terms.
  • Rate Limiting: Don't bombard a server with too many requests too quickly; it can be seen as a DoS attack. Introduce delays.

Scrape responsibly and ethically.

Quick Check

You've learned about fetching web content and parsing HTML. Which Python library is primarily used for navigating and extracting data from HTML documents?

Recap & Next Steps

In this lesson, you learned how to use Python's requests and BeautifulSoup libraries for web scraping. We covered fetching raw HTML and then parsing it to extract specific information.

You also understood how this scraped data contributes to data augmentation, enabling AI agents to access current and specific information, thereby enriching their knowledge and capabilities dynamically. Always remember to scrape responsibly!

よくある質問

「ウェブスクレイピングとデータ拡張」レッスンは無料ですか？

はい。「ウェブスクレイピングとデータ拡張」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、AI Agents with LangChain & Autonomous Workflowsコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 AI Agents with LangChain & Autonomous Workflowsコースには全4レッスンが含まれています。

「ウェブスクレイピングとデータ拡張」で何を学びますか？

エージェントがウェブサイトから情報を抽出し、知識ベースを動的に拡充する手法を実装します。 ブラウザで直接実行するハンズオンコードでAI Agents with LangChain & Autonomous Workflowsを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。

AI Agents with LangChain & Autonomous Workflowsを始めるのに経験は必要ですか？

事前経験は必要ありません。CoddyKitのAI Agents with LangChain & Autonomous Workflowsは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。 これはレッスン3/4です。

「ウェブスクレイピングとデータ拡張」レッスンにはどのくらい時間がかかりますか？

ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。

このAI Agents with LangChain & Autonomous Workflowsレッスンでコードを書いて実行できますか？

はい。すべてのAI Agents with LangChain & Autonomous Workflowsレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。

このコースのすべてのレッスン

  1. カスタムLangChainツールの作成
  2. 外部APIとの統合
  3. ウェブスクレイピングとデータ拡張
  4. ツールキットと構造化ツール入力
← AI Agents with LangChain & Autonomous Workflowsに戻る