0Pricing
Web Scraping & Bots · レッスン

URLに対するRequestsの利用

PythonのRequestsライブラリを使ってGETおよびPOSTリクエストを送信し、Webページのコンテンツを取得する方法を学びます。

「URLに対するRequestsの利用」はCoddyKit上の無料Web Scraping & Botsレッスンです。 これはレッスン2/4です。 下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。 これはWeb Scraping & Bots学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 Web Scraping & Botsコースには全4レッスンが含まれています。

このレッスンの一部はまだ翻訳されておらず、英語で表示されています。

Meet Python's Requests Library

Welcome to using Python for web interactions! In this lesson, we'll dive into the Requests library, your go-to tool for making HTTP requests.

Requests simplifies how your Python program talks to websites, acting like a web browser but without a graphical interface. It's essential for fetching web page content before you can extract data.

Setting Up Requests

Before we can use Requests, we need to install it. If you haven't already, open your terminal or command prompt and run the following command:

pip install requests

This command downloads and installs the library, making it available for your Python scripts. It's a one-time setup for your environment.

Your First GET Request

The most common type of request is GET. It's used to retrieve data from a specified resource, much like when your browser loads a webpage.

Let's make a simple GET request to fetch content from example.com. We'll then print the HTTP status code and a snippet of the page's text.

import requests

# Define the URL you want to fetch
url = "http://www.example.com"

# Send a GET request
response = requests.get(url)

# Print the status code and first 200 characters of the content
print(f"Status Code: {response.status_code}")
print(f"Content snippet:\n{response.text[:200]}")

Understanding the Response Object

When you make a request, the requests.get() function returns a Response object. This object holds all the information about the server's reply.

  • response.status_code: An integer indicating the HTTP status (e.g., 200 for OK, 404 for Not Found).
  • response.text: The content of the response, usually HTML, as a string.
  • response.url: The actual URL of the response.

Basic Error Checking

It's good practice to check if your request was successful. The response.raise_for_status() method is a simple way to do this.

If the HTTP status code indicates an error (e.g., 4xx or 5xx), this method will raise an HTTPError. Otherwise, it does nothing.

import requests

url = "http://www.example.com/nonexistent-page"

try:
    response = requests.get(url)
    response.raise_for_status() # Raises an HTTPError for bad responses (4xx or 5xx)
    print("Request successful!")
except requests.exceptions.HTTPError as err:
    print(f"HTTP Error occurred: {err}")
except requests.exceptions.ConnectionError as err:
    print(f"Connection Error occurred: {err}")
except Exception as err:
    print(f"An unexpected error occurred: {err}")

Adding Request Headers

Sometimes, websites check for specific HTTP headers to determine if a request is coming from a legitimate browser or a bot.

You can customize your request by passing a dictionary of headers. A common header to set is User-Agent to mimic a browser.

import requests

url = "http://httpbin.org/get" # A service that echoes your request

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9"
}

response = requests.get(url, headers=headers)

print("Your request headers sent to httpbin.org:")
# httpbin.org returns JSON, so we can use .json()
print(response.json()['headers']['User-Agent'])
print(response.json()['headers']['Accept-Language'])

Understanding POST Requests

While GET requests are for retrieving data, POST requests are used to send data to a server, typically for creating or updating a resource.

Think of submitting a form on a website – you're usually sending data via a POST request. The data is included in the request body, not in the URL.

Sending Data with POST

To send a POST request with data, you use requests.post() and pass your data as a dictionary to the data parameter.

Let's try sending some simple form data to httpbin.org/post, which will echo back the data it received.

import requests

url = "http://httpbin.org/post"

# Data to send in the POST request
payload = {
    "name": "Coddy",
    "city": "Kitland",
    "age": "5"
}

response = requests.post(url, data=payload)

print("Data received by httpbin.org:")
# The 'form' key in the JSON response contains the sent data
print(response.json()['form'])

GET vs. POST: Key Differences

It's crucial to understand when to use GET versus POST:

  • GET: Retrieves data, parameters are visible in the URL, requests can be bookmarked and cached. Best for non-sensitive data retrieval.
  • POST: Sends data to be processed, parameters are in the request body (not visible in URL), requests are not cached or bookmarked. Best for submitting forms, uploading files, or sensitive data.

Check Your Understanding

You've learned about GET and POST requests. Consider the following scenario:

You want to retrieve the current weather forecast for a specific city from a public API. Which HTTP request method is most appropriate?

Lesson Summary

Great job! You've taken a significant step in understanding how to interact with the web using Python's Requests library.

  • We installed the requests library.
  • We learned to send GET requests to fetch web content.
  • We explored the response object, checking status codes and content.
  • We briefly touched on handling request errors.
  • We customized requests with headers.
  • We learned to send data using POST requests.
  • Finally, we distinguished between GET and POST for different web interactions.

Next, we'll learn how to parse the HTML content you've fetched!

よくある質問

「URLに対するRequestsの利用」レッスンは無料ですか？

はい。「URLに対するRequestsの利用」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、Web Scraping & Botsコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 Web Scraping & Botsコースには全4レッスンが含まれています。

「URLに対するRequestsの利用」で何を学びますか？

PythonのRequestsライブラリを使ってGETおよびPOSTリクエストを送信し、Webページのコンテンツを取得する方法を学びます。 ブラウザで直接実行するハンズオンコードでWeb Scraping & Botsを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。

Web Scraping & Botsを始めるのに経験は必要ですか？

事前経験は必要ありません。CoddyKitのWeb Scraping & Botsは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。 これはレッスン2/4です。

「URLに対するRequestsの利用」レッスンにはどのくらい時間がかかりますか？

ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。

このWeb Scraping & Botsレッスンでコードを書いて実行できますか？

はい。すべてのWeb Scraping & Botsレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。

このコースのすべてのレッスン

  1. 環境のセットアップ
  2. URLに対するRequestsの利用
  3. BeautifulSoupによるデータ抽出
  4. CSS セレクターで DOM をたどる
← Web Scraping & Botsに戻る