Web Scraping & Bots · 课时

合乎道德的网络抓取实践

实施速率限制、正确标识用户代理和尊重服务器负载等最佳实践，负责任地进行网络抓取。

合乎道德的网络抓取实践 是 CoddyKit 上的免费 Web Scraping & Bots 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Web Scraping & Bots 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Web Scraping & Bots 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Scrape Ethically?

Welcome to Ethical Scraping Practices! Web scraping is a powerful tool, but it comes with responsibilities.

Being an ethical scraper means more than just avoiding legal trouble. It's about being a good internet citizen, respecting website resources, and ensuring the sustainability of your scraping efforts.

Respect Server Load

Imagine thousands of requests hitting a website at once. This can overwhelm the server, slow down the site for other users, or even crash it. This is similar to a Denial-of-Service (DoS) attack.

An ethical scraper avoids putting undue strain on a website's infrastructure. We want to collect data, not cause problems!

Implement Rate Limiting

The best way to respect server load is through rate limiting. This means introducing delays between your requests to a website.

By waiting a few seconds between each page fetch, you give the server time to process your request and serve other users, mimicking human browsing behavior.

Rate Limiting Example

Here's a simple Python example using time.sleep() to introduce a delay between requests. Try running it!

import requests
import time

def fetch_url_with_delay(url, delay_seconds):
  print(f"Fetching {url}...")
  try:
    response = requests.get(url)
    print(f"Status: {response.status_code}")
  except requests.exceptions.RequestException as e:
    print(f"Error fetching {url}: {e}")
  time.sleep(delay_seconds) # Wait before next request

if __name__ == "__main__":
  target_url = "https://httpbin.org/get" # A safe test URL
  print("Starting requests with delays...")
  for i in range(2):
    fetch_url_with_delay(target_url, 3) # Wait 3 seconds
  print("Finished scraping with delays.")

Identify Yourself (Politely!)

When your browser makes a request, it sends a User-Agent header. This header tells the server information about the client, like the browser type (e.g., Chrome, Firefox) and operating system.

As an ethical scraper, you should set a custom, descriptive User-Agent. Include your bot's name and contact information so website administrators can reach you if there are issues.

Custom User-Agent

Setting a custom User-Agent is straightforward with the requests library. Here’s how you can do it:

import requests

def fetch_with_custom_ua(url):
  headers = {
    "User-Agent": "CoddyKitScraper/1.0 (contact@example.com)",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
  }
  print(f"Fetching {url} with custom User-Agent...")
  try:
    response = requests.get(url, headers=headers)
    print(f"Status: {response.status_code}")
    print(f"User-Agent sent: {response.request.headers['User-Agent']}")
  except requests.exceptions.RequestException as e:
    print(f"Error fetching {url}: {e}")

if __name__ == "__main__":
  target_url = "https://httpbin.org/get" # A safe test URL
  fetch_with_custom_ua(target_url)
  print("Finished request with custom User-Agent.")

Check robots.txt (Again!)

Even if you're rate limiting and using a proper User-Agent, always remember to check a website's robots.txt file.

This file is a standard way for websites to communicate their scraping policies, telling you which parts of the site they prefer you don't access. Respecting it is a cornerstone of ethical scraping.

Handle Data Responsibly

Ethical scraping extends beyond just the act of collecting data; it also covers what you do with it afterward. Consider these points:

Privacy: Avoid collecting personally identifiable information (PII) without explicit consent.
Anonymization: Anonymize data where possible to protect individuals.
Compliance: Adhere to data privacy regulations like GDPR or CCPA.
Misuse: Do not misrepresent, resell, or exploit scraped data in ways that harm individuals or businesses.

Key Ethical Practices

To summarize, here are the core ethical practices for web scraping:

Respect robots.txt: Always check and follow its directives.
Rate Limit Your Requests: Introduce delays to avoid overwhelming servers.
Use a Descriptive User-Agent: Identify your bot with contact information.
Handle Data Responsibly: Prioritize privacy and legal compliance.
Monitor Server Load: Be aware of your impact and adjust if necessary.

Ethical Scraper Quiz

Test your understanding of ethical scraping practices.

Recap & Next Steps

You've learned that ethical scraping is crucial for responsible data collection. This involves respecting server load through rate limiting, clearly identifying your bot with a proper User-Agent, and handling collected data responsibly.

Always strive to be a good internet citizen! In the next lessons, we'll explore more advanced topics like data storage and building your first bot.

常见问题解答

「合乎道德的网络抓取实践」课时是免费的吗？

是的 — 「合乎道德的网络抓取实践」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Web Scraping & Bots 课程的其余内容，请升级到 CoddyKit PRO。 Web Scraping & Bots 课程共包含 4 节课。

「合乎道德的网络抓取实践」这节课中我会学到什么？

实施速率限制、正确标识用户代理和尊重服务器负载等最佳实践，负责任地进行网络抓取。你通过在浏览器中直接运行的动手代码来练习 Web Scraping & Bots，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Web Scraping & Bots 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Web Scraping & Bots 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「合乎道德的网络抓取实践」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Web Scraping & Bots 课中编写并运行代码吗？

能。每节 Web Scraping & Bots 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Web Scraping & Bots