Web Scraping & Bots · 课时

Robots.txt 与抓取伦理

了解网页抓取的法律与伦理基础：robots.txt、服务条款、速率限制和负责任的行为。

Robots.txt 与抓取伦理 是 CoddyKit 上的免费 Web Scraping & Bots 课时。这是第 4 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Web Scraping & Bots 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Web Scraping & Bots 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Scraping Responsibly

Scraping touches other people’s servers and data. Before your first scraper, learn the rules and ethics that keep you safe and respectful.

What Is robots.txt?

robots.txt lives at a site’s root and tells automated clients which paths they may or may not access.

# https://example.com/robots.txt
User-agent: *
Disallow: /admin/
Allow: /public/

Reading the Directives

Read the directives: User-agent targets a crawler, Disallow blocks paths, Allow adds exceptions, Crawl-delay sets a wait.

Checking robots.txt in Python

Python’s built-in urllib.robotparser reads and evaluates robots.txt for you — just ask it whether you can fetch a URL.

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url('https://example.com/robots.txt')
rp.read()
print(rp.can_fetch('*', 'https://example.com/public/page'))

robots.txt Is Not Law

Remember: robots.txt is a request, not a lock. Ignoring it isn’t a crime by itself, but it can breach terms of service. Respect it anyway.

Terms of Service

A site’s Terms of Service may flat-out forbid automated access. Breaking them can mean account bans or legal trouble — always check first.

Personal & Copyrighted Data

Tread carefully with personal data (privacy laws like GDPR) and copyrighted content. Collecting or republishing them can carry real consequences.

Rate Limiting

Rapid-fire requests can overload a server — and get you blocked. Add a delay between requests to stay polite.

import time

for url in urls:
    fetch(url)
    time.sleep(2)  # be polite

Identify Yourself

Set an honest User-Agent header, ideally with contact info, so site owners can reach you instead of just blocking you.

headers = {
    'User-Agent': 'MyResearchBot/1.0 (contact@example.com)'
}

Prefer Official APIs

If a site offers an API, use it. APIs are stable, sanctioned, and far gentler than scraping raw HTML.

Cache to Reduce Load

Cache responses locally so you don’t re-fetch the same page over and over while developing.

import os

if not os.path.exists('page.html'):
    save(fetch(url), 'page.html')
html = open('page.html').read()

Quick Check

What is the correct way to think about robots.txt?

Recap

You’ve got scraping ethics: respecting robots.txt, terms, privacy and copyright, rate limiting, honest User-Agents, and preferring APIs and caching.

常见问题解答

「Robots.txt 与抓取伦理」课时是免费的吗？

是的 — 「Robots.txt 与抓取伦理」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Web Scraping & Bots 课程的其余内容，请升级到 CoddyKit PRO。 Web Scraping & Bots 课程共包含 4 节课。

「Robots.txt 与抓取伦理」这节课中我会学到什么？

了解网页抓取的法律与伦理基础：robots.txt、服务条款、速率限制和负责任的行为。你通过在浏览器中直接运行的动手代码来练习 Web Scraping & Bots，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Web Scraping & Bots 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Web Scraping & Bots 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 4 节课，共 4 节。

「Robots.txt 与抓取伦理」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Web Scraping & Bots 课中编写并运行代码吗？

能。每节 Web Scraping & Bots 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Web Scraping & Bots