Web Scraping & Bots · 课时

了解 Robots.txt

解读并遵守 `robots.txt` 文件，了解网站的抓取政策和限制。

了解 Robots.txt 是 CoddyKit 上的免费 Web Scraping & Bots 课时。这是第 1 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Web Scraping & Bots 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Web Scraping & Bots 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

What is Robots.txt?

When building a web scraper or bot, it's crucial to be a good internet citizen. The robots.txt file is a key part of this.

It's a text file that websites use to communicate with web crawlers and other bots. It tells them which parts of the site they are allowed to access and which parts they should avoid.

Finding Robots.txt

Every website that uses a robots.txt file places it in a standard location: the root directory of its domain.

This means you can always find it by adding /robots.txt to the end of the website's main URL. For example:

https://www.example.com/robots.txt
https://www.google.com/robots.txt

You can simply type this into your browser to view a site's rules.

The User-agent Directive

The User-agent directive specifies which bot the following rules apply to. Think of it as addressing a specific bot or all bots.

User-agent: *: This applies to ALL web crawlers and bots.
User-agent: Googlebot: This applies only to Google's specific web crawler.
User-agent: MyCustomBot: You can even specify rules for your own bot if the website owner knows its name.

Each set of rules starts with a User-agent line.

Blocking Access: Disallow

The Disallow directive is used to tell bots which URLs or directories they should NOT access. It's the primary way to restrict crawling.

Here are some examples:

Disallow: /: Disallows access to the entire website (except for robots.txt itself).
Disallow: /private/: Disallows access to the /private/ directory and everything within it.
Disallow: /search?: Disallows URLs starting with /search?, often used for search results pages.

Always respect these rules!

Allowing Exceptions: Allow

Sometimes, a website might want to disallow a whole directory but allow access to a specific file or sub-directory within it. This is where the Allow directive comes in.

Allow rules override Disallow rules for more specific paths.

For example:

User-agent: *
Disallow: /images/
Allow: /images/public/

This means all bots should avoid the /images/ folder, but they ARE allowed to access content within /images/public/.

Guiding with Sitemap

The Sitemap directive isn't about restricting access; it's about helping bots discover content.

It points to the XML Sitemap file(s) for the website. A sitemap lists all the pages and files a website owner wants search engines to crawl and index.

Example:

Sitemap: https://www.example.com/sitemap.xml

This helps well-behaved bots find your content more efficiently.

Fetching Robots.txt with Python

You can easily fetch a website's robots.txt file using Python's requests library. This allows your script to programmatically read and interpret the rules.

Try running this example to see the robots.txt for Wikipedia:

import requests

def get_robots_txt(domain):
    try:
        response = requests.get(f"https://{domain}/robots.txt")
        response.raise_for_status() # Raise HTTPError for bad responses
        print(f"--- {domain}/robots.txt ---")
        print(response.text)
        print("--------------------------")
    except requests.exceptions.RequestException as e:
        print(f"Error fetching robots.txt for {domain}: {e}")

if __name__ == "__main__":
    get_robots_txt("www.wikipedia.org")
    # You can try other domains too!
    # get_robots_txt("www.google.com")

Interpreting Complex Rules

Let's look at a combined example to understand how rules interact:

User-agent: *
Disallow: /temp/
Disallow: /admin/
Allow: /admin/public/

User-agent: MyBot
Disallow: /

A general bot (*) cannot access /temp/ or /admin/, but it CAN access /admin/public/.
A bot named MyBot cannot access ANYTHING on the site.

The most specific rule usually wins, especially Allow over Disallow for sub-paths.

Robots.txt is a Guideline, Not Security

It's crucial to understand that robots.txt is a voluntary agreement for well-behaved bots. It's not a security mechanism!

Malicious bots can (and often will) ignore these rules.
The content of robots.txt itself is public. Don't put sensitive information there.
It's for managing server load and respecting content preferences, not hiding data.

Always scrape ethically and respect website policies.

Quick Check: Robots.txt Rules

Consider the following robots.txt content:

User-agent: *
Disallow: /private/
Allow: /private/data.html
Disallow: /temp/

According to these rules, which path is a general bot (User-agent: *) explicitly allowed to access?

Recap: Respecting Robots.txt

In this lesson, we explored the robots.txt file, a fundamental component of ethical web scraping.

You learned how to locate it and its core directives: User-agent, Disallow, Allow, and Sitemap.
We saw how to fetch and interpret these rules using Python.
Crucially, we emphasized that robots.txt is a guideline for respectful bots, not a security measure.

Always check and respect a website's robots.txt before scraping!

常见问题解答

「了解 Robots.txt」课时是免费的吗？

是的 — 「了解 Robots.txt」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Web Scraping & Bots 课程的其余内容，请升级到 CoddyKit PRO。 Web Scraping & Bots 课程共包含 4 节课。

「了解 Robots.txt」这节课中我会学到什么？

解读并遵守 `robots.txt` 文件，了解网站的抓取政策和限制。你通过在浏览器中直接运行的动手代码来练习 Web Scraping & Bots，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Web Scraping & Bots 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Web Scraping & Bots 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 1 节课，共 4 节。

「了解 Robots.txt」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Web Scraping & Bots 课中编写并运行代码吗？

能。每节 Web Scraping & Bots 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Web Scraping & Bots