Web Scraping & Bots · 강의

Robots.txt 이해하기

웹사이트의 스크래핑 정책과 제한을 이해하기 위해 `robots.txt` 파일을 해석하고 준수합니다.

Robots.txt 이해하기은(는) CoddyKit의 무료 Web Scraping & Bots 강의입니다. 이것은 4개 중 1번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 Web Scraping & Bots 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. Web Scraping & Bots 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

What is Robots.txt?

When building a web scraper or bot, it's crucial to be a good internet citizen. The robots.txt file is a key part of this.

It's a text file that websites use to communicate with web crawlers and other bots. It tells them which parts of the site they are allowed to access and which parts they should avoid.

Finding Robots.txt

Every website that uses a robots.txt file places it in a standard location: the root directory of its domain.

This means you can always find it by adding /robots.txt to the end of the website's main URL. For example:

https://www.example.com/robots.txt
https://www.google.com/robots.txt

You can simply type this into your browser to view a site's rules.

The User-agent Directive

The User-agent directive specifies which bot the following rules apply to. Think of it as addressing a specific bot or all bots.

User-agent: *: This applies to ALL web crawlers and bots.
User-agent: Googlebot: This applies only to Google's specific web crawler.
User-agent: MyCustomBot: You can even specify rules for your own bot if the website owner knows its name.

Each set of rules starts with a User-agent line.

Blocking Access: Disallow

The Disallow directive is used to tell bots which URLs or directories they should NOT access. It's the primary way to restrict crawling.

Here are some examples:

Disallow: /: Disallows access to the entire website (except for robots.txt itself).
Disallow: /private/: Disallows access to the /private/ directory and everything within it.
Disallow: /search?: Disallows URLs starting with /search?, often used for search results pages.

Always respect these rules!

Allowing Exceptions: Allow

Sometimes, a website might want to disallow a whole directory but allow access to a specific file or sub-directory within it. This is where the Allow directive comes in.

Allow rules override Disallow rules for more specific paths.

For example:

User-agent: *
Disallow: /images/
Allow: /images/public/

This means all bots should avoid the /images/ folder, but they ARE allowed to access content within /images/public/.

Guiding with Sitemap

The Sitemap directive isn't about restricting access; it's about helping bots discover content.

It points to the XML Sitemap file(s) for the website. A sitemap lists all the pages and files a website owner wants search engines to crawl and index.

Example:

Sitemap: https://www.example.com/sitemap.xml

This helps well-behaved bots find your content more efficiently.

Fetching Robots.txt with Python

You can easily fetch a website's robots.txt file using Python's requests library. This allows your script to programmatically read and interpret the rules.

Try running this example to see the robots.txt for Wikipedia:

import requests

def get_robots_txt(domain):
    try:
        response = requests.get(f"https://{domain}/robots.txt")
        response.raise_for_status() # Raise HTTPError for bad responses
        print(f"--- {domain}/robots.txt ---")
        print(response.text)
        print("--------------------------")
    except requests.exceptions.RequestException as e:
        print(f"Error fetching robots.txt for {domain}: {e}")

if __name__ == "__main__":
    get_robots_txt("www.wikipedia.org")
    # You can try other domains too!
    # get_robots_txt("www.google.com")

Interpreting Complex Rules

Let's look at a combined example to understand how rules interact:

User-agent: *
Disallow: /temp/
Disallow: /admin/
Allow: /admin/public/

User-agent: MyBot
Disallow: /

A general bot (*) cannot access /temp/ or /admin/, but it CAN access /admin/public/.
A bot named MyBot cannot access ANYTHING on the site.

The most specific rule usually wins, especially Allow over Disallow for sub-paths.

Robots.txt is a Guideline, Not Security

It's crucial to understand that robots.txt is a voluntary agreement for well-behaved bots. It's not a security mechanism!

Malicious bots can (and often will) ignore these rules.
The content of robots.txt itself is public. Don't put sensitive information there.
It's for managing server load and respecting content preferences, not hiding data.

Always scrape ethically and respect website policies.

Quick Check: Robots.txt Rules

Consider the following robots.txt content:

User-agent: *
Disallow: /private/
Allow: /private/data.html
Disallow: /temp/

According to these rules, which path is a general bot (User-agent: *) explicitly allowed to access?

Recap: Respecting Robots.txt

In this lesson, we explored the robots.txt file, a fundamental component of ethical web scraping.

You learned how to locate it and its core directives: User-agent, Disallow, Allow, and Sitemap.
We saw how to fetch and interpret these rules using Python.
Crucially, we emphasized that robots.txt is a guideline for respectful bots, not a security measure.

Always check and respect a website's robots.txt before scraping!

자주 묻는 질문

“Robots.txt 이해하기” 강의는 무료인가요?

네 — “Robots.txt 이해하기” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 Web Scraping & Bots 강의 전체를 잠금 해제할 수 있습니다. Web Scraping & Bots 강의에는 총 4개의 강의가 포함되어 있습니다.

“Robots.txt 이해하기”에서 뭘 배우나요?

웹사이트의 스크래핑 정책과 제한을 이해하기 위해 `robots.txt` 파일을 해석하고 준수합니다. 브라우저에서 직접 실행하는 실습 코드로 Web Scraping & Bots을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

Web Scraping & Bots을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 Web Scraping & Bots은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 1번째 강의입니다.

“Robots.txt 이해하기” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 Web Scraping & Bots 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 Web Scraping & Bots 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← Web Scraping & Bots(으)로 돌아가기