0Pricing
Web Scraping & Bots · Aula

Entendendo Robots.txt

Interprete e respeite o arquivo `robots.txt` para entender as políticas e restrições de raspagem de um site.

Entendendo Robots.txt é uma aula grátis de Web Scraping & Bots no CoddyKit. Esta é a aula 1 de 4. Você pode ler a aula completa abaixo gratuitamente — depois pratica ao vivo no navegador com um editor de código integrado e um tutor de IA 24/7. Faz parte do caminho de aprendizado de Web Scraping & Bots, e seu progresso é sincronizado entre a web e o app CoddyKit. O curso de Web Scraping & Bots inclui 4 aulas no total.

Partes desta aula ainda não foram traduzidas e aparecem em inglês.

What is Robots.txt?

When building a web scraper or bot, it's crucial to be a good internet citizen. The robots.txt file is a key part of this.

It's a text file that websites use to communicate with web crawlers and other bots. It tells them which parts of the site they are allowed to access and which parts they should avoid.

Finding Robots.txt

Every website that uses a robots.txt file places it in a standard location: the root directory of its domain.

This means you can always find it by adding /robots.txt to the end of the website's main URL. For example:

  • https://www.example.com/robots.txt
  • https://www.google.com/robots.txt

You can simply type this into your browser to view a site's rules.

The User-agent Directive

The User-agent directive specifies which bot the following rules apply to. Think of it as addressing a specific bot or all bots.

  • User-agent: *: This applies to ALL web crawlers and bots.
  • User-agent: Googlebot: This applies only to Google's specific web crawler.
  • User-agent: MyCustomBot: You can even specify rules for your own bot if the website owner knows its name.

Each set of rules starts with a User-agent line.

Blocking Access: Disallow

The Disallow directive is used to tell bots which URLs or directories they should NOT access. It's the primary way to restrict crawling.

Here are some examples:

  • Disallow: /: Disallows access to the entire website (except for robots.txt itself).
  • Disallow: /private/: Disallows access to the /private/ directory and everything within it.
  • Disallow: /search?: Disallows URLs starting with /search?, often used for search results pages.

Always respect these rules!

Allowing Exceptions: Allow

Sometimes, a website might want to disallow a whole directory but allow access to a specific file or sub-directory within it. This is where the Allow directive comes in.

Allow rules override Disallow rules for more specific paths.

For example:

User-agent: *
Disallow: /images/
Allow: /images/public/

This means all bots should avoid the /images/ folder, but they ARE allowed to access content within /images/public/.

Guiding with Sitemap

The Sitemap directive isn't about restricting access; it's about helping bots discover content.

It points to the XML Sitemap file(s) for the website. A sitemap lists all the pages and files a website owner wants search engines to crawl and index.

Example:

Sitemap: https://www.example.com/sitemap.xml

This helps well-behaved bots find your content more efficiently.

Fetching Robots.txt with Python

You can easily fetch a website's robots.txt file using Python's requests library. This allows your script to programmatically read and interpret the rules.

Try running this example to see the robots.txt for Wikipedia:

import requests

def get_robots_txt(domain):
    try:
        response = requests.get(f"https://{domain}/robots.txt")
        response.raise_for_status() # Raise HTTPError for bad responses
        print(f"--- {domain}/robots.txt ---")
        print(response.text)
        print("--------------------------")
    except requests.exceptions.RequestException as e:
        print(f"Error fetching robots.txt for {domain}: {e}")

if __name__ == "__main__":
    get_robots_txt("www.wikipedia.org")
    # You can try other domains too!
    # get_robots_txt("www.google.com")

Interpreting Complex Rules

Let's look at a combined example to understand how rules interact:

User-agent: *
Disallow: /temp/
Disallow: /admin/
Allow: /admin/public/

User-agent: MyBot
Disallow: /
  • A general bot (*) cannot access /temp/ or /admin/, but it CAN access /admin/public/.
  • A bot named MyBot cannot access ANYTHING on the site.

The most specific rule usually wins, especially Allow over Disallow for sub-paths.

Robots.txt is a Guideline, Not Security

It's crucial to understand that robots.txt is a voluntary agreement for well-behaved bots. It's not a security mechanism!

  • Malicious bots can (and often will) ignore these rules.
  • The content of robots.txt itself is public. Don't put sensitive information there.
  • It's for managing server load and respecting content preferences, not hiding data.

Always scrape ethically and respect website policies.

Quick Check: Robots.txt Rules

Consider the following robots.txt content:

User-agent: *
Disallow: /private/
Allow: /private/data.html
Disallow: /temp/

According to these rules, which path is a general bot (User-agent: *) explicitly allowed to access?

Recap: Respecting Robots.txt

In this lesson, we explored the robots.txt file, a fundamental component of ethical web scraping.

  • You learned how to locate it and its core directives: User-agent, Disallow, Allow, and Sitemap.
  • We saw how to fetch and interpret these rules using Python.
  • Crucially, we emphasized that robots.txt is a guideline for respectful bots, not a security measure.

Always check and respect a website's robots.txt before scraping!

Perguntas Frequentes

A aula “Entendendo Robots.txt” é grátis?

Sim — o texto completo de “Entendendo Robots.txt” é grátis para ler aqui na web. Para praticá-la interativamente (um editor de código integrado e um tutor de IA 24/7) e desbloquear o restante do curso de Web Scraping & Bots, atualize para CoddyKit PRO. O curso de Web Scraping & Bots inclui 4 aulas no total.

O que vou aprender em “Entendendo Robots.txt”?

Interprete e respeite o arquivo `robots.txt` para entender as políticas e restrições de raspagem de um site. Você pratica Web Scraping & Bots com código prático que executa diretamente no navegador, e um tutor de IA 24/7 responde suas dúvidas enquanto trabalha na aula.

Preciso ter experiência prévia para começar Web Scraping & Bots?

Nenhuma experiência prévia é necessária. Web Scraping & Bots no CoddyKit é estruturado para alunos iniciantes até avançados, então você pode começar aqui ou desde o início e aprender no seu ritmo. Esta é a aula 1 de 4.

Quanto tempo leva a aula “Entendendo Robots.txt”?

A maioria das aulas CoddyKit leva cerca de 5–10 minutos. Cada uma é compacta e interativa, então você faz progresso constante e retoma exatamente de onde parou entre web e app.

Posso escrever e executar código nesta aula de Web Scraping & Bots?

Sim. Cada aula de Web Scraping & Bots inclui um editor de código integrado, então você escreve e executa código real direto no navegador e recebe feedback de IA instantaneamente — nenhuma configuração local necessária.

Todas as aulas deste curso

  1. Entendendo Robots.txt
  2. Termos de serviço e direitos autorais
  3. Práticas éticas de raspagem
  4. Limitação de Taxa e Navegação Respeitosa
← Voltar para Web Scraping & Bots