Web Scraping & Bots · Lesson

Robots.txt & Scraping Ethics

Understand the legal and ethical foundations of web scraping: robots.txt, terms of service, rate limiting and responsible behaviour.

Scraping Responsibly

Scraping touches other people’s servers and data. Before your first scraper, learn the rules and ethics that keep you safe and respectful.

robots.txt lives at a site’s root and tells automated clients which paths they may or may not access.

# https://example.com/robots.txt
User-agent: *
Disallow: /admin/
Allow: /public/