Robots.txt & Scraping Ethics
Understand the legal and ethical foundations of web scraping: robots.txt, terms of service, rate limiting and responsible behaviour.
Robots.txt & Scraping Ethics is a free Web Scraping & Bots lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Web Scraping & Bots learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Scraping Responsibly
Scraping touches other people’s servers and data. Before your first scraper, learn the rules and ethics that keep you safe and respectful.
What Is robots.txt?
robots.txt lives at a site’s root and tells automated clients which paths they may or may not access.
# https://example.com/robots.txt
User-agent: *
Disallow: /admin/
Allow: /public/Reading the Directives
Read the directives: User-agent targets a crawler, Disallow blocks paths, Allow adds exceptions, Crawl-delay sets a wait.
Checking robots.txt in Python
Python’s built-in urllib.robotparser reads and evaluates robots.txt for you — just ask it whether you can fetch a URL.
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url('https://example.com/robots.txt')
rp.read()
print(rp.can_fetch('*', 'https://example.com/public/page'))robots.txt Is Not Law
Remember: robots.txt is a request, not a lock. Ignoring it isn’t a crime by itself, but it can breach terms of service. Respect it anyway.
Terms of Service
A site’s Terms of Service may flat-out forbid automated access. Breaking them can mean account bans or legal trouble — always check first.
Personal & Copyrighted Data
Tread carefully with personal data (privacy laws like GDPR) and copyrighted content. Collecting or republishing them can carry real consequences.
Rate Limiting
Rapid-fire requests can overload a server — and get you blocked. Add a delay between requests to stay polite.
import time
for url in urls:
fetch(url)
time.sleep(2) # be politeIdentify Yourself
Set an honest User-Agent header, ideally with contact info, so site owners can reach you instead of just blocking you.
headers = {
'User-Agent': 'MyResearchBot/1.0 (contact@example.com)'
}Prefer Official APIs
If a site offers an API, use it. APIs are stable, sanctioned, and far gentler than scraping raw HTML.
Cache to Reduce Load
Cache responses locally so you don’t re-fetch the same page over and over while developing.
import os
if not os.path.exists('page.html'):
save(fetch(url), 'page.html')
html = open('page.html').read()Quick Check
What is the correct way to think about robots.txt?
Recap
You’ve got scraping ethics: respecting robots.txt, terms, privacy and copyright, rate limiting, honest User-Agents, and preferring APIs and caching.
Frequently asked questions
Is the “Robots.txt & Scraping Ethics” lesson free?
Yes — the full text of “Robots.txt & Scraping Ethics” is free to read here on the web, and the Web Scraping & Bots course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Web Scraping & Bots course, upgrade to CoddyKit PRO.
What will I learn in “Robots.txt & Scraping Ethics”?
Understand the legal and ethical foundations of web scraping: robots.txt, terms of service, rate limiting and responsible behaviour. You practise Web Scraping & Bots with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Web Scraping & Bots?
No prior experience is required. Web Scraping & Bots on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Robots.txt & Scraping Ethics” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Web Scraping & Bots lesson?
Yes. Every Web Scraping & Bots lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- What is Web Scraping?
- HTTP Requests & Responses
- Inspecting Web Pages
- Robots.txt & Scraping Ethics