PricingLoginStart free
Python Academy · Lesson

Building a Scrapy Spider

Create a Scrapy project, write a spider, and follow links automatically.

Lesson 3 of 413 steps

Building a Scrapy Spider is a free Python Academy lesson on CoddyKit. This is lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Python Academy learning path, and your progress syncs across the web and the CoddyKit app. The Python Academy course includes 4 lessons in total.

What Is Scrapy?

Scrapy is a full-featured web scraping framework with built-in asynchronous crawling, item pipelines, middlewares, and exporters.

# pip install scrapy

# Create a project:
# scrapy startproject myspider

# myspider/
#   scrapy.cfg
#   myspider/
#     settings.py
#     spiders/
#       quotes_spider.py

Creating a Spider

Subclass scrapy.Spider, set name and start_urls, then implement parse().

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
            }

CSS and XPath Selectors

Use response.css("selector") or response.xpath("//xpath") to extract data. .get() returns the first match; .getall() returns all.

def parse(self, response):
    # CSS:
    titles = response.css("h2.title::text").getall()

    # XPath:
    links = response.xpath("//a/@href").getall()

    yield {"titles": titles, "links": links}

Following Links

Yield a scrapy.Request to follow a link and parse the next page.

import scrapy

class Spider(scrapy.Spider):
    name = "crawler"
    start_urls = ["https://quotes.toscrape.com"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {"text": quote.css("span.text::text").get()}

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Items and ItemLoaders

Define Item classes to structure scraped data, and use ItemLoader for cleaning and transforming fields.

import scrapy

class ProductItem(scrapy.Item):
    name  = scrapy.Field()
    price = scrapy.Field()
    url   = scrapy.Field()

def parse(self, response):
    item = ProductItem()
    item["name"]  = response.css("h1::text").get(strip=True)
    item["price"] = response.css(".price::text").re_first(r"[\d.]+")
    item["url"]   = response.url
    yield item

Running the Spider

Run from the project directory. Export to JSON, CSV, or a custom pipeline.

# Run and print to console:
# scrapy crawl quotes

# Save to JSON:
# scrapy crawl quotes -o quotes.json

# Save to CSV:
# scrapy crawl quotes -o quotes.csv

# Scrapy shell for interactive testing:
# scrapy shell "https://quotes.toscrape.com"

Settings: DOWNLOAD_DELAY and CONCURRENT_REQUESTS

Be polite: add delays and limit concurrent requests to avoid overloading servers.

# settings.py
DOWNLOAD_DELAY = 1          # 1 second between requests
CONCURRENT_REQUESTS = 8     # max 8 at a time
CONCURRENT_REQUESTS_PER_DOMAIN = 2
ROBOTSTXT_OBEY = True       # respect robots.txt

Middlewares

Middlewares intercept requests and responses. Use built-in ones for user-agent rotation, retries, and cookies.

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
    "myspider.middlewares.RotateUserAgentMiddleware": 400,
}

Item Pipelines

Pipelines process items after scraping: clean data, validate, deduplicate, and save to a database.

# pipelines.py
class CleanPipeline:
    def process_item(self, item, spider):
        item["name"] = item["name"].strip()
        return item

class DBPipeline:
    def open_spider(self, spider):
        self.conn = connect_db()

    def process_item(self, item, spider):
        self.conn.insert(dict(item))
        return item

Handling Login

Use Scrapy's FormRequest to submit a login form and persist the session cookie.

import scrapy

class AuthSpider(scrapy.Spider):
    name = "auth"
    login_url = "https://example.com/login"

    def start_requests(self):
        yield scrapy.Request(self.login_url, self.login)

    def login(self, response):
        yield scrapy.FormRequest.from_response(
            response,
            formdata={"user": "me", "pass": "secret"},
            callback=self.parse
        )

scrapy-playwright for JS Pages

The scrapy-playwright plugin integrates Playwright into Scrapy for JavaScript-rendered pages.

# pip install scrapy-playwright
# playwright install

# settings.py:
# DOWNLOAD_HANDLERS = {
#     "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
#     "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
# }

# In spider:
# yield scrapy.Request(url, meta={"playwright": True})

Quick Check

What does response.follow(href, callback) do in a Scrapy spider?

Answer it in the interactive lesson

Recap

Scrapy spiders subclass scrapy.Spider, declare start_urls, and yield items or further Requests from parse(). Use CSS/XPath selectors, be polite with DOWNLOAD_DELAY, and use pipelines to persist data.

Free to start

Learn Python with an AI tutor — free

Write and run real code in your browser, get instant help from a 24/7 AI tutor, and pick up where you left off on the web or in the app.

Courses
76
Lessons
320
Start learning freeView the full course

Frequently Asked Questions

Is the “Building a Scrapy Spider” lesson free?

Yes — the full text of “Building a Scrapy Spider” is free to read here on the web. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Python Academy course, upgrade to CoddyKit PRO. The Python Academy course includes 4 lessons in total.

What will I learn in “Building a Scrapy Spider”?

Create a Scrapy project, write a spider, and follow links automatically. You practise Python Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Python Academy?

No prior experience is required. Python Academy on CoddyKit is structured for beginners through advanced learners, so you can start here or from the beginning and move at your own pace. This is lesson 3 of 4.

How long does the “Building a Scrapy Spider” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Python Academy lesson?

Yes. Every Python Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. HTTP Requests with requests and httpx
  2. Parsing HTML with BeautifulSoup
  3. Building a Scrapy Spider
  4. Handling JavaScript and Anti-scraping Measures
← Back to Python Academy