Python Academy · Lesson

Building a Scrapy Spider

Create a Scrapy project, write a spider, and follow links automatically.

Building a Scrapy Spider is a free Python Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Python Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is Scrapy?

Scrapy is a full-featured web scraping framework with built-in asynchronous crawling, item pipelines, middlewares, and exporters.

# pip install scrapy

# Create a project:
# scrapy startproject myspider

# myspider/
#   scrapy.cfg
#   myspider/
#     settings.py
#     spiders/
#       quotes_spider.py

Creating a Spider

Subclass scrapy.Spider, set name and start_urls, then implement parse().

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
            }

CSS and XPath Selectors

Use response.css("selector") or response.xpath("//xpath") to extract data. .get() returns the first match; .getall() returns all.

def parse(self, response):
    # CSS:
    titles = response.css("h2.title::text").getall()

    # XPath:
    links = response.xpath("//a/@href").getall()

    yield {"titles": titles, "links": links}

Following Links

Yield a scrapy.Request to follow a link and parse the next page.

import scrapy

class Spider(scrapy.Spider):
    name = "crawler"
    start_urls = ["https://quotes.toscrape.com"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {"text": quote.css("span.text::text").get()}

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Items and ItemLoaders

Define Item classes to structure scraped data, and use ItemLoader for cleaning and transforming fields.

import scrapy

class ProductItem(scrapy.Item):
    name  = scrapy.Field()
    price = scrapy.Field()
    url   = scrapy.Field()

def parse(self, response):
    item = ProductItem()
    item["name"]  = response.css("h1::text").get(strip=True)
    item["price"] = response.css(".price::text").re_first(r"[\d.]+")
    item["url"]   = response.url
    yield item

Running the Spider

Run from the project directory. Export to JSON, CSV, or a custom pipeline.

# Run and print to console:
# scrapy crawl quotes

# Save to JSON:
# scrapy crawl quotes -o quotes.json

# Save to CSV:
# scrapy crawl quotes -o quotes.csv

# Scrapy shell for interactive testing:
# scrapy shell "https://quotes.toscrape.com"

Settings: DOWNLOAD_DELAY and CONCURRENT_REQUESTS

Be polite: add delays and limit concurrent requests to avoid overloading servers.

# settings.py
DOWNLOAD_DELAY = 1          # 1 second between requests
CONCURRENT_REQUESTS = 8     # max 8 at a time
CONCURRENT_REQUESTS_PER_DOMAIN = 2
ROBOTSTXT_OBEY = True       # respect robots.txt

Middlewares

Middlewares intercept requests and responses. Use built-in ones for user-agent rotation, retries, and cookies.

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
    "myspider.middlewares.RotateUserAgentMiddleware": 400,
}

Item Pipelines

Pipelines process items after scraping: clean data, validate, deduplicate, and save to a database.

# pipelines.py
class CleanPipeline:
    def process_item(self, item, spider):
        item["name"] = item["name"].strip()
        return item

class DBPipeline:
    def open_spider(self, spider):
        self.conn = connect_db()

    def process_item(self, item, spider):
        self.conn.insert(dict(item))
        return item

Handling Login

Use Scrapy's FormRequest to submit a login form and persist the session cookie.

import scrapy

class AuthSpider(scrapy.Spider):
    name = "auth"
    login_url = "https://example.com/login"

    def start_requests(self):
        yield scrapy.Request(self.login_url, self.login)

    def login(self, response):
        yield scrapy.FormRequest.from_response(
            response,
            formdata={"user": "me", "pass": "secret"},
            callback=self.parse
        )

scrapy-playwright for JS Pages

The scrapy-playwright plugin integrates Playwright into Scrapy for JavaScript-rendered pages.

# pip install scrapy-playwright
# playwright install

# settings.py:
# DOWNLOAD_HANDLERS = {
#     "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
#     "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
# }

# In spider:
# yield scrapy.Request(url, meta={"playwright": True})

Quick Check

What does response.follow(href, callback) do in a Scrapy spider?

Recap

Scrapy spiders subclass scrapy.Spider, declare start_urls, and yield items or further Requests from parse(). Use CSS/XPath selectors, be polite with DOWNLOAD_DELAY, and use pipelines to persist data.

Frequently asked questions

Is the “Building a Scrapy Spider” lesson free?

Yes — the full text of “Building a Scrapy Spider” is free to read here on the web, and the Python Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Python Academy course, upgrade to CoddyKit PRO.

What will I learn in “Building a Scrapy Spider”?

Create a Scrapy project, write a spider, and follow links automatically. You practise Python Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Python Academy?

No prior experience is required. Python Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Building a Scrapy Spider” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Python Academy lesson?

Yes. Every Python Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

HTTP Requests with requests and httpx
Parsing HTML with BeautifulSoup
Building a Scrapy Spider
Handling JavaScript and Anti-scraping Measures

← Back to Python Academy