Building a Scrapy Spider
Create a Scrapy project, write a spider, and follow links automatically.
What Is Scrapy?
Scrapy is a full-featured web scraping framework with built-in asynchronous crawling, item pipelines, middlewares, and exporters.
# pip install scrapy
# Create a project:
# scrapy startproject myspider
# myspider/
# scrapy.cfg
# myspider/
# settings.py
# spiders/
# quotes_spider.pyCreating a Spider
Subclass scrapy.Spider, set name and start_urls, then implement parse().
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com"]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
}