基于队列的任务分发
学习 Redis 和 Celery 等消息队列如何将 URL 发现与抓取解耦，从而将抓取任务扩展到多个工作进程。
基于队列的任务分发 是 CoddyKit 上的免费 Web Scraping & Bots 课时。 这是第 4 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Web Scraping & Bots 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Web Scraping & Bots 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
The Scaling Bottleneck
A single-process scraper is limited by one machine's CPU and network. To scale, you split work across many workers running in parallel, possibly on different servers.
A task queue is the glue that distributes work safely.
Producers and Consumers
The queue pattern has two roles:
- Producers discover URLs and push tasks onto the queue.
- Consumers (workers) pull tasks and fetch the pages.
Decoupling them lets each side scale independently.
A Redis-Backed Queue
Redis lists make a simple, fast queue. Producers LPUSH URLs; workers BRPOP them, blocking until work is available.
import redis
r = redis.Redis()
# producer
r.lpush('urls', 'https://site.com/page1')
# worker
_, url = r.brpop('urls')
print('processing', url)Why Not Share a Python List
An in-memory list only works within one process. A Redis queue is shared across processes and machines, persists if a worker crashes, and handles concurrency atomically.
Introducing Celery
Celery is a full task framework built on a broker like Redis. You define tasks as functions and call them asynchronously; workers pick them up automatically.
from celery import Celery
app = Celery('scraper', broker='redis://localhost:6379/0')
@app.task
def scrape(url):
return fetch_and_parse(url)Dispatching Tasks
Calling .delay() enqueues the task and returns immediately. Workers running celery worker consume and execute them in parallel.
for url in discovered_urls:
scrape.delay(url)Retries and Failures
Celery can automatically retry failed tasks with backoff, so a transient network error does not lose a URL.
@app.task(bind=True, max_retries=3, default_retry_delay=10)
def scrape(self, url):
try:
return fetch_and_parse(url)
except ConnectionError as e:
raise self.retry(exc=e)Deduplicating URLs
In distributed crawling the same URL can be discovered twice. Use a Redis set as a 'seen' filter so each page is fetched once.
if r.sadd('seen', url):
scrape.delay(url) # sadd returns 1 only if newly addedBackpressure and Concurrency
Tune worker concurrency to match target-site politeness and your bandwidth. Too many workers overwhelm the site; too few leave the queue backed up. Monitor queue length to find balance.
# start 4 worker processes
// celery -A scraper worker --concurrency=4Results and Storage
Workers should write parsed data to a shared store (a database or object storage), not return it through the queue. The queue carries tasks; the datastore holds results.
Priority Queues
Not all URLs are equal. Route urgent tasks (a category index that unlocks many child pages) to a high-priority queue so workers handle them before low-value pages.
scrape.apply_async(args=[url], priority=9) # higher runs soonerQuick Check
Test your understanding of queue-based distribution.
Recap
You learned to scale scraping with task queues: the producer/consumer pattern, Redis-backed queues, Celery tasks with .delay(), automatic retries, URL deduplication with Redis sets, tuning concurrency, and writing results to shared storage.
常见问题解答
「基于队列的任务分发」课时是免费的吗？
是的 — 「基于队列的任务分发」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Web Scraping & Bots 课程的其余内容，请升级到 CoddyKit PRO。 Web Scraping & Bots 课程共包含 4 节课。
「基于队列的任务分发」这节课中我会学到什么？
学习 Redis 和 Celery 等消息队列如何将 URL 发现与抓取解耦，从而将抓取任务扩展到多个工作进程。 你通过在浏览器中直接运行的动手代码来练习 Web Scraping & Bots，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 Web Scraping & Bots 需要有经验吗？
无需任何先前经验。CoddyKit 上的 Web Scraping & Bots 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 4 节课，共 4 节。
「基于队列的任务分发」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 Web Scraping & Bots 课中编写并运行代码吗？
能。每节 Web Scraping & Bots 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。
此课程中的所有课时
- 使用 Scrapy 进行分布式抓取
- 用于网络抓取的云函数
- 监控与日志记录
- 基于队列的任务分发