Scrapy Deep Dive: A Self-Hosted Crawler from Engine to Pipeline
Scrapy separates crawling into the Engine, Scheduler, Downloader, Spider, Item Pipeline, and middleware; it fits high-volume, rule-driven HTTP crawling where you need control over scheduling, throttling, retries, and storage.