- It's Not Just "Download the Page"
- Who Decides How to Scrape: Five Approaches
- Whole-Site Crawling: Firecrawl Leads, but Watch the License
- AI Browser Agents: Let AI Operate Like a Human
- Document Conversion: No Scraping, Just Format Translation
- Smart Extraction: Self-Healing Selectors
- Anti-Detection & Infrastructure: The Foundation for Reliable Data Access
- Quick Reference Table
- Bottom Line
- Changelog
- References
🌏 中文版
"Scraping data for AI" has spawned an entire tool ecosystem. From MarkItDown at 169k stars to various niche tools in the low thousands, at least 34 active projects on GitHub address this problem. This article categorizes them along five axes — whole-site crawling, AI browser agents, document conversion, smart extraction, and anti-detection infrastructure — to help you pick the right tool for your scenario instead of building a scraper from scratch.
It's Not Just "Download the Page"
Scraping data for AI involves three independent core problems, and no single tool solves all of them:
- Format conversion: How to turn unstructured web pages or documents into LLM-digestible formats (Markdown, structured JSON)
- Interactive crawling: How to handle pages requiring login, JS rendering, or dynamic loading
- Reliable access: How to get data consistently in an environment of increasingly aggressive anti-bot measures
Who Decides How to Scrape: Five Approaches
The biggest divergence among tools is "who's the decision maker":
| Approach | Representative Tools | Decision Maker | Trade-off |
|---|---|---|---|
| Rule-driven | Scrapy, Crawlee | Developer writes selectors | High maintenance — breaks on redesign |
| AI-driven (DOM) | ScrapeGraphAI, Stagehand | LLM reads DOM | Token cost, latency |
| AI-driven (vision) | Skyvern, Browser-Use | LLM reads screenshots | Slowest, most expensive, but cross-platform |
| Adaptive | Scrapling, AgentQL | Smart selectors self-repair | No AI cost, but learning curve |
| Format conversion | MarkItDown, MinerU, Marker | Doesn't scrape — only converts | Needs an upstream crawler |
The five categories below follow these approaches.
Whole-Site Crawling: Firecrawl Leads, but Watch the License
The top pick is Firecrawl (155k stars, AGPL-3.0) — the most feature-complete option with built-in JS rendering, Markdown output, and sitemap scanning. Its API is designed specifically for LLM input scenarios. The catch is the AGPL license: if you use it to provide a network service, your integration code must also be open-sourced.
If the license is a deal-breaker, Crawl4AI (75k stars, Apache-2.0) is the closest alternative — Python-based, lighter weight, and growing fast.
For million-page scale, use Scrapy (63k stars, BSD-3) — the veteran Python framework with a mature distributed architecture, though you write your own parsers. JS/TS teams should look at Crawlee (25k stars, Apache-2.0) by Apify, with clean APIs and Playwright/Cheerio support. Non-engineers can try Maxun (~17k stars), a no-code interface for marking elements to scrape directly in the browser.
AI Browser Agents: Let AI Operate Like a Human
These tools let AI operate a browser autonomously — clicking, filling forms, scrolling, taking screenshots, all self-directed.
Browser-Use (106k stars, MIT) has the largest community. It's a Python autonomous agent loop where every step relies on LLM reasoning — ideal for "let AI complete tasks on the web" scenarios. Stagehand (24k stars, MIT) has the cleanest API — three primitives (act / extract / observe) cover both interaction and data extraction, built on Playwright in TypeScript, better for stable automation scripts. Skyvern (23k stars, AGPL-3.0) takes a vision-first approach — no DOM parsing, pure screenshot-based decisions, best cross-platform capability but slowest per step.
Browser-MCP (~7k stars) exposes browser operations as MCP tools, making it easy to plug into Claude or LLM agent workflows.
For a deep dive into the pure-vision approach, see our Midscene.js analysis — it takes the extreme stance of "screenshots only, no DOM" and even removed its DOM action mode in v1.0.
Document Conversion: No Scraping, Just Format Translation
These tools don't scrape — they convert PDF / Office / HTML into LLM-friendly formats.
MarkItDown (169k stars, MIT) by Microsoft supports the widest range of formats — PDF, Word, Excel, PowerPoint, HTML, and images, all to Markdown. It's the highest-starred project in this entire space. MinerU (76k stars) excels at table and math formula extraction — the go-to for academic PDFs. Marker (38k stars, Apache-2.0) is fast with low GPU requirements, good for batch conversion. Docling (64k stars, MIT) from IBM Research emphasizes structured output (JSON schema), ideal for scenarios requiring precise document structure preservation.
anydoc (746 stars, MIT, queried 2026-08-06) is a Rust library from Firecrawl, open-sourced on 2026-08-03, taking a different route from all of the above: office documents only, no OCR at all, but all 14 formats covered (including legacy .doc / .ppt / .xls) at a 4.7ms median. Note that its license differs from the Firecrawl main project — the main project is AGPL-3.0, anydoc is MIT, so commercial integration carries no copyleft concerns. See anydoc: 14 Office Formats to Markdown for a full comparison.
Lightweight options: Trafilatura (~6k stars) specializes in "extract body text from web pages, filter ads" — stable and reliable for preprocessing. Jina Reader (12k stars, Apache-2.0) requires zero setup — prepend r.jina.ai/ to any URL to get Markdown. Readability (~9k stars) is the engine behind Firefox's Reader Mode, often embedded as a preprocessing step in other tools.
Smart Extraction: Self-Healing Selectors
Rule-driven scrapers break on website redesigns. These tools use AI or adaptive mechanisms for more resilient extraction.
Scrapling (71k stars, BSD-3) uses adaptive selectors — no LLM involved, just smart algorithms that automatically repair broken selectors after site redesigns. Fast and token-free. ScrapeGraphAI (29k stars, MIT) takes a different path: describe what data you want in natural language, and it uses an LLM to build the scraping pipeline automatically — great for one-off extraction tasks. AutoScraper (8k stars, MIT) is even simpler — give it a sample page and the data you want, and it learns the selectors itself.
AgentQL (~1k stars) replaces CSS/XPath with semantic queries, Parsera is a lightweight LLM extraction library, and ferret (~6k stars, Go) offers a declarative extraction language.
Anti-Detection & Infrastructure: The Foundation for Reliable Data Access
curl-impersonate (7k stars, MIT) spoofs TLS fingerprints so HTTP requests look like they're from a real browser. CloakBrowser (~29k stars) is a stealth Chromium that can drop-in replace Playwright's browser instance. botasaurus (~6k stars) is a Python anti-detection scraping framework, and SeleniumBase (~13k stars) is Selenium on steroids with built-in stealth mode.
changedetection.io (~33k stars) does something different — it monitors web page changes and notifies you, useful for tracking prices, inventory, or policy updates. scrcpy (146k stars, Apache-2.0) isn't strictly a scraper but an Android screen mirroring tool, useful when you need to extract data from mobile apps. brightdata-mcp (~3k stars) is a commercial-grade MCP server for AI agents to access data through Bright Data's infrastructure.
For more on anti-detection techniques, see our guide to bypassing Cloudflare anti-bot (in Chinese) comparing nodriver / stealth / camoufox. For a practical example of connecting scrapers to MCP, see turning a scraper script into an MCP Server (in Chinese).
Quick Reference Table
Major tools verified via GitHub API (queried 2026-07-24), sorted by stars:
| Tool | Stars | License | Language | Purpose |
|---|---|---|---|---|
| MarkItDown | 169k | MIT | Python | Document → Markdown |
| Firecrawl | 155k | AGPL-3.0 | TS | Whole-site crawl + LLM output |
| scrcpy | 146k | Apache-2.0 | C | Android screen mirroring |
| Browser-Use | 106k | MIT | Python | AI browser agent |
| MinerU | 76k | — | Python | PDF table/formula extraction |
| Crawl4AI | 75k | Apache-2.0 | Python | Lightweight whole-site crawl |
| Scrapling | 71k | BSD-3 | Python | Adaptive selectors |
| Docling | 64k | MIT | Python | Structured document conversion |
| Scrapy | 63k | BSD-3 | Python | Large-scale crawling framework |
| Marker | 38k | Apache-2.0 | Python | Fast PDF conversion |
| ScrapeGraphAI | 29k | MIT | Python | Natural language → scraper |
| Crawlee | 25k | Apache-2.0 | TS | JS/TS crawling framework |
| Stagehand | 24k | MIT | TS | Clean-API browser agent |
| Skyvern | 23k | AGPL-3.0 | Python | Vision-first browser agent |
| Jina Reader | 12k | Apache-2.0 | TS | URL → Markdown |
| AutoScraper | 8k | MIT | Python | Example-driven extraction |
| curl-impersonate | 7k | MIT | C | TLS fingerprint spoofing |
An additional 17 tools — including changedetection.io, CloakBrowser, Maxun, SeleniumBase, Readability, Browser-MCP, Trafilatura, ferret, botasaurus, AnyCrawl, Markdowner, CyberScraper-2077, brightdata-mcp, webclaw, Parsera, AgentQL, and Craw4LLM — are documented in the research notes, mostly in the 1k–10k star range.
Bottom Line
Tools in this space distribute along two axes: "rules vs. AI" and "general vs. specialized." The 2024–2025 trend is clear: AI-driven scrapers (ScrapeGraphAI, Browser-Use, Stagehand) and document-to-LLM-format converters (MinerU, Marker, Docling) are growing explosively. But rule-driven veterans (Scrapy, Crawlee) remain irreplaceable at million-page scale.
The key to selection isn't "which is best" — it's scenario matching:
- Markdown output + don't want to deal with JS rendering → Firecrawl (watch AGPL) or Crawl4AI
- Login / complex interaction → Browser-Use or Stagehand
- PDF / Office conversion → MarkItDown (general) or MinerU (academic PDFs)
- Site keeps redesigning, selectors keep breaking → Scrapling
- Blocked by Cloudflare → curl-impersonate + Cloudflare bypass guide (in Chinese)
Changelog
- 2026-08-06: Added anydoc to the "Document Conversion" section (Firecrawl's Rust conversion library — 14/14 formats, 4.7ms median, 746 stars, MIT licensed), noting how its license differs from the AGPL-3.0 Firecrawl main project. For the full selection logic on this layer, see the document parsing series. Star counts for the other tools in this post remain as queried on 2026-07-24 and were not re-verified.
References
- Firecrawl (GitHub)
- Crawl4AI (GitHub)
- Browser-Use (GitHub)
- Crawlee (GitHub)
- Scrapy (GitHub)
- MarkItDown (GitHub)
- Scrapling (GitHub)
- ScrapeGraphAI (GitHub)
- Stagehand (GitHub)
- Skyvern (GitHub)
- MinerU (GitHub)
- Marker (GitHub)
- Docling (GitHub)
- anydoc (GitHub)
- Jina Reader (GitHub)
- Trafilatura (GitHub)
- AutoScraper (GitHub)
- curl-impersonate (GitHub)
- changedetection.io (GitHub)
- scrcpy (GitHub)
- Midscene.js: Vision-First UI Automation (on this site, in Chinese)
- Bypassing Cloudflare Anti-Bot Guide (on this site, in Chinese)
- Turning a Scraper into an MCP Server (on this site, in Chinese)
Loading...