Skip to content
All tags

#web-scraping

3 posts
ai deep-dive

AI Web Scraping Tools Landscape: A Selection Guide for 34 Open-Source Projects

From MarkItDown (169k stars, MIT) to curl-impersonate (7k stars), a survey of 34 open-source tools for feeding data to AI. Categorized along five axes: whole-site crawling, AI browser agents, document conversion, smart extraction, and anti-detection infrastructure. The key to selection isn't which tool is best — it's scenario matching.

#web-scraping#ai-agent#browser-automation#llm#open-source
ai guide

AI-Ready Content: The Complete Guide to Making Your Website an AI-Readable Data Source

In 2025-2026, websites need to be readable not just by humans but by AI. From llms.txt and Schema Markup to GEO and RAG ingestion pipelines, this post maps out the complete technical landscape for turning your website into an AI-consumable data source.

#ai-ready-content#llms-txt#geo#rag#web-scraping#structured-data#mcp#seo#rsl#webmcp
tech guide

Complete Guide to Bypassing Cloudflare Anti-Bot for AI Agents: From Debugging to Building an MCP Server

Standard Playwright gets blocked by Cloudflare. Both playwright-extra + stealth and nodriver can bypass it. The final step is wrapping the solution into an MCP server so AI agents can use it automatically.

#cloudflare#anti-bot#playwright#nodriver#stealth#mcp#ai-agent#web-scraping