Web Extraction Quality Benchmark: Crawl4AI, Firecrawl, Jina Reader, and Readability

🌏 中文版

Published benchmark specification. The 20 URLs, output contract, scoring rubric, and failure labels are fixed. However, the environment on 2026-08-22 has no FIRECRAWL_API_KEY, and the four adapters have not completed one version-locked raw run. No raw artifacts means no ranking; the results table should wait for that run.

After search finds a URL, the next step is not to send raw HTML to an LLM. It is to extract the body reliably. Crawl4AI, Firecrawl, Jina Reader, and Mozilla Readability all produce cleaner content, but they differ in rendering, body detection, Markdown structure, metadata, and cost.

This article does not repeat the 34-tool crawler landscape, and it does not compare vendor demo pages. It asks one question: given exactly the same URLs, which path retains the information needed to answer a question, and which merely produces clean-looking text?

These are four different kinds of pipeline

Readability does not fetch pages or convert HTML to Markdown. The benchmark uses one Playwright fetcher to obtain the final DOM before Readability selects the article body; otherwise, it would compare HTTP clients rather than extraction. Mozilla also warns that Readability does not sanitize output, so extracted HTML still needs a sanitizer before rendering.

Crawl4AI separates raw_markdown and fit_markdown; fit output passes through a content filter. To avoid treating query-specific pruning as baseline extraction, round one compares default cleaned Markdown. Fit Markdown becomes a separate configuration in round two.

Fix 20 URLs before seeing results

The corpus uses public, anonymously accessible pages whose important structures can be labeled manually. Every case stores retrieved_at, final URL, HTTP status, content hash, and a human-authored ground-truth manifest.

The list is versioned, not immortal. If a page disappears or is redesigned, retain the old hash and publish a new corpus version. Never silently replace a URL and join the two runs into one trend.

Every adapter returns one contract

Raw responses are archived in full, then normalized into one JSON shape:

{
  "case_id": "docs-01",
  "adapter": "crawl4ai-default",
  "adapter_version": "0.9.x",
  "requested_url": "https://playwright.dev/docs/auth",
  "final_url": "https://playwright.dev/docs/auth",
  "retrieved_at": "2026-08-22T00:00:00Z",
  "status": "success",
  "latency_ms": null,
  "cost_usd": null,
  "title": null,
  "author": null,
  "published_at": null,
  "markdown": null,
  "links": [],
  "raw_artifact": "raw/docs-01/crawl4ai-default.json",
  "error": null
}

null and 0 are not interchangeable. Missing billing data is null, not zero cost. A missing author is null, not an empty string. Timeout, blocked, fetch error, parse empty, and partial content stay separate; otherwise, a login page can inflate the success rate.

Score required evidence before visual cleanliness

Ground truth stores required_spans, required_structures, and known_noise, not one complete canonical rendering. Reviewers are blinded to the adapter and score six dimensions separately:

1. Body completeness: recall of required spans.
2. Noise ratio: navigation, cookie banners, recommendations, and footer content.
3. Structure retention: headings, tables, code blocks, lists, and links remain recognizable.
4. Metadata: title, author, publication time, and canonical URL are correct.
5. Traceability: links retain their original targets and content retains section anchors or locators.
6. Efficiency: p50/p95 latency, request count, paid cost, and output bytes.

Do not collapse these into one score. Missing half the body should not cancel against low latency. Regression gates should be separate: required-span recall cannot fall, code retention cannot fall, and p95 latency cannot exceed a pre-agreed tolerance.

Four ways to invalidate the benchmark

Calling fetch success extraction success

HTTP 200 may contain a consent page, login page, bot challenge, or app shell. Verify title, required spans, and minimum body content before marking success.

Fetching each pipeline at a different time

Home pages, docs, and SPAs change. Run all four adapters for each case in one short window and retain a source snapshot.

Using character count as completeness

Long output may be navigation and footer noise. Short output may be precise. Score required spans and noise separately.

Trusting extraction output

HTML, Markdown, and metadata can all contain prompt injection or malicious markup. Mozilla explicitly says Readability does not sanitize. Markdown from a managed service is not automatically safe either. Sanitize, retain provenance, and separate content from tool instructions.

The only current conclusion is the test design

Official capabilities can be described; winners cannot. Crawl4AI provides a local browser and configurable Markdown/filter pipeline. Firecrawl provides managed scrape formats. Jina Reader offers a low-friction URL-to-text endpoint. Readability is the lightest local article heuristic, while fetch, render, Markdown conversion, and sanitization remain the operator's responsibility.

This article should add a results table only after all four adapters finish the same corpus version, raw artifacts are archived, and blinded annotations are complete. Anything less is a comparison of product descriptions, not an experiment.

References

• Crawl4AI — Markdown Generation
• Firecrawl — Scrape
• Jina AI — Reader API
• Mozilla Readability — README and API
• Playwright — Browser automation documentation