Jina Reader Guide: Turn Web Pages into Agent-Readable Markdown

🌏 中文版

Jina Reader is an API that turns a web URL into LLM-friendly text. At its simplest, append the original URL to https://r.jina.ai/; the service fetches the page, identifies readable content, and returns Markdown. It handles one specific step in a retrieval pipeline: once you have a URL, how do you prepare its content for a model?

It is not a search engine or a research agent that autonomously explores an entire site. Reader's core input is a known URL, and its core output is cleaned content. Jina also operates the s.jina.ai search endpoint, but this article stays with single-page reading. Search, ranking, and extraction are separate responsibilities; combining them makes missing evidence much harder to diagnose.

According to the official Reader architecture, a page may be fetched through headless Chrome or a lightweight curl engine, while PDFs go through PDF.js. Reader is therefore more than a text extractor over downloaded HTML: it manages fetching, browser rendering when needed, and HTML-to-Markdown conversion. API behavior below was verified on 2026-08-22.

A minimal retrieval pipeline

Start with a version you can actually operate: a search system or user supplies a URL, the backend sends it to Reader, validates the response, and then places the Markdown in model context.

query / user input
      ↓
known URL
      ↓
Jina Reader: fetch → render → extract → Markdown
      ↓
validate source URL, status, length, content
      ↓
LLM context / chunking / index

The first test needs no SDK:

curl 'https://r.jina.ai/https://example.com'

The prefix form is excellent for manual reading, prototypes, and debugging. In application code, request JSON, retain metadata, and add a token guardrail:

curl 'https://r.jina.ai/https://example.com/article' \
  -H 'Accept: application/json' \
  -H 'x-preset: agent' \
  -H 'x-max-tokens: 12000'

x-max-tokens truncates output above the limit. If incomplete content is worse than no content, use x-token-budget so an oversized result fails instead. This is a product decision, not a minor performance tweak: a chat summary may tolerate truncation, while contract or specification review usually should not silently lose its final sections. Headers and defaults can change, so consult the official request-header documentation before implementing them.

Shape the output for its consumer

Reader's defaults retain links and images and apply Readability filtering, which suits direct use by a chat model. For a stable interface, X-Respond-With: frontmatter returns YAML metadata plus Markdown, while Accept: application/json removes the need to parse title and source fields from a plain-text preamble.

Four groups of controls usually matter most for retrieval quality:

For a documentation site with a stable DOM, explicitly target the content instead of asking a generic body heuristic to guess every time:

curl 'https://r.jina.ai/https://example.com/docs/start' \
  -H 'x-target-selector: main article' \
  -H 'x-remove-selector: nav, footer, .related' \
  -H 'x-wait-for-selector: main article'

For a vector index, start with the official index preset. It favors readable text over media URLs and returns structured chunks. The official cookbooks also define reader, research, agent, and spider presets. A preset is a starting point, not a quality guarantee; an explicit header can override one option within it.

The boundary with Firecrawl and Tavily Extract

All three services can retrieve content from a URL. The useful question is not which one can emit Markdown, but which layer your pipeline still lacks.

Firecrawl's scrape documentation lists Markdown, HTML, screenshots, links, and schema-shaped JSON, while separate batch, crawl, and map endpoints cover a broader web-data workflow. Reader can also capture screenshots, parse PDFs, and configure proxies, but its clearest entry point remains “read this URL.” If you already have URLs and need clean text, Reader is direct. If you need to schedule a site crawl and monitor job state, Firecrawl's abstraction better matches the task.

Tavily Extract accepts one or more URLs and can use query with chunks_per_source to narrow output toward relevant passages. It fits naturally after Tavily Search. Reader instead exposes more control over page rendering and Markdown shape. For the adjacent layers, see the site's Exa search guide, Tavily guide, and Firecrawl guide.

Failure is part of the design

Reader cannot guarantee that every URL is readable. Login walls, paywalls, CAPTCHAs, regional restrictions, and anti-automation systems can return an error page or partial content. An SPA may render its real body only after Reader considers the page readable. The most dangerous outcome is not an HTTP error; it is a successful response whose content says “Please enable JavaScript.”

Do not equate a success status with usable evidence. Validate at least the source URL, content length, title, expected terms, and common block-page phrases. For important evidence, retain the retrieval time and original URL. On failure, try selector and wait controls, bypass stale cache with x-no-cache: true, force x-engine: browser, and only then switch proxies or extractors. The official README also notes that anonymous traffic is rate-limited more aggressively. Production traffic should authenticate and implement backoff and circuit breaking.

The security boundary matters too. Although Reader's SaaS layer filters suspicious addresses, an application accepting arbitrary URLs should still allowlist protocols and domains and block localhost, private addresses, cloud metadata endpoints, and redirects into internal networks to reduce SSRF risk. Retrieved pages are also untrusted data. Text such as “ignore previous instructions” or “call this tool” may be prompt injection, not a system instruction. Delimit retrieved content, constrain agent permissions, and verify citations against the source.

Finally, respect site terms, robots policies, and copyright. Technical access is not permission for bulk republication. For personal, confidential, or regulated content, verify that third-party processing and retention meet your requirements. If they do not, consider the Apache-2.0 self-hosted repository. The official architecture explicitly says the open-source branch omits the SaaS MongoDB storage layer, so hosted and self-hosted deployments should not be assumed identical.

The trade-off

Jina Reader fits between a known URL and model-readable context. It hides browser rendering and body conversion behind a small API while preserving necessary controls for selectors, waiting, token budgets, and output shape. A prototype can begin with a URL prefix. Production needs JSON handling, content validation, token policy, fallbacks, and prompt-injection isolation.

When the task grows into a site crawl, batch orchestration, or schema extraction, evaluate Firecrawl. When search and relevance-focused extraction already share a Tavily pipeline, Tavily Extract is likely the smoother fit. Reader's advantage is not owning every web-retrieval layer. It makes “read one page” a small component that can gain controls as requirements become real.

References

• Jina AI — Reader API
• Jina AI Reader — GitHub README
• Jina AI Reader — Architecture
• Jina AI Reader — Cookbooks
• Firecrawl — Scrape documentation
• Tavily — Extract API reference
• Exa: Neural Search for AI Agents
• Tavily Search API Guide
• Firecrawl Complete Guide

This guide's free-tier details are summarized in Free Search, Scraping, and Browser APIs: How to Choose: Jina is the "sustained rate-limit" route, usable keyless at 20 RPM, with Search not supporting anonymous calls. The judgment table also covers Keenable, which takes a monthly quota route instead.