llms.txt: The Copy of Your Docs Written for Machines

🌏 中文版

One criterion keeps recurring in this series: pick libraries whose documentation machines can read. This post covers the vehicle for that criterion. llms.txt resembles the file its name winks at — robots.txt, a plain-text convention at the site root — but points the opposite way: robots.txt tells crawlers what not to look at; llms.txt tells language models what they should read, and where.

Origin and spec

llms.txt was proposed by Jeremy Howard of fast.ai on September 3, 2024. The official one-liner: "A proposal to standardise on using an /llms.txt file to provide information to help agents use a website." The spec is actively maintained — as of August 2026 it stands at v2 — and its own opening notes how the times changed: when first written in 2024, "agents routinely using websites" was largely a prediction; today it is routine (coding agents fetching a library's docs to get an API call right, search-equipped assistants reading pages to answer product questions).

The format is deliberately simple: structured Markdown with an H1 site name, a blockquote one-line summary, then H2 sections of link lists to the pages that matter, each with a short description. It addresses three practical problems LLMs have with websites: the whole site doesn't fit in a context window, HTML is polluted with navigation/ads/JS noise, and the model can't tell which pages are authoritative. llms.txt is the site answering, "if the model can only read ten pages, these are the ten."

In practice a companion convention has grown alongside it: llms-full.txt — not an index but the entire documentation concatenated into one large Markdown file, suited to stuffing into a big context window or serving as RAG corpus. Anthropic (docs.anthropic.com), Cloudflare (developers.cloudflare.com), the AI SDK, and Zod all serve both files (each hand-verified reachable, August 2026).

Who ships it: a six-site test

While writing the selection overview, I tested six frontend docs sites (August 2026, HTTP status):

The difference is concrete. Say your coding agent needs to write file-based routing with TanStack Router — a library under-represented in training data (a weakness the series' third post covers), so bare-handed output drifts into React Router idioms. With llms.txt the workflow is: fetch the index → locate the file-based-routing page → fetch it → write against the current API. Without it, the agent scrapes HTML and hopes, or writes from stale memory. llms.txt is a niche library's main weapon against the training-data deficit — which also explains the adoption pattern: the most eager adopters are young ecosystems like TanStack and the AI SDK, while React Router, with the largest installed base, hasn't moved. When the corpus already knows you, the pain is smaller.

The honest part: disputes and limits

llms.txt is not uncontroversial; two things are worth remembering. First, it is not a standard backed by the search giants: Google's John Mueller publicly compared it to the long-abandoned keywords meta tag — self-declared signals can't be used for ranking — and noted that no major AI service has said it uses the file, with server logs showing they don't even check for it (as reported by Search Engine Journal). Mainstream AI search products are opaque about crawling; you cannot confirm whether ChatGPT search or Perplexity reads yours. Its most solid consumers today are not AI search engines but coding agents and docs tooling — workflows where a developer explicitly feeds llms.txt into context. Second, it carries a maintenance cost: an index that drifts from the docs turns from help into misdirection — the same "content rots" problem this site keeps re-verifying, except this time it's the machine-facing copy that rots.

So the sane expectation: treat it as a cheap investment in (and signal of) agent-friendliness, not SEO-style ranking magic.

What it means for this blog

The natural question at this point: should content sites follow? This site's AEO/GEO series covered the KDD 2024 GEO research — AI search preferentially cites content with concrete numbers and inline-attributed sources. llms.txt is another lever on the same goal: telling models directly which articles are the authoritative ones. For docs sites it's near table stakes; for content sites it's a cheap, sensible add — one Markdown index listing the articles most worth citing.

Overall

llms.txt bets on a premise that is coming true: the readership of websites has permanently changed, and the machine share only grows. Technically it is almost naive — one hand-written Markdown index — but it is positioned precisely: on the question of "what should the model read," it gives site owners their first formal say. For library authors it's a cheap must-do; for anyone choosing libraries it's an honest signal — a project that maintains llms.txt probably also cares about the agent-era developer experience; for content authors it's the cheapest tool in the GEO toolbox.

References

• The llms.txt spec (llmstxt.org)
• TanStack llms.txt (example)
• Anthropic llms-full.txt (example)
• GEO paper (KDD 2024, arXiv 2311.09735)
• Google Says LLMs.Txt Comparable To Keywords Meta Tag (Search Engine Journal)
• On this site: Choosing a React Stack in the AI Era, AEO/GEO Strategy for the AI Search Era