What is llms.txt — the proposal, the format, and what the evidence says about it

llms.txt is a proposed Markdown file, served at the root of a website, that tells an AI agent what the site is and which of its pages are worth reading. It was published on 3 September 2024 by Jeremy Howard of Answer.AI, and the specification lives at llmstxt.org.

It is a proposal, not a standard. No standards body ratified it. That distinction is not pedantry: robots.txt became an internet standard as RFC 9309 in 2022, and sitemap.xml is an established protocol, and llms.txt is neither of those things.

What problem it solves

A language model has a limited context window, and a web page is mostly navigation, scripts, layout and advertising. The proposal's own reasoning is one sentence: site authors know best, and can provide a list of content that an LLM should use.

So the file is a hand-drawn directory for machines that read text rather than render pages. The long version, llms-full.txt, inlines the content instead of pointing at it.

The format

The order is fixed, and only one part of it is required:

1. An H1 with the name of the site or project. This is the only required

element — a file with a title and nothing else is valid, and useless.

1. A blockquote with a short summary. This is the first thing an agent

reads, and the line it is most likely to quote.

1. Optional prose, in paragraphs or lists, with no headings — the format

reserves headings for the title and the link sections.

1. H2 sections, each a list of links written as

- [title](url): one line about what the page answers.

Version 2, published in August 2026, added three things: the file may live at any path and cover the pages beneath it, with the most specific file winning; pages may be served as Markdown copies (page.html.md or page.md); and a page may point at its Markdown copy with rel="alternate" type="text/markdown", and at the covering manifest with rel="describedby".

The ## Optional section — conventionally the pages an agent can drop when its context is short — is a convention in version 1 and only a convention in version 2, which removed its special meaning.

What the evidence says

This is the part most guides skip, and it is the part that decides whether the file is worth an hour.

Four sources, and they agree:

Mueller's comparison is the sharpest version of it: the file is a site owner's self-declared description of their own site — the keywords meta tag, in everything but age.

Where it does earn its place is narrower and real: developer-facing and agentic tools. Coding assistants and retrieval agents can use the file to pull the right pages instead of scraping rendered HTML, and the token saving is measurable. Anthropic, Stripe, Zapier, Cloudflare and Vercel all publish one for their documentation. Anthropic's own split is the clearest example of the trade-off — roughly 8,000 tokens for the index against roughly 481,000 for the full file.

What it is not

• Not access control. Nothing obliges a crawler to read it. Whether a

named AI crawler may fetch your pages is decided in robots.txt.

• Not a sitemap. A sitemap lists every URL; this file curates. A file

with hundreds of links is a sitemap with extra steps.

• Not a ranking factor, on the evidence above.
• Not a training opt-out. It is an orientation file, not a permission

file.

The honest summary

Publish it as good housekeeping and as a courtesy to agents that do look for it. Do not file it under SEO and expect a ranking. The value is in the discipline it forces — writing one line about what each page answers, which is the same skill as writing a good meta description, and is worth having on its own.

On this platform the file is generated from the published site, so it cannot disagree with the site, and a tenant gets it without maintaining a second copy of the site's structure.

Sources

• The specification, at llmstxt.org
• The original proposal, at Answer.AI
• RFC 9309 — the Robots Exclusion Protocol
• A published example, at docs.anthropic.com