What is llms.txt in simple terms?

llms.txt is a markdown file at the root of a website that lists the pages and endpoints an AI model should read, one link per line, each with a short description. It works like a map written for machines: an agent that lands on the domain with no menus and no nav can fetch one predictable URL and get the curated tour instead of crawling.

Is llms.txt a real standard?

No. It is a voluntary convention from a 2024 proposal published at llmstxt.org. There is no RFC, no standards body, and no rule that says an agent must look for the file. Sites adopt it because publishing costs one static file and the agents that do check get a better map.

Do AI agents actually read llms.txt?

Some do. Reading the file has to be built into the crawler: the agent looks for llms.txt at the conventional location, same idea as checking robots.txt. When a checking agent arrives, it gets your curated link list in one fetch; when no checking agent ever comes, you have lost one static file. Discovery is not guaranteed, which is why the file is cheap to publish rather than a dependency.

← Catalog

What is llms.txt?

Published 2026-09-10. Every example below is fetchable: the live file sits at jayjex.github.io/data-vault/llms.txt.

The short answer

llms.txt is a markdown file that lives at the root of a website and lists the pages and endpoints an LLM should read, one link per line, each with a short description. An agent that lands on your domain has no menus, no navigation, and no way to know which of your pages matter. The file hands it the curated tour in one fetch.

The name comes from large language models, and the format came out of a 2024 proposal by Jeremy Howard, published at llmstxt.org. Sites adopted it voluntarily from there. Nothing enforces it, but the location is predictable (https://yourdomain.com/llms.txt) and the format is plain markdown that any tool can parse, which is enough for a convention to work.

An agent fetching our file sees this, verbatim from the live version:

# Data Vault

> Sample-first data catalog by Matchbook Labs. Free CSV/JSON samples for every dataset, machine-readable index for agents, full packs sold via Getly.

Base URL: https://jayjex.github.io/data-vault
Catalog (start here): https://jayjex.github.io/data-vault/catalog.json

Everything after those lines is link lists grouped under headings: agent docs, one section per dataset pack, and the access rules. The llms.txt structure guide annotates the full file line by line.

How an agent uses the file

The fetch chain is short. The agent resolves the domain, requests /llms.txt the way it would request robots.txt, and reads the blockquote first. That one paragraph decides relevance: if the site has nothing to do with the task, the agent stops there and spent one request finding out. If the site is relevant, the agent picks links from the sections and fetches them.

On this catalog the chain looks like this. The blockquote says sample-first data catalog, so an agent hunting for rent data reads one more line and finds catalog.json. The dataset sections carry row counts, so it can pick the HUD rent pack over 90,000-row Airbnb listings without opening a page. The sample URLs point at JSON, so the next fetch returns structured rows instead of markup. And the header carries the MCP install command, so a tool-using agent can go from one text file to querying the full released data with no human in the loop.

That is the value in one sentence: the file turns a blind crawl into a directed read. Descriptions are the part a URL list lacks, and they are what the agent reasons over.

Where it sits next to robots.txt and sitemap.xml

A site root can carry three files with three audiences. robots.txt tells crawlers what to skip. sitemap.xml lists every URL for search engines, exhaustively and without descriptions. llms.txt suggests what an LLM should read, curated, with a one-line summary per link. Keeping all three is normal because each does a job the others don't. The llms.txt vs sitemap.xml comparison shows the two files side by side on this site.

The honest status of the format

Three facts keep expectations calibrated. First, it is a convention: no RFC, no standards body, no registry. Adoption is real but partial; Anthropic's documentation site publishes one (docs.anthropic.com/llms.txt), and many sites have never heard of the format.

Second, discovery is up to each crawler. A tool only reads the file if its developers built that check in. When a checking agent arrives, it gets your map in one fetch. When none ever comes, you have lost one static file.

Third, the mechanism differs from robots.txt in direction. robots.txt is exclusion: it tells compliant crawlers where they may not go. llms.txt is an invitation: it suggests where an automated reader should go. Nothing forces an agent to follow either one, but publishing the invitation costs nothing and removes guesswork for the tools that look.

Three misconceptions, corrected

"It's a standard now." There is no RFC and no governing body. It is a proposal from 2024 that enough sites adopted to make the location and format predictable. Treat it that way: free to publish, cheap to keep current, safe to rely on as a courtesy channel rather than a guarantee.

"It replaces my sitemap." The audiences are different. Search engines want the exhaustive URL list; LLMs want the curated shortlist with context. Deleting your sitemap for an llms.txt trades a working channel for an experimental one. Run both, as this site does.

"More links make the file better." Curation is the point. A file that lists every page is a second sitemap with worse tooling behind it, and it spends the agent's context window on pages that never mattered. The how-to guide covers the editorial decisions: which pages earn a slot, what each one does in a line.

FAQ

What is llms.txt in simple terms?: llms.txt is a markdown file at the root of a website that lists the pages and endpoints an AI model should read, one link per line, each with a short description. It works like a map written for machines: an agent that lands on the domain with no menus and no nav can fetch one predictable URL and get the curated tour instead of crawling.
Is llms.txt a real standard?: No. It is a voluntary convention from a 2024 proposal published at llmstxt.org. There is no RFC, no standards body, and no rule that says an agent must look for the file. Sites adopt it because publishing costs one static file and the agents that do check get a better map.
Do AI agents actually read llms.txt?: Some do. Reading the file has to be built into the crawler: the agent looks for llms.txt at the conventional location, the same way it checks robots.txt. When a checking agent arrives, it gets your curated link list in one fetch; when no checking agent ever comes, you have lost one static file. Discovery is not guaranteed, which is why the file is cheap to publish rather than a dependency.

Read ours, then go deeper

Fetch the live file to see the whole format in use. The llms.txt guide for AI agents covers the minimal structure and annotates our full file section by section. The how-to walks writing your own in six steps, no generator needed. For how the file differs from the sitemap next to it, read the side-by-side comparison, and the agent access page shows what our links route to.