How to create an llms.txt file, no generator required

The quick answer

An llms.txt file is one markdown file at your site root that tells an LLM which pages and endpoints to read. Building one takes a text editor and the six steps below. What generators sell is the mechanical half: crawling your pages and assembling a link list. The half that makes the file work is editorial, because only you know which pages an agent actually needs and what each one does in a line. Write those parts yourself, run the link check, ship.

What a generator can and cannot do

Tools that generate llms.txt crawl a sitemap or your HTML and draft the link list for you. That part is real work saved, on a site with hundreds of URLs. What the crawler cannot do is rank. The file earns its keep because it is curated: the JSON index sits above the press page, the licensing rules ride along in the same payload, and every link carries one description written for a reader with no context. A generated file that lists everything is a second sitemap, which is the one thing the format is not for. Our llms.txt vs sitemap.xml comparison shows the difference on a site that runs both files, and the llms.txt guide covers the format itself: status, minimal structure, and why a data platform publishes one.

Split the work: hand-write the file with the recipe below. If you already run a generator, use its output as a link inventory to edit down, not as the file itself.

Six steps to your own file

1. Inventory what an agent can use

Make two lists. Pages worth an agent's context: docs, guides, reference material, policies that affect automated use. Machine-readable endpoints: JSON indexes, APIs, downloads, feeds. Leave out pages that only matter to humans, like marketing and testimonials. For this catalog the inventory came to a handful of entry points (catalog index, agent docs, MCP install line) plus one section per dataset pack.

2. Write the H1 and the blockquote

Site name as the H1. Below it, one blockquote paragraph saying what the site offers and to whom. Ask one question while you write it: could an agent decide whether your site is relevant from that paragraph alone? If yes, it stops reading headers and starts fetching your links.

3. Group links under H2 headings

## Docs for pages, ## Endpoints for anything machine-readable, plus any section that fits your content, like ## Datasets. Every link is markdown, absolute, with a one-line description after it:

## Endpoints

- [catalog.json](https://example.com/catalog.json): machine-readable index of every item
- [openapi.yaml](https://example.com/openapi.yaml): full API spec

4. Put entry points before the first H2

Anything a tool needs without a section belongs at the top: base URL, the main index, install commands. Our file carries the catalog URL, the agent docs path, and the MCP install line above the first heading, so a reader that never scrolls still gets the entry points.

5. Save it at the root as llms.txt

The file lives at https://yourdomain.com/llms.txt, same directory as robots.txt and sitemap.xml. On a static host that is one file in the repository root. The llmstxt.org convention also defines an optional llms-full.txt with expanded content, but the short file is the orientation point, so build it first.

6. Verify links, then read it cold

First check that every link resolves, because the file gets read far from your site:

curl -s https://yourdomain.com/llms.txt | grep -oE 'https://[^ )"]+' | sort -u

Confirm each URL in that list returns what you expect; the guide has a loop that prints the HTTP status next to every URL. Second, open the file cold, one fetch, no site context, and read it as an agent would. If the blockquote alone does not tell you what the site offers, rewrite the blockquote.

The live example: this catalog's file

Our file runs 61 lines and covers six dataset packs plus an MCP server. Header and one section, trimmed from the live file:

# Data Vault

> Sample-first data catalog by Matchbook Labs. Free CSV/JSON samples for every dataset, machine-readable index for agents, full packs sold via Getly.

Base URL: https://jayjex.github.io/data-vault
Catalog (start here): https://jayjex.github.io/data-vault/catalog.json
Agent docs: https://jayjex.github.io/data-vault/agents/
MCP server: npx --allow-git=all -y github:jayjex/dataset-mcp

## Datasets

### HUD Fair Market Rents FY2026 (hud-fmr-2026)
FY2026 fair market rents for 51,895 ZIP codes, 3,229 counties, and 52 states. Five bedroom sizes per row. Niche: gov-data. Stats: {"rows": 51895, "tables": 3, "columns": 11}.
- Sample JSON: https://jayjex.github.io/data-vault/data/hud-fmr-2026/sample.json
- Page: https://jayjex.github.io/data-vault/datasets/hud-fmr-2026.html

What each choice buys: row counts in every section let an agent pick a pack by size without opening an HTML page; sample URLs point at JSON, so the next fetch is structured data instead of markup; the access rules at the bottom settle license and attribution questions in the same payload. The llms.txt guide annotates the full file section by section.

Mistakes that sink a new file

• Relative links. Agents may read the file without loading your site, so /docs breaks where https://yourdomain.com/docs works.
• Bare URLs with no description. The one-liner is the only thing telling the agent what the fetch returns.
• Listing every page. That is the sitemap's job. Curation is the point; if a page would not help an automated reader, cut it.
• Letting it go stale. When you ship a notable page or endpoint, add its line in the same commit.
• Burying the endpoints. The machine-readable links are the highest-value lines for an agent; keep them near the top or in their own clearly named section.

FAQ

Do I need an llms.txt generator?

No. The format is plain markdown with no required tooling. A generator that crawls your site can save you typing the link list, but it cannot make the two decisions that make the file useful: which pages earn a slot, and what each one does in one line. Write the file by hand with the steps above; if you run a generator, treat its output as a link inventory to edit down.

How long should an llms.txt file be?

The convention sets no limit. Aim for scannable: a title, one blockquote, and links grouped under H2 headings with a one-line description each. Our file runs 61 lines for a catalog of six dataset packs plus an MCP server, and it covers everything an agent needs to start querying.

Where does the llms.txt file go?

At the root of your domain, https://example.com/llms.txt, the same place robots.txt and sitemap.xml live. On a static host that is one file in the repository root. The convention also allows an optional llms-full.txt alongside it with expanded content, but the short file is the one agents fetch for orientation.

Read ours, then write yours

Fetch the live file to see the whole thing, then use the agent access page to see what the links route to: curl and Python examples against catalog.json and the sample endpoints. For what the format is and is not, start with the llms.txt guide; for the definition and how agents use the file, read what is llms.txt; for how it differs from your sitemap, see the side-by-side comparison.