llms.txt vs sitemap.xml: what each file does

The short answer

sitemap.xml is a URL list for search engine crawlers. It tells Googlebot and friends: here is every page on this site, please index them. llms.txt is a guided reading list for LLMs and AI agents. It tells them: start here, these are the pages and endpoints worth your context window, with a one-line description for each.

One file wants maximum coverage. The other wants curation. That difference drives everything else: format, discovery, and who reads it.

What sitemap.xml does

A sitemap is machine-generated XML: one <url> block per page with a location and an optional last-modified date. Here are the first lines of ours, unedited:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://jayjex.github.io/data-vault/</loc><lastmod>2026-09-09</lastmod></url>
  <url><loc>https://jayjex.github.io/data-vault/agents/llms-txt-guide.html</loc><lastmod>2026-09-09</lastmod></url>
  <url><loc>https://jayjex.github.io/data-vault/datasets/hud-fmr-2026.html</loc><lastmod>2026-09-07</lastmod></url>
  <url><loc>https://jayjex.github.io/data-vault/guides/csv-vs-api.html</loc><lastmod>2026-09-09</lastmod></url>
</urlset>

Every publishable URL goes in, whether it is a dataset page, a guide, or the printables index. Notice what is missing: descriptions. A crawler gets the URL and the date, and it learns what the page says only by fetching and parsing it. That is by design, because a search index wants to rank the page itself, not trust your summary of it.

Also missing: everything that is not an HTML page. Our catalog.json, the sample CSV endpoints, and the MCP install command have no place in a sitemap. Search crawlers index pages; they do not need a JSON index of datasets.

What llms.txt does

llms.txt is markdown written for a different reader. An agent arrives with no browser session and a limited context window. It cannot afford to fetch and parse 30 HTML pages to find the one dataset it needs. The file below is the top of ours, unedited:

# Data Vault

> Sample-first data catalog by Matchbook Labs. Free CSV/JSON samples for every dataset, machine-readable index for agents, full packs sold via Getly.

Base URL: https://jayjex.github.io/data-vault
Catalog (start here): https://jayjex.github.io/data-vault/catalog.json
Agent docs: https://jayjex.github.io/data-vault/agents/
MCP server: npx --allow-git=all -y github:jayjex/dataset-mcp (v1.1.0; tools: list_datasets, get_dataset_info, get_sample, query_dataset, get_stats)

## Datasets

### HUD Fair Market Rents FY2026 (hud-fmr-2026)
FY2026 fair market rents for 51,895 ZIP codes, 3,229 counties, and 52 states. Five bedroom sizes per row. Niche: gov-data.
- Sample JSON: https://jayjex.github.io/data-vault/data/hud-fmr-2026/sample.json
- Page: https://jayjex.github.io/data-vault/datasets/hud-fmr-2026.html

Three things a sitemap cannot do, this file does in the first screenful:

• It describes. Each entry carries a one-line summary, so an agent can decide relevance before fetching anything else. The blockquote alone settles "what is this site."
• It covers endpoints, not just pages. catalog.json, sample JSON URLs, and the MCP install command are first-class entries. An agent that only reads this file still knows how to query the data.
• It carries the rules. License and attribution terms sit in the same payload, so an agent reads them before copying data, not after.

The tradeoff is reach. Nothing guarantees an agent will look for the file. It is a convention sites adopt voluntarily, read by the agent products that choose to check for it. Google's crawlers ignore it entirely.

Same site, both files: what each one contributes

This catalog runs both, which makes the division of labor easy to see. The sitemap lists all 34 pages and exists so Google indexes the dataset and guide pages at all. The llms.txt is shorter, curated, and ends with access rules instead of URL blocks.

The two files overlap on almost nothing. A dataset like HUD fair market rents appears in both, but as different payloads: a bare URL with a date in the sitemap, versus row counts, sample endpoints, and a link to the full-data page in llms.txt. The sitemap entry gets the page indexed for someone searching rent data. The llms.txt entry gets the dataset used by an agent building something with rent data.

If you deleted one file, the loss would be asymmetric. Lose the sitemap and search indexing degrades to whatever links crawlers stumble across. Lose llms.txt and agents fall back to crawling HTML, which is slower and burns their context, but still possible. That is the honest ranking of stakes: the sitemap protects an established channel, llms.txt improves an emerging one.

Do you need both?

If your site serves humans through search and serves agents through data or docs, yes, and the setup cost is two static files. Sitemap first: it is a settled protocol, search consoles accept it directly, and any content site benefits. Add llms.txt when agents are or should be part of your audience, which today mostly means sites with APIs, datasets, or substantial documentation.

Two rules keep the files honest. Keep the sitemap exhaustive and the llms.txt curated. And keep llms.txt links absolute, because agents may read the file far from your site where relative paths break. Our llms.txt structure guide covers the format in depth, including a link-checker script, and the what is llms.txt explainer keeps the definition separate from the mechanics.

FAQ

Can llms.txt replace sitemap.xml?

No. The two files serve different readers. sitemap.xml speaks to search engine crawlers that index pages, and llms.txt speaks to LLMs and agents that need to pick the right pages and endpoints to read. A site that wants both search traffic and agent traffic publishes both.

Do search engines use llms.txt for ranking?

Google has said its crawlers ignore llms.txt and it plays no role in ranking. The file is read by AI agents and agent products that choose to fetch it, not by the search index. Publish it for agents, keep the sitemap for search.

Where does robots.txt fit next to llms.txt and sitemap.xml?

robots.txt is the exclusion file: it tells crawlers what not to fetch and, in most setups, carries a Sitemap: line pointing at sitemap.xml. llms.txt is the opposite direction. It recommends what an LLM should read. A typical root directory holds all three, each doing one job.

Which file should I add first?

The sitemap. It is a defined protocol at sitemaps.org, search consoles accept it directly, and it benefits any site that wants pages indexed. Add llms.txt when you also want agents and LLMs to find your machine-readable endpoints, APIs, or key docs without crawling HTML.

Get the full pack

The packs this catalog sells are the full versions of the datasets above: complete CSVs with data dictionaries, source checksums, and licensing for personal and commercial use. Free samples for every dataset stay open, and agents can query them through the free MCP server.

Checkout and download run through Getly. Want to see data quality first? Fetch any free sample or read the pipeline behind the packs.