llms.txt: what it is, and whether it does anything yet

The proposal was published in September 2024 by Jeremy Howard, co-founder of Answer.AI, at llmstxt.org. The reasoning is straightforward. Language models read websites badly: modern pages arrive wrapped in navigation, cookie banners, scripts and markup that consume a model's limited context window before the actual content does. A human skims past the clutter; a model pays for every token of it. So the proposal asks site owners to publish a small markdown file at a predictable address, /llms.txt, that says what the site is and points to its most important pages in a form models parse cheaply.

The analogy the proposal reaches for is robots.txt: one well-known file at one well-known location, established by convention rather than by any standards body. That analogy explains the name and the appeal, and it is also where the trouble starts, because robots.txt succeeded for a reason llms.txt has not yet reproduced. Crawlers actually honor robots.txt. The value of a convention file is precisely the list of consumers who have committed to reading it, and for llms.txt in mid 2026, that list of confirmed major engines is empty.

It is worth being precise about what the file is for, because coverage routinely muddles it. llms.txt does not control crawling, does not grant or deny permission to train on your content, and does not affect search indexing. It is an offer of curated context: a menu, published in the hope that a model fetching your site at inference time will read the menu instead of rummaging through the kitchen. Whether anyone reads the menu is the entire question, and this guide answers it as honestly as the evidence allows.

An llms.txt file is markdown with a loose expected shape. It opens with an h1 carrying the site or project name, followed by a blockquote holding a short summary of what the site is. After that come optional sections, each an h2 naming a category such as documentation, guides or policies, containing bulleted lists of links with one-line descriptions. The proposal also reserves a section conventionally titled Optional for links a model can skip when context is tight.

A minimal example for a SaaS looks like this:

# Reachroller

> Reachroller tracks whether AI assistants mention your brand
> when buyers ask, and generates the pages that fix the gaps.

## Product

- [How it works](https://reachroller.com/how-it-works): the
  track, fix and recheck loop explained
- [Pricing](https://reachroller.com/pricing): plans from $29/mo

## Guides

- [Measure AI visibility](https://reachroller.com/blog/how-to-measure-ai-visibility):
  the honest measurement method

The companion convention, llms-full.txt, goes further: instead of linking to pages, it inlines their full content into one large markdown document, so a model can ingest the whole site in a single fetch. That suits documentation sites, where AI coding tools sometimes consume docs in bulk, and it is heavier to maintain because every content change should regenerate the file. For a marketing site, the short form is the sensible ceiling of effort.

Here is the uncomfortable center of this guide: no major AI engine has confirmed using llms.txt. OpenAI documents its crawlers, including OAI-SearchBot, and describes robots.txt as the control surface; its documentation does not commit to reading llms.txt. The same silence holds across Anthropic, Perplexity and Google as of mid 2026. Google's search relations voices have gone further than silence, publicly comparing llms.txt to the keywords meta tag, the self-declared signal from the early web that search engines learned to distrust precisely because site owners wrote it about themselves.

That comparison carries a real argument, worth taking seriously. Engines prefer signals that are hard to fake. What your pages actually say, who links to and cites them, how they rank in indexes built over decades of spam-fighting: those are earned signals. A file in which you declare your own importance is exactly the kind of input retrieval systems have historically routed around. Any future in which llms.txt matters is one where engines decide the curation value outweighs the manipulation risk, and none has made that call publicly.

Meanwhile the crawler traffic that observably exists tells you where engine attention actually goes. OpenAI has roughly tripled its web crawl since August 2025 according to Botify's analysis, and that crawl fetches pages, the same pages search indexes hold. The infrastructure engines demonstrably invest in is page retrieval, which is why the access files that govern page retrieval, covered in our guide to the AI crawler boom, are worth an hour of your attention before llms.txt gets ten minutes.

Having been blunt about the evidence, here is the fair case for the file. First, asymmetry: the cost is ten minutes and the known downside is zero, while the upside, however improbable, is being early if any engine announces support. Cheap options on uncertain futures are how sensible teams handle emerging conventions, and this is about as cheap as options get.

Second, the ecosystem is wider than the big consumer engines. A number of AI-adjacent tools and developer assistants fetch llms.txt where it exists, and documentation-heavy products have adopted the convention for their own docs so that coding assistants ingest them cleanly. If your buyers interact with your product through AI development tools, the file has a plausible audience today even while the major answer engines stay silent. For a marketing site, this audience is thinner, which is why the honest recommendation scales with how technical your readership is.

Third, the exercise itself has a side benefit: writing a one-paragraph summary of your site and choosing its ten most important pages is a useful clarity test. If you cannot fill the file crisply, that says something about the site's information architecture that no crawler caused. Just keep the effort proportionate: the file is a footnote to your AI visibility work, and the moment it starts consuming content time, the priorities have inverted.

Minute one: create a plain text file named llms.txt. Minutes two through three: write the h1 with your brand name and a blockquote of one to three sentences saying what the site is and who it serves, in plain declarative language a model can quote. Minutes four through eight: add two or three h2 sections, product, guides, policies, whatever fits, each listing your genuinely most important pages with absolute URLs and one-line descriptions. Be selective: ten strong links beat fifty weak ones, and curation is the file's entire claimed value.

Minutes nine and ten: deploy it so it resolves at your domain root, yourdomain.com/llms.txt, served as plain text with a 200 status. On most stacks that means dropping the file into the public or static directory; no server configuration, headers or registration anywhere else is involved. Verify it loads in a browser, and you are done.

Then set two low-effort reminders. Revisit the file quarterly so it never describes pages you have moved or retired, because a stale map is worse than no map. And add /llms.txt to whatever log review you already do, so that if AI crawlers ever begin requesting it, you notice. That is the complete, honest implementation: anything more elaborate is effort the evidence does not currently justify.

The reason to rank levers by evidence is that the evidence differs so much. At the top sits content structure, the only lever with peer-reviewed measurement: the Princeton and Georgia Tech GEO study, published at KDD 2024, found that adding statistics, quotations and cited sources lifted a page's visibility in generative engine responses by up to roughly 40 percent, while keyword stuffing performed near the bottom. Every hour spent making a page more citable has a measured mechanism behind it, documented further in how to write content AI engines actually cite.

Next comes plumbing: being indexed in Google and Bing, whose indexes feed most AI retrieval, and keeping the search crawlers unblocked in robots.txt. Then third-party presence, since citation analyses consistently show engines leaning on independent pages over brand-owned domains. Somewhere below that sits schema markup, where the evidence is genuinely mixed, correlation without proven lift, weighed honestly in schema markup for AI search. And below schema sits llms.txt, supported so far by no confirmation at all. The gap between those tiers is the whole argument of this post.

The stakes make prioritization worth the discipline: G2's 2026 research found 51 percent of B2B tech brands have zero citations across ChatGPT, Perplexity and Gemini. A brand in that half does not close the gap with a manifest file; it closes it by finding the buyer questions it loses and publishing the answers, the loop walked through step by step in how to get your brand mentioned by ChatGPT. Reachroller exists to run exactly that loop: it tracks your questions through official engine APIs, generates a publish-ready fix page for every answer you lose, and rechecks the answer after you ship, from $29 per month.

Because the file's value depends entirely on adoption by engines, the signals to watch are external and specific. The first is official documentation: an engine stating that its crawler reads llms.txt, the way OpenAI documents OAI-SearchBot's behavior today. The second is your own server logs showing AI crawler user agents requesting /llms.txt in meaningful volume. Either one would justify promoting the file up the priority list; enthusiasm in marketing blogs, including this one, justifies nothing.

Be wary of the success stories that circulate in the meantime. A brand adds llms.txt, sees an AI mention two weeks later, and credits the file, when the same fortnight included new content, fresh crawls and the ordinary churn of probabilistic answers. Without a controlled comparison, that is correlation dressed as causation. The measurement discipline that catches this, baselines, repeated runs, and one change at a time, is laid out in how to measure AI visibility without lying to yourself.

Our own position, stated plainly: Reachroller measures outcomes rather than rituals. Its reports track whether engines actually mention your brand, answer by stored answer, and its rechecks show whether a change you shipped flipped a real answer. If llms.txt earns engine support someday, that will show up in flipped answers, and the loop will catch it. Until then, spend the ten minutes if you like the optionality, then get back to the work the evidence already pays for.

Developer products and documentation sites: add it, and consider llms-full.txt too. This is the one corner of the ecosystem where the file has a live audience today, because AI coding assistants and developer tools sometimes ingest documentation in bulk, and a clean markdown surface genuinely helps them. If your users paste your docs into assistants or ask coding tools how to integrate your API, the ten minutes is trivially justified and the quarterly maintenance is worth scheduling properly.

Marketing sites for SaaS and services: add the short form if the ten minutes is truly spare, and expect nothing measurable from it. Your AI visibility on the surfaces that decide deals, the chat engines and answer features buyers actually use, runs through indexed, citable pages and third-party mentions, and none of the engines serving those buyers has confirmed reading the file. Write it once, date it, revisit quarterly, and resist every temptation to expand it into a project.

Publishers and content businesses: the calculus is the same as marketing sites with one addition: if your concern is controlling AI access to content rather than encouraging it, your instrument is robots.txt and the licensing conversation, and llms.txt is beside the point entirely. In all three cases the file never rises above footnote status on current evidence, and the decision rule stays one sentence long: spend ten minutes if the optionality appeals, and spend the next hour on a page an engine can cite.

"llms.txt controls whether AI trains on your content." It does not. Training consent is negotiated through robots.txt directives aimed at training crawlers, and through the legal and licensing layer above them. llms.txt has no directive vocabulary at all: it cannot allow, deny or rate-limit anything. A site owner who wants to welcome search crawlers while refusing training crawlers expresses that entirely in robots.txt, user agent by user agent, and publishing an llms.txt file changes nothing about who may fetch what.

"Big sites are adding it, so the engines must be reading it."The first half is often true and the second does not follow. Publishing a file is a site owner's decision; reading it is an engine's, and only the second one creates value. Plenty of conventions have enjoyed enthusiastic publisher adoption while the consuming side stayed away, the keywords meta tag being the canonical example, which is precisely the comparison Google's search voices have drawn. Count confirmations from engines, and as of mid 2026 the count is zero.

"It cannot hurt, so there is no reason not to go deep on it." Half right. The file itself is harmless, and this guide recommends adding one. The hurt arrives through substitution: teams with one content afternoon per week spending it curating a manifest instead of publishing an answer to a question they lose. Against a lever with a measured lift of up to roughly 40 percent in the GEO study, an unconfirmed convention should never win the same afternoon. Cap the investment at its ten minutes, keep the file honest at quarterly reviews, and let the priority list stay ordered by evidence.