llms.txt

The llms.txt proposal starts from a real problem: language models work with limited context windows, and typical web pages bury their substance under navigation, scripts and boilerplate. The proposed fix is a plain markdown file at /llms.txt containing the site name, a short summary, and organized lists of links to the pages that matter, so an AI system could grasp a site's structure in one small, clean read. An optional companion convention serves markdown versions of individual pages, and a fuller llms-full.txt variant inlines the content itself.

It is worth being precise about what the file claims to do, because the name invites confusion with robots.txt. robots.txt is an access control convention: it tells crawlers what they may fetch, and the major AI operators document that their crawlers honor it. llms.txt grants nothing and forbids nothing. It is an offer of curation, useful only if an AI system chooses to read it, and no permission or restriction flows from publishing one.

Adoption on the publishing side is genuine. Documentation platforms generate llms.txt automatically, directories track tens of thousands of live files, and Ahrefs measured that roughly 28 percent of 137,000 domains it examined had published one. The open question has never been whether site owners would adopt it. It is whether anyone on the other side reads it.

The best available data comes from an Ahrefs study of 137,000 domains published in 2026, and its headline finding is blunt: about 97 percent of llms.txt files received zero requests at all. Among the small minority of files that did get traffic, 96 percent of requests came from bots, and most of those were SEO audit tools, generic crawlers and profiling services rather than AI systems. AI retrieval bots tied to ChatGPT and Perplexity accounted for about 1 percent of requests to the files.

The official statements point the same direction. Google's John Mueller compared llms.txt to the keywords meta tag, a self-declared description with no consumer on the other end, and Google's guidance now states plainly that you do not need special AI text files or markdown to appear in Google Search or its generative features, because Google does not use them. None of OpenAI, Anthropic or Perplexity has announced that its answer engines consume llms.txt.

One honest nuance survives the negative results. In the Ahrefs data, the AI-related traffic that did touch llms.txt files skewed toward coding agents and training crawlers, with tools like Claude Code and GPTBot among the top individual requesters. A curated index of documentation appears to have a real, narrow audience among developer tools reading docs on a user's behalf. That is a legitimate reason for a developer-facing product to keep the file, and a poor reason to expect citation gains from it.

The cost-benefit math is lopsided in both directions, which is why reasonable teams land differently. Generating the file takes minutes, many platforms produce it automatically, and there is no documented penalty for having one. Against that, the measured probability that an AI search engine reads it rounds to zero, so treating llms.txt as an AI visibility tactic means spending attention on a channel with no demonstrated consumer.

A sensible policy: add it if it is free, skip it if it displaces real work, and never report it as progress. The failure mode worth avoiding is organizational rather than technical, where publishing an llms.txt file substitutes for the harder tasks that measurably move AI visibility: confirming crawler access in robots.txt, structuring pages so they are quotable, earning third-party mentions, and checking what engines actually say about you. If a vendor pitches llms.txt as the centerpiece of an AI strategy, that tells you more about the vendor than the file.

The verification habit matters more than the file. What changes AI answers is what engines can crawl and what they choose to cite, so test crawler access directly, with the free checker at /bot-access, and judge tactics by whether tracked answers change. If engines announce llms.txt support someday, the data will show it, and adding the file then will take the same five minutes it takes today.