llms.txt is not doing anything for you

Publishing an llms.txt file has no measurable effect on how AI systems see your site. A study across 137,210 domains found that 97% of published llms.txt files received zero requests over a month. The largest single category of requester was not an AI crawler at all. It was SEO audit tools checking whether the file existed.

Google's documentation states plainly that Google Search does not use them.

If your team shipped one, no harm done. If a proposal you are reading lists llms.txt as a deliverable, that line item is doing nothing, and you should ask what else on the list was chosen for the same reason.

Where the idea came from

The proposal was reasonable on paper. Robots.txt tells crawlers what they may fetch. Sitemap.xml tells them what exists. So llms.txt would tell language models which pages matter, in clean markdown, with a short description of each.

It was suggested in late 2024, spread fast through SEO communities in 2025, and became a standard checklist item almost overnight. The reason it spread was not evidence. It was that it takes twenty minutes to implement and produces a visible artefact a client can look at.

No major AI provider adopted it. OpenAI, Anthropic, Google and Perplexity all continue to retrieve pages the same way they always have, through crawling and through search indexes. None of them documented llms.txt as an input.

The mechanism, which is the part that settles it

Language models do not browse your site looking for a manifest. Two things happen instead.

Training crawls fetch pages at scale and store what they find. That happens through normal crawling paths, and it is governed by robots.txt and by whatever agreements the provider has with publishers.

Grounded retrieval, which is what actually produces citations in ChatGPT, Perplexity, Gemini and Google AI Overviews, works off a search index. The system generates queries, hits an index, retrieves passages, then writes an answer citing what it retrieved. Perplexity and ChatGPT lean on their own crawls plus search partners. Google's AI features run on the Google index.

At no point in either path does the system stop and consult a curated list you wrote about yourself. Even if one did, the file is unverifiable self-description. A system that trusted it would be trivially gamed within a week.

The pattern to watch for

llms.txt is a useful diagnostic, because it tells you how a supplier thinks. A tactic that is easy to implement, easy to show a client, impossible to measure and untethered to any documented mechanism will spread faster than a tactic that works.

Three others in the same family right now:

Chunking content into tiny fragments for LLMs. Google's May 2026 optimisation guide says there is no requirement to break content into small pieces. Sites that restructured entire libraries around this got nothing for it.

Manufacturing brand mentions across the web. Google's guidance says this is not as helpful as it sounds. Paying for scattered unlinked mentions is close to the modern version of buying directory links.

Adding structured data specifically to win AI citations. The same guide states structured data is not required for generative AI search. Schema still earns its place for other reasons, which is a separate argument, but not this one.

The common thread is that all four were invented by the marketing industry and then attributed to the platforms. None came from a platform announcement.

How to test a tactic before you fund it

You do not need a technical team to filter this. Four questions kill most of it.

1. 1.Where is the primary source? Not a blog post citing a blog post. A platform doc, an engineering paper, or a named executive on record. If the trail ends at an agency article, it is folklore.
2. 2.What is the mechanism? Someone should be able to explain, in two sentences, the path from the change to the outcome. Saying it helps AI understand your site is not a mechanism. Saying it gets the page into the index, and grounded retrieval only cites indexed pages that are eligible for a snippet, is one.
3. 3.How would we know it worked? If the only proof available is that a tool reports the file exists, there is no measurement, only confirmation.
4. 4.What does it cost if it is wrong? Twenty minutes for llms.txt, so the cost is low and the real damage is opportunity. Restructuring 800 pages for chunking is a different question entirely.

What to do with the file you already have

Leave it. It does not hurt anything, it costs nothing to host, and if a provider ever adopts the convention you are already there. Just take it off the reporting deck, because listing it as an AI search deliverable creates a false sense that the work is done.

Then redirect the effort. Here is where the same hour actually pays:

Audit snippet directives. Any page carrying nosnippet, or a max-snippet value that suppresses text, is ineligible for AI features. This is one of the only documented controls Google gives you, and large sites routinely carry legacy directives nobody remembers adding.

Check indexation on your commercial pages. Grounded retrieval only reaches what is indexed. A page excluded by a stray canonical or a crawl budget problem cannot be cited, no matter what you write on it.

Fix the robots rules for AI crawlers deliberately. Decide, as a business, whether you want GPTBot, ClaudeBot, PerplexityBot and the rest fetching your content. That is a real decision with real consequences on both sides. Most companies have never made it consciously; they inherited whatever the CMS shipped with.

Read Google's own guide. It is short, it is free, and it contradicts a large share of what is being sold as AI SEO right now.

The gap between what platforms document and what the market repeats is where budget goes to die.