If you added an llms.txt file to your site sometime over the past year or two, you probably did it because a blog post or a plugin told you it would help ChatGPT, Claude, Perplexity and Google's AI Overviews find your content. Here is the uncomfortable part. It almost certainly did nothing. No major AI engine has committed to reading the file, the log data says they mostly do not even fetch it, and Google has now put in writing that you can ignore it.
This is not a dunk on the people who came up with the idea. The concept was reasonable, and I will explain why in a second. It just never got the one thing a web standard actually needs, which is the engines on the receiving end agreeing to use it.
What llms.txt was actually supposed to do
The idea came from Jeremy Howard, who published the proposal at llmstxt.org on September 3, 2024, with the spec living in Answer.AI's GitHub org. The reasoning was sound. Websites are messy. A model trying to read yours has to wade through navigation, cookie banners, scripts and markup to find the few paragraphs that actually answer a question, and the context window it has to work with is small. So the proposal was a single plain Markdown file at yoursite.com/llms.txt that hands the model a clean, curated map of your key pages, plus clean Markdown copies of the pages themselves. The llms-full.txt variant that inlines everything into one big file came later, popularized by the tooling vendors rather than the original spec. Think of it as a sitemap written for a reader that wants prose instead of XML.
Here is the part almost everyone missed, and it matters for the rest of this piece. llms.txt was never an access or permissions file. It does not grant anything, it does not block anything, and no crawler is under any obligation to look at it. It is a content curation format, a friendly suggestion about what is worth reading on your site. That is a very different thing from robots.txt, which is a real standard that engines actually obey. Keep that distinction in your back pocket, because it turns out to be the whole story.
Why everyone added it anyway
Two reasons, and neither one is "because it worked." The first is that it was basically free. Dropping a text file on your server costs nothing and feels proactive, like you are getting ahead of the AI shift. The second, and this is the bigger one, is that the documentation platforms turned it on for everybody. Mintlify started auto generating llms.txt and llms-full.txt for every docs site on its platform on November 20, 2024, pitching it in its own words as "SEO for AI." Overnight, thousands of sites had the file without a single person on those sites deciding to add it.
So adoption looked explosive. By June 2026, Ahrefs found that 28% of the 137,210 domains it studied were publishing a valid llms.txt. That is a big number. But adoption on your side of the fence tells you nothing about whether the engines on the other side are reading it. Publishing is not the same as being consumed, and that gap is exactly where the whole thing falls apart.
The evidence that no engine actually reads it
Start with Google, which has now disavowed the file in three separate ways. Back in April 2025, Google Search Advocate John Mueller said on Reddit that no AI service he was aware of had said it uses llms.txt, that server logs show the services do not even check for the file, and, in a line that got quoted everywhere, "To me, it's comparable to the keywords meta tag" (via Search Engine Journal). In that same thread, one person who hosts around 20,000 domains reported that "no bots are really grabbing these." Then in July 2025, Google's Gary Illyes said on stage that Google does not support llms.txt and has no plans to (reported by Search Engine Roundtable).
And it is not just offhand comments. Google's own documentation says it in writing now. The official guide on optimizing your site for AI features on Google Search has a mythbusting section listing things you can ignore, and the very first item is "LLMS.txt files and other 'special' markup." You do not need machine readable text files or Markdown copies of your pages, it says, "as Google Search itself doesn't use them."
Now go one layer out, to the companies that actually run the crawlers, because this is the cleanest evidence of the lot. I read all four of the major vendor crawler docs on July 10, 2026, and not one of them mentions honoring llms.txt anywhere. What they document instead is robots.txt. OpenAI lists GPTBot, OAI-SearchBot and ChatGPT-User, all controlled through robots.txt. Anthropic lists ClaudeBot, Claude-User and Claude-SearchBot, same deal. Google documents Google-Extended for Gemini. Perplexity documents PerplexityBot. Every one of them tells you how to control access with robots.txt. Not a single one tells you to write an llms.txt.
Then there is the log data, which is where it gets brutal. That same Ahrefs study looked at who actually requests these files and found that 97% of valid llms.txt files got zero requests in May 2026. Zero. Of the small slice that did get hit, the top fetchers were not AI engines at all, they were SEO audit tools, generic web crawlers and tech profiling services. The real AI bot requests were a minority of an already tiny pile. So for the overwhelming majority of the 28% of sites that publish it, the file just sits there untouched.
To be fair, it is not literally never. One SEO posted a log screenshot in July 2025 claiming OpenAI was polling his llms.txt every fifteen minutes (also via Search Engine Roundtable). It was a single unconfirmed anecdote, OpenAI never acknowledged it, and its bot docs still do not mention the file a year later. So the honest version is not "no bot ever touches it," it is "no engine commits to reading it, and almost none actually do."
Want proof from your own site? Check your logs
If you can get at your server logs or your CDN analytics, filter for requests to /llms.txt over the last month, then filter for requests to your normal HTML pages from GPTBot, ClaudeBot and PerplexityBot. (Google-Extended will not show up in logs; it is a robots.txt token, not a bot that fetches pages.) You will see the crawlers hitting your real pages constantly and your llms.txt basically never. The bots vote with their requests, and they are not requesting the file.
The whole thing gets funnier the closer you look. Anthropic and Stripe both publish an llms.txt on their own docs, yet no vendor, Anthropic included, documents reading anyone else's. And Google's Search team says to ignore the file while, per Search Engine Journal, its own Lighthouse tool shipped an experimental check for whether your site has one. Even the companies closest to this cannot agree with themselves.
We have seen this exact movie before
Mueller's keywords meta tag comparison was not a throwaway. It is the perfect parallel. The meta keywords tag was the same basic idea: you listed the terms you wanted to rank for, right there in your own HTML, and the engine was supposed to trust it. Search engines learned to ignore it almost entirely, for one simple reason. A signal where you tell the engine what your page is about, with nothing to verify it, is trivially gamed and therefore close to worthless. Everybody just stuffed it with their dream keywords.
llms.txt has the same flaw baked in. It is a curated highlight reel of your site, written by the brand being evaluated, with no way for the engine to confirm a word of it. These systems already spent years learning not to trust self reported signals. They also already have two better ways to read you: they crawl your actual pages, and they pull from a search index at question time. Being handed a summary by the very site they are assessing is not a shortcut they want. If anything, it is the sort of thing they are built to look past.
Where to point that energy instead
The hour you were about to spend maintaining a file nobody reads can go into two things that genuinely decide whether you show up in AI answers: making sure the engines can reach you, and being the answer they want to quote.
Get crawler access right
This is the file that actually matters, and it is robots.txt, not llms.txt. robots.txt is a real standard (RFC 9309, finalized in September 2022) and the engines honor it. Allow GPTBot, ClaudeBot, Google-Extended and PerplexityBot and you are readable. Block them, which a lot of sites did in an early panic and never revisited, and you all but vanish from that engine. Most of the "why am I not showing up in AI" cases I see are some version of a crawler getting turned away by accident.
Two free tools make this a five minute job. Run our AI crawler checker to see, in about two minutes, which of the major AI bots your robots.txt is currently allowing or blocking and what each block actually means. If you need to fix it, our AI robots.txt generator builds the exact rules to allow or block each bot by name. For the full rundown of every crawler and what it does, we wrote a companion piece on which AI crawlers can reach your site.
Then check one more layer, because robots.txt is voluntary and lives at the polite end of the internet. Your firewall or CDN can block a crawler before it ever reaches your server, and it does not care what your robots.txt says. Cloudflare in particular started blocking AI crawlers by default for new domains on July 1, 2025, and gives you granular controls to allow or block each AI service. If a crawler is getting a 403, the fix lives there, not in a Markdown file.
Be the answer worth quoting
Access gets you into the room. It does not get you cited. Being quoted is a separate game, and none of it is touched by llms.txt. The engines build answers from pages that clearly and directly answer the question, from brands that independent third parties corroborate, and from sources that look current. So the work is the unglamorous stuff: write a genuinely good page for the exact question your buyer is asking, earn mentions in the listicles and reviews the engines already trust, and keep your facts consistent everywhere you are described. We break the whole method down in our guide to generative engine optimization and the more tactical how to get cited by ChatGPT. The file to obsess over is your best answer page, not llms.txt.
Measure what the engines actually say about you
The last piece is knowing where you stand, because you cannot improve a number you are not watching. Instead of publishing a file and hoping, run the real prompts your buyers type into ChatGPT, Claude, Perplexity and Gemini, and see who gets recommended and cited today. That is the whole idea behind MentionFlow, the AI visibility platform I built. It tracks your share of voice across the engines and shows which prompts mention you versus your competitors, so the work above has a scoreboard.
If you would rather do it by hand first, that is completely fine, and we wrote up the manual version step by step in our DIY AI visibility audit. Either way, checking what the engines actually say beats maintaining a file they never open.
So should you delete your llms.txt?
You do not have to. A static text file sitting on your server costs nothing and hurts nothing. Google's own guidance is that it neither helps nor harms, so if a plugin generated one for you, leave it and move on. There is even one honest use for the full version: pasting your llms-full.txt straight into a coding assistant as context is a legitimate manual trick, and Mueller himself reportedly called the format a "temporary crutch" for exactly that kind of tool (per Ahrefs). That is a human copying a file on purpose, though, which is a completely different thing from a crawler discovering it in the wild.
What you should not do is spend another minute maintaining it, and you should absolutely not pay anyone for "llms.txt optimization," because there is nothing there to optimize. Put that hour into a page that answers a real buyer question better than anyone else in your category, make sure the crawlers can reach it, and measure what the engines say back. That is the stuff these systems actually reward. The file everyone added was never going to be.