Key takeaways
- llms.txt is a short text file, proposed in 2024, that summarises a website for AI tools. It’s a suggestion, not an official standard.
- Google says Google Search, including its AI features, ignores llms.txt. No other major AI company has said its assistant reads it on business websites.
- In Ahrefs’ study of 137,210 websites, 97% of llms.txt files got no requests at all in May 2026.
- For most local and service businesses, it’s not worth paying for. The same hour spent on your robots.txt, your Google Business Profile and your service pages does far more.
“My web guy says we need an llms.txt file so ChatGPT can find us. Do we?”
I hear some version of this every week, usually attached to a quote, a plugin upgrade or an “AI readiness” score.
Here’s the short answer: probably not, and you certainly shouldn’t pay much for it. The rest of this guide shows you the evidence, explains who llms.txt is actually for, and tells you what to do with the time instead.
1. What is llms.txt?
llms.txt is a plain text file, written in a simple format called Markdown, that sits on your website and gives AI tools a short summary of your site with links to its most important pages. Jeremy Howard of Answer.AI proposed it in September 2024. It’s a proposal, not an agreed standard.
Web pages are built for people, with menus, pop-ups and scripts that make them messy for machines. The llms.txt proposal suggests a single file at yourwebsite.com/llms.txt that tells an AI tool, in plain words, what the site is and where the good information lives.
The proposal was updated to version 2 in August 2026. The only required part is a heading with the name of the site. Everything else is optional: a one line summary, some notes, and lists of links under headings.
Here’s what a simple one could look like for Harbour Electrical, an invented electrician in Durbanville:
“` # Harbour Electrical
Electricians in Durbanville, Cape Town. Fault finding, DB board upgrades, certificates of compliance and backup power installs for homes and small businesses.
Services
- DB board upgrades: what the job involves and how long it takes
- Certificates of compliance: when you need one and how the inspection works
Contact
“`
That’s all it is. It doesn’t change what visitors see, and it doesn’t control anything. Think of it as a note left at the front door for a visitor who may or may not ever knock.
2. Is llms.txt actually used?
Mostly, no. Ahrefs checked 137,210 websites and found that 97% of the llms.txt files they published got no requests at all in May 2026. Where files were fetched, the bots that power AI search answers made about 1% of the requests. Coding tools and SEO checkers made far more.
Ahrefs’ study, published in June 2026 and updated in August, looked at every website using its analytics product that had traffic in May 2026. It found that 28% of them published an llms.txt file. Ahrefs notes that its customers are more technical than most, so that figure is an upper limit for the web as a whole.
Then it checked who asked for those files:
- 97% of files were never requested. Not by a bot, not by a person.
- AI search bots barely showed up. The bots that fetch pages to answer live questions, like OpenAI’s OAI-SearchBot and PerplexityBot, made 1.1% of requests to the files that were read.
- Nothing went looking. Zero AI bots asked for an llms.txt file on sites that didn’t have one.
- Coding tools were the real readers. AI agents, including Anthropic’s coding tool Claude Code, made more requests than any other type of AI bot.
Ahrefs adds that a request isn’t proof anything was used, so even these numbers are generous.
That fits the proposal itself, which says the files are used most heavily for software documentation.
3. Does Google, ChatGPT or any other AI tool read llms.txt?
Google says Google Search ignores it, including AI Overviews and AI Mode. Chrome’s Lighthouse tool checks for the file, but only for AI agents browsing sites. OpenAI, Anthropic, Perplexity and Microsoft haven’t said their assistants read llms.txt on business websites. Crawler access is set in robots.txt.
Google Search. Google’s guide to optimising for generative AI features, last updated in July 2026, lists llms.txt under things you don’t need to do. It says Google Search doesn’t use these files, and that creating one “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”
Chrome’s Lighthouse. This is where the confusion starts. Lighthouse, Google’s free website testing tool, now has an llms.txt check in a new group of tests for AI agents that browse websites. Read the detail, though. The check only fails if your server errors when asked for the file. If you don’t have one, the test is marked “not applicable”, because providing the file is “optional at the moment”. It has nothing to do with Google Search rankings.
Google, in person. When asked about the two messages, Google’s John Mueller described llms.txt as “not done for search” and a “temporary crutch, perhaps to save some tokens” for AI coding tools, as reported by Ahrefs.
OpenAI, Anthropic, Perplexity and Microsoft. I searched for a statement from each on 28 September 2026 and didn’t find one saying their assistants read llms.txt on ordinary websites. OpenAI and Anthropic publish llms.txt files for their own developer documentation, as Google does for Gemini, which is exactly the use the file was made for. That’s not the same as reading yours.
4. Is llms.txt mandatory, and how is it different from robots.txt?
No, it’s not mandatory for anything. It’s also not a control file. robots.txt tells crawlers what they may access, and AI companies do follow it. llms.txt only suggests what’s worth reading. It can’t block an AI crawler, let one in, or stop your content being used.
The names look alike and both sit in the same place on your site, but they do different jobs.
robots.txt is a long established file that tells automated crawlers which parts of your site they’re allowed to visit. This is the one that matters for AI. OpenAI’s crawler documentation lists separate bots for search (OAI-SearchBot) and for training (GPTBot), and you control each one in robots.txt. OpenAI’s help article on searching the web with ChatGPT says sites must allow OAI-SearchBot to appear in ChatGPT’s search answers. Changes take about 24 hours to be picked up.
llms.txt is a curated summary for AI tools that choose to look for it. The proposal itself says it’s used “on demand” when an agent needs information, which is different from a crawler visiting on a schedule.
So to keep AI companies out, or let them in, check robots.txt. llms.txt won’t do either.
5. Should a local or service business add llms.txt?
Only if it’s free and takes minutes, or if developers use your documentation. Don’t pay for it, and don’t expect it to get you into ChatGPT or AI Overviews. For a plumber, dentist or accountant, the evidence says it will almost certainly never be read.
Here’s how I’d decide, in order.
Do developers or AI coding tools use your documentation? If you sell software, an API or a product with technical manuals, add one. That’s the audience the file was built for, and the audience Ahrefs found actually reading it.
Does your website platform already make one for free? The proposal lists several that do, including Yoast SEO and AIOSEO for WordPress, and Wix, which generates one for every Wix site. If yours is switched on, leave it. Open yourwebsite.com/llms.txt once and check the business name, the summary and the links are right.
Were you quoted a fee? Say no. No major AI company says it reads the file to decide who to recommend, and Google says Google Search ignores it.
Otherwise, skip it. No AI bot goes looking for a missing file, according to Ahrefs, so you’re not missing a knock at the door.
What doesn’t work, whatever you’ve been told
- “llms.txt gets you cited by AI.” No major AI company says this, and Ahrefs found AI search bots rarely fetch the file at all.
- “Lighthouse flags it, so Google wants it.” The check is for AI agents, and a missing file is marked not applicable.
- “Use llms.txt to block AI.” It can’t. That’s robots.txt’s job.
- “Stuff it with keywords or instructions.” Ahrefs found a research bot studying llms.txt files for prompt injection, which is hiding instructions for AI tools inside content. In our view, a file full of sales copy or commands is more likely to be distrusted than rewarded.
6. How do you create llms.txt (or llms-full.txt) if you decide to?
Switch it on in your SEO plugin or website builder if it offers the option, then check the result. To write one yourself, create a text file with your business name, a one line summary and links to your key pages, and upload it to your site’s main folder. It’s a ten minute job.
If your platform does it. Check your SEO plugin’s settings for an llms.txt option, then open the file in your browser and read it as a stranger would. Plugins often pull in every page, including old ones. Trim it to what matters.
If you write it yourself.
- Open a plain text editor and start with a heading line: a hash sign, a space and your business name.
- Add one line starting with the greater than sign (>) that says what you do and where.
- Add a heading for your services, then one link per key service page with a short note.
- Add your contact or service area page.
- Save it as llms.txt and ask your web developer or host to put it in your site’s main folder, so it opens at yourwebsite.com/llms.txt.
What about llms-full.txt? It’s a bigger version that pastes the full text of your pages into one file. Mintlify, a documentation platform, says it developed the format with Anthropic. The current proposal doesn’t describe it. For a business website, it’s more to maintain for no clear gain. Skip it.
Keep it safe. Ahrefs recommends treating the file like code: limit who can edit it, keep it to plain links and descriptions, and link only to pages you control. If your details change, update it with everything else, or remove it.
7. What should you do instead to show up in AI answers?
Make sure AI search bots can reach your site, keep your Google Business Profile complete and accurate, and write service pages with specific, first-hand detail. Google’s own guide says unique, useful content will likely influence its AI features more than anything else it recommends.
Check your robots.txt. Open yourwebsite.com/robots.txt. If you see “Disallow: /” under OAI-SearchBot or under “User-agent: *”, ChatGPT’s search can’t use your pages. Security plugins sometimes add these rules without telling you.
Be indexed and quotable. Google’s guide says a page must be indexed and eligible to show with a snippet to appear in its AI features. If a page isn’t in Google, it can’t be in AI Overviews either.
Fix your business profiles. Google’s guide names Google Business Profile as the way local business details get into its AI answers. Start with our list of 12 Google Business Profile mistakes that quietly cost you calls.
Earn what others say about you. In BrightLocal’s 2026 survey, 45% of US consumers had used ChatGPT or another AI tool to find local business recommendations. They build answers from reviews, directories and articles too, as we explain in Where does ChatGPT get its information? A guide for business owners.
Write what only you can write. Real prices in your area, jobs you’ve done, problems you see every week. There’s more in our guide to writing content AI actually quotes.
Measure it. Ask the AI tools your customers use the questions they’d ask, and note who gets named. Our guide on how to check what AI says about your business walks you through it.
It’s the same lesson as Schema markup for AI search: what it does and doesn’t do: neat code helps machines read your facts, but can’t make you the best answer.
How this looked for Harbour Electrical (invented example). The owner searched ChatGPT for “electrician in Durbanville for a DB board upgrade” and wasn’t named. A developer then offered to set up llms.txt. Instead, they checked robots.txt and found a security plugin blocking all unknown bots, OAI-SearchBot included. They fixed that, corrected their Sunday hours on Google, and rewrote the DB board page with how pricing works, how long the job takes and what the inspection involves. Their SEO plugin already made an llms.txt file, so they left it on and trimmed it to five links.
What to do this week
- Open yourwebsite.com/llms.txt to see whether your platform already made one, and check it’s accurate if so.
- Open yourwebsite.com/robots.txt and make sure OAI-SearchBot isn’t blocked.
- Decline any fee for llms.txt as a route into AI answers.
- Check your Google Business Profile hours, categories and phone number against your website.
- Ask ChatGPT and Google the question your best customer would ask, and note who gets named.
- Pick one service page and add real detail: timelines, what’s included, what goes wrong.
- Ask one happy customer for a review this week.
Frequently asked questions
Is llms.txt mandatory?
No. It’s a proposal, not an official web standard, and no search engine or AI tool requires it. Chrome’s Lighthouse marks a missing file as not applicable because providing one is optional. Google says it neither helps nor harms your visibility in Google Search.
Does llms.txt help SEO?
Not in Google. Google says Google Search ignores llms.txt, including AI Overviews and AI Mode, so it won’t affect your rankings either way.
Does ChatGPT use llms.txt?
OpenAI hasn’t said that it does, and in Ahrefs’ study its search bot made very few requests for these files. To appear in ChatGPT search, make sure OAI-SearchBot can reach your site.
What is llms-full.txt?
A larger version that puts the full text of a site’s pages into one file, mostly used for software documentation. Mintlify says it created the format with Anthropic. The current llms.txt proposal doesn’t describe it. Most business websites don’t need it.
Can llms.txt block AI from using my content?
No. llms.txt doesn’t control access at all. To manage AI crawlers, use robots.txt, where companies like OpenAI list separate bots for search and for training. Blocking a search bot can also remove you from that tool’s answers, so decide carefully.
Does Yoast or my website builder create llms.txt?
Some do. The proposal lists Yoast SEO and AIOSEO for WordPress, and Wix. If yours makes it for free, leaving it on does no harm. Check it once for outdated pages.
Should I pay someone to set up llms.txt?
Not for AI visibility. It’s a short text file that a plugin can make for free or you can write in ten minutes. If someone is selling it as a way into ChatGPT or AI Overviews, ask which AI company has said it reads the file.
Want to know what AI says about you?
Most owners get sold files, scores and quick fixes. Our free audit tests real customer questions across the major AI engines and shows you what’s actually holding you back.
Book a free discovery call and choose the audit that fits: AI Visibility & GEO, SEO, or Local Business SEO. No pitch until you’ve seen the gap.
Related reading
- What is GEO? A plain-English guide for business owners
- Schema markup for AI search: what it does and doesn’t do
- Where does ChatGPT get its information? A guide for business owners
About the author
Bantu Nonyukela is the founder of SearchLimelight. He has spent ten years in SEO, after starting out as a front-end developer, and is a Toptal-verified SEO expert. He helps small and medium businesses in the United States, Australia, the United Kingdom and South Africa get found in search and named by AI. Connect on LinkedIn
Sources
- Jeremy Howard, “The /llms.txt file, v2,” llmstxt.org, published September 2024, modified August 2026: llmstxt.org
- Chrome for Developers, Lighthouse llms.txt audit: developer.chrome.com
- Google Search Central, “Optimizing your website for generative AI features on Google Search,” updated July 2026: developers.google.com
- Ahrefs, “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read,” June 2026, updated August 2026: ahrefs.com
- Ahrefs, “What Is llms.txt, and Should You Care About It?”, updated June 2026 (source of the John Mueller quotes): ahrefs.com
- OpenAI, Overview of OpenAI crawlers: developers.openai.com
- OpenAI Help, Searching the web with ChatGPT: help.openai.com
- Mintlify, “What is llms.txt? Breaking down the skepticism”: mintlify.com
- BrightLocal, Local Consumer Review Survey 2026: brightlocal.com
