An llms.txt file is a plain Markdown file on your website that gives AI agents a short, curated guide to your most useful pages. It is a community proposal, not an official standard. Google Search ignores it, and no major AI provider has documented reading it from other sites, so treat it as optional.
What is llms.txt?
The llms.txt proposal was written by Jeremy Howard and first published on September 3, 2024. The page was updated in August 2026 and now carries the label v2, with changes the author says are based on two years of adoption.
The idea is simple. AI agents increasingly visit websites on behalf of people, for example to look up documentation or compare options. A full website is often too large and too cluttered for an agent to take in at once. An llms.txt file gives the agent a short index it can read first, with links to the pages that matter.
The proposal is aimed at the moment an agent is helping a user, not at model training. The author writes that he expected the file to be mainly useful at that stage, and says that is how it has been used.
What does an llms.txt file look like?
The file is written in Markdown and normally lives at the root of your site, so the address ends in /llms.txt. The v2 proposal also allows a file inside a subfolder, which then covers the pages under that folder.
The proposal sets out these parts, in this order:
- A top-level heading with the name of the site or project. This is the only required part.
- A short summary in a blockquote.
- Optional extra detail in plain paragraphs or lists.
- Sections with second-level headings, each holding a list of links with brief notes.
- By convention, a section titled Optional for secondary links an agent can skip.
Here is a small example for a fictional business:
# Example Heating and Air
> Example Heating and Air is a fictional HVAC company used here for illustration. It installs and repairs home heating and cooling systems.
## Services
- [Furnace repair](https://www.example.com/furnace-repair.md): what we fix and how visits work
- [Service areas](https://www.example.com/service-areas.md): towns we cover
## Optional
- [Company history](https://www.example.com/about.md)
The proposal also suggests offering a clean Markdown version of important pages at the same address with .md added, which is why the example links end that way.
How is llms.txt different from robots.txt and sitemap.xml?
The three files do different jobs. The proposal itself describes llms.txt as a complement to the other two.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Tells crawlers what they may access | Lists the pages you want search engines to find | Gives AI agents a short, curated guide to key content |
| Format | Plain text rules | XML list of addresses | Markdown with headings and links |
| Who it is for | Search and AI crawlers | Search engines | AI agents working for a user |
| Controls access? | Yes, for crawlers that follow it | No | No |
| Documented use by major providers | Yes. OpenAI, Anthropic and Perplexity document robots.txt controls | Yes. Google’s documentation recommends submitting one | Google Search says it ignores it. Other providers have not documented reading it |
If you want to manage which AI crawlers can reach your site, robots.txt is the tool. We cover the specific crawler names in how to get your business cited in AI search results.
Do Google and AI providers use llms.txt?
Here is what each has said in material you can open and check.
Google’s position on Search is clear. Its guide to optimizing for generative AI features says you do not need new machine-readable files, AI text files or Markdown to appear in Google Search. On these files the guide says, “Google Search itself doesn’t use them.” It adds that it is fine to create and maintain an llms.txt file, and that doing so will neither harm nor help your visibility or rankings. Google’s documentation changelog shows this note was added on June 15, 2026.
Google’s John Mueller has said the same thing in public posts. In June 2025 he wrote on Bluesky that no AI system was using llms.txt at that time, as reported by Search Engine Roundtable. In 2026 he wrote on Reddit that Google does not use llms.txt, as reported by Search Engine Journal.
These statements are about Google Search. The llms.txt proposal notes that Chrome’s Lighthouse tool now checks sites for the file as part of its checks for AI agent browsing, which is a separate product with a separate goal.
OpenAI, Anthropic and Perplexity
We read the public crawler documentation from OpenAI, Anthropic and Perplexity. All three explain how to allow or block their crawlers with robots.txt. None of them says its crawlers read llms.txt files on other websites.
There is one detail worth knowing. The OpenAI and Perplexity pages each point readers to an llms.txt index of their own documentation, and the llms.txt proposal lists OpenAI, Anthropic and Gemini as publishing llms.txt files for their developer docs. So these companies publish the file for agents that read their documentation. That is different from confirming they use yours when they answer questions.
Should your website have one?
For most business websites, llms.txt is optional. Here is a fair way to weigh it.
Reasons to add one:
- It is cheap. The proposal notes that tools such as Yoast SEO, AIOSEO and Wix can generate the file automatically.
- Google says it does no harm.
- If you publish documentation, product specifications or reference material, coding agents and research agents may make use of it. The proposal says thousands of sites now publish one.
Reasons to wait:
- Google Search ignores it, so it will not improve rankings or AI Overview citations.
- No major AI provider has documented reading it from other sites.
- A stale file is worse than none. If it lists old services or dead links, you are handing agents wrong information.
A sensible order of work for a service business is to fix crawler access, publish clear and accurate pages, and add structured data first. Then add llms.txt if your platform can generate it and you will keep it current.
The short version
llms.txt is a proposed Markdown file that helps AI agents find your best content. Google Search ignores it, and other AI providers have not documented using it for answers. It is low cost and harmless if kept accurate, so add it when it is easy, and put your main effort into crawl access and clear content.



