Skip to content
Createfirm

What is llms.txt, and should your website have one?

llms.txt is a proposed Markdown file that helps AI agents find your key pages. Google Search ignores it, so treat it as optional. Here is how to decide.

By the Createfirm teamPublished 6 min read

Illustration of an llms.txt file: a single sheet on a steel pedestal with crawlers approaching along orbit lines

An llms.txt file is a plain Markdown file on your website that gives AI agents a short, curated guide to your most useful pages. It is a community proposal, not an official standard. Google Search ignores it, and no major AI provider has documented reading it from other sites, so treat it as optional.

What is llms.txt?

The llms.txt proposal was written by Jeremy Howard and first published on September 3, 2024. The page was updated in August 2026 and now carries the label v2, with changes the author says are based on two years of adoption.

The idea is simple. AI agents increasingly visit websites on behalf of people, for example to look up documentation or compare options. A full website is often too large and too cluttered for an agent to take in at once. An llms.txt file gives the agent a short index it can read first, with links to the pages that matter.

The proposal is aimed at the moment an agent is helping a user, not at model training. The author writes that he expected the file to be mainly useful at that stage, and says that is how it has been used.

What does an llms.txt file look like?

The file is written in Markdown and normally lives at the root of your site, so the address ends in /llms.txt. The v2 proposal also allows a file inside a subfolder, which then covers the pages under that folder.

The proposal sets out these parts, in this order:

  1. A top-level heading with the name of the site or project. This is the only required part.
  2. A short summary in a blockquote.
  3. Optional extra detail in plain paragraphs or lists.
  4. Sections with second-level headings, each holding a list of links with brief notes.
  5. By convention, a section titled Optional for secondary links an agent can skip.

Here is a small example for a fictional business:

# Example Heating and Air

> Example Heating and Air is a fictional HVAC company used here for illustration. It installs and repairs home heating and cooling systems.

## Services

- [Furnace repair](https://www.example.com/furnace-repair.md): what we fix and how visits work
- [Service areas](https://www.example.com/service-areas.md): towns we cover

## Optional

- [Company history](https://www.example.com/about.md)

The proposal also suggests offering a clean Markdown version of important pages at the same address with .md added, which is why the example links end that way.

How is llms.txt different from robots.txt and sitemap.xml?

The three files do different jobs. The proposal itself describes llms.txt as a complement to the other two.

robots.txt sitemap.xml llms.txt
Purpose Tells crawlers what they may access Lists the pages you want search engines to find Gives AI agents a short, curated guide to key content
Format Plain text rules XML list of addresses Markdown with headings and links
Who it is for Search and AI crawlers Search engines AI agents working for a user
Controls access? Yes, for crawlers that follow it No No
Documented use by major providers Yes. OpenAI, Anthropic and Perplexity document robots.txt controls Yes. Google’s documentation recommends submitting one Google Search says it ignores it. Other providers have not documented reading it

If you want to manage which AI crawlers can reach your site, robots.txt is the tool. We cover the specific crawler names in how to get your business cited in AI search results.

Do Google and AI providers use llms.txt?

Here is what each has said in material you can open and check.

Google

Google’s position on Search is clear. Its guide to optimizing for generative AI features says you do not need new machine-readable files, AI text files or Markdown to appear in Google Search. On these files the guide says, “Google Search itself doesn’t use them.” It adds that it is fine to create and maintain an llms.txt file, and that doing so will neither harm nor help your visibility or rankings. Google’s documentation changelog shows this note was added on June 15, 2026.

Google’s John Mueller has said the same thing in public posts. In June 2025 he wrote on Bluesky that no AI system was using llms.txt at that time, as reported by Search Engine Roundtable. In 2026 he wrote on Reddit that Google does not use llms.txt, as reported by Search Engine Journal.

These statements are about Google Search. The llms.txt proposal notes that Chrome’s Lighthouse tool now checks sites for the file as part of its checks for AI agent browsing, which is a separate product with a separate goal.

OpenAI, Anthropic and Perplexity

We read the public crawler documentation from OpenAI, Anthropic and Perplexity. All three explain how to allow or block their crawlers with robots.txt. None of them says its crawlers read llms.txt files on other websites.

There is one detail worth knowing. The OpenAI and Perplexity pages each point readers to an llms.txt index of their own documentation, and the llms.txt proposal lists OpenAI, Anthropic and Gemini as publishing llms.txt files for their developer docs. So these companies publish the file for agents that read their documentation. That is different from confirming they use yours when they answer questions.

Should your website have one?

For most business websites, llms.txt is optional. Here is a fair way to weigh it.

Reasons to add one:

  • It is cheap. The proposal notes that tools such as Yoast SEO, AIOSEO and Wix can generate the file automatically.
  • Google says it does no harm.
  • If you publish documentation, product specifications or reference material, coding agents and research agents may make use of it. The proposal says thousands of sites now publish one.

Reasons to wait:

  • Google Search ignores it, so it will not improve rankings or AI Overview citations.
  • No major AI provider has documented reading it from other sites.
  • A stale file is worse than none. If it lists old services or dead links, you are handing agents wrong information.

A sensible order of work for a service business is to fix crawler access, publish clear and accurate pages, and add structured data first. Then add llms.txt if your platform can generate it and you will keep it current.

The short version

llms.txt is a proposed Markdown file that helps AI agents find your best content. Google Search ignores it, and other AI providers have not documented using it for answers. It is low cost and harmless if kept accurate, so add it when it is easy, and put your main effort into crawl access and clear content.

Frequently asked questions

Does Google use llms.txt?
Not in Google Search. Google's guide to generative AI features says Google Search does not use llms.txt files, and that creating one will neither harm nor help your site's visibility or rankings. Google added that note in June 2026. Appearing in AI Overviews or AI Mode depends on being indexed and eligible for a snippet in Google Search.
Is llms.txt the same as robots.txt?
No. A robots.txt file tells crawlers which parts of a site they may access. An llms.txt file is a curated guide written in Markdown that points AI agents to your most useful pages. The llms.txt proposal describes the two as complementary. An llms.txt file does not grant or block access to anything.
What goes in an llms.txt file?
The proposal calls for a Markdown file that starts with a top-level heading naming the site or project, which is the only required part. It can then include a short summary in a blockquote, extra detail, and sections of links to key pages with brief notes. A section titled Optional holds secondary links that an agent can skip.
Will an llms.txt file get my business cited by ChatGPT, Claude or Perplexity?
No provider has documented that. The crawler documentation from OpenAI, Anthropic and Perplexity explains how to allow or block their crawlers with robots.txt and does not say they read llms.txt files on other sites. If you want to be cited, allow their search crawlers and publish clear, original content first.

Sources

Every fact in this article was checked against these pages on November 14, 2026.

  1. The /llms.txt file, v2, llmstxt.org (Jeremy Howard)
  2. Optimizing your website for generative AI features on Google Search, Google Search Central
  3. Latest documentation updates, Google Search Central
  4. Google: No AI System Currently Uses LLMs.txt, Search Engine Roundtable
  5. Google Answers Question About LLMs-Author.txt For SEO, Search Engine Journal
  6. Overview of OpenAI Crawlers, OpenAI
  7. Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic
  8. Perplexity Crawlers, Perplexity

Keep reading