Short answer: llms.txt is a plain Markdown file at the root of a website that gives AI models a short summary of the site and a list of its most important pages. It is a proposed standard, not an official one. It is genuinely useful for documentation that AI coding tools read, but there is no evidence that it improves rankings or citations in ChatGPT, Google AI Overviews or other AI search engines.
Few files have been hyped as much in AI search as llms.txt. Some guides call it “robots.txt for AI” and promise it will get you cited by ChatGPT. The reality is more modest, and more useful to know before you spend time on it. This guide covers what llms.txt is, what the evidence says about whether it works, and how to create one properly if you decide to.
What llms.txt is
llms.txt is a proposal published by Jeremy Howard, co-founder of Answer.AI, in September 2024 at llmstxt.org. The idea: websites are built for humans, with navigation, scripts and layout that make them slow and noisy for a language model to read. A single, clean Markdown file at /llms.txt can tell a model what the site is and where the important content lives, in a format it can read in one pass.
The file lives at the root of your domain, the same place as robots.txt:
https://example.com/llms.txtThe name is llms.txt, with an s. You will see “llm.txt” in searches and some guides, but that name isn’t part of the proposal.
llms.txt vs robots.txt vs sitemap.xml
The “robots.txt for AI” label is misleading. The three files do different jobs:
| File | Purpose | Who reads it |
|---|---|---|
| robots.txt | Tells crawlers what they may and may not crawl | Search engines and AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) |
| sitemap.xml | Lists every URL you want indexed | Search engines |
| llms.txt | Summarizes the site and points to key pages in readable Markdown | AI tools and agents that choose to look for it |
llms.txt doesn’t control access. If you want to allow or block AI crawlers, that is still done in robots.txt.
The llms.txt format
The specification is short. In order, the file contains:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary of what the site is.
- Optional paragraphs or lists with more detail (any Markdown except headings).
- H2 sections with lists of links, each written as
[name](url)and optionally followed by a colon and a short note. - An optional section called “Optional” for secondary links that a tool can skip if it needs to save space.
The proposal also suggests offering clean Markdown versions of important pages at the same URL with .md added, so tools can fetch a page without the HTML around it.
Some documentation sites also publish llms-full.txt: one file with the entire documentation in Markdown. That is a common convention, not part of the core proposal.
Who actually uses llms.txt
Plenty of serious companies publish one. Anthropic, Stripe, Cloudflare and Vercel all do, mostly for their developer documentation. Stripe’s even includes direct instructions for AI coding agents, such as checking for the latest package version instead of relying on memory.
That points to where llms.txt is most useful today: AI coding assistants and agents that a developer points at a documentation site, or that look up docs while working. For them, a clean map of the docs saves time and tokens.
Publishing a file is not the same as AI search engines reading it, though. That is where the evidence gets thin.
Does llms.txt work for AI search?
For getting cited in AI search, the evidence so far says no:
- Google doesn’t use it. Google’s documentation on AI features in Search states: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” Google representatives have also said publicly that Google doesn’t support llms.txt.
- The major AI crawlers aren’t documented as reading it. OpenAI, Anthropic and Perplexity document how their crawlers follow robots.txt. None of them document using llms.txt to decide what to cite. Independent log analyses report few or no requests for the file from AI crawlers.
- No measurable effect on citations. SE Ranking analyzed about 300,000 domains and found no clear relationship between having an llms.txt file and how often a site was cited by AI systems. In their model, removing the llms.txt signal actually made predictions slightly more accurate.
That matches how AI search works. Engines like ChatGPT search, Perplexity and AI Overviews find sources by searching the web and reading the pages that rank. They cite pages that are relevant, trusted and easy to quote. A summary file sitting at your root doesn’t change any of those.
Should you add one anyway?
Our view: yes if it’s cheap, no if it’s a project.
Add one if:
- You run documentation, an API or a developer product. AI coding tools are the clearest real use.
- You can produce a good file in under an hour, as part of keeping your site’s basic facts consistent.
- You want to be ready if AI platforms adopt it later. The cost of being early is close to zero.
Don’t prioritize it if it would take time away from what actually drives AI visibility: content that answers buyer questions clearly, presence on the third-party sources AI engines read, and consistent company information everywhere. We cover those in what is generative engine optimization.
And be skeptical of anyone selling llms.txt as an AI ranking factor.
How to create an llms.txt file
- Write a one-sentence summary of who you are and what you do, in plain language.
- Pick the 10 to 30 pages that matter most: services or products, pricing, key guides, about and contact. Don’t list every URL; that is what your sitemap is for.
- Group them under H2 headings, with a one-line note on each link.
- Put secondary links under “Optional”.
- Save it as
llms.txtat your domain root and check that it loads as plain text atyourdomain.com/llms.txt. - Keep it current. An outdated file that contradicts your site is worse than none.
Here is a shortened version of our own llms.txt:
# Shmeer Digital
> Shmeer Digital is a full-service digital marketing agency founded in 2016.
> It specializes in generative engine optimization (GEO), SEO, paid media,
> web design and local SEO, and built InfluenceLayer, a platform that
> measures how AI systems describe and recommend brands.
## Services
- [Generative Engine Optimization](https://shmeer.com/services/generative-engine-optimization): getting brands cited by ChatGPT, Perplexity, Gemini and AI Overviews
- [SEO & Content](https://shmeer.com/services/seo): technical SEO, content and link building
## Guides
- [What Is Generative Engine Optimization?](https://shmeer.com/blog/what-is-generative-engine-optimization): definition and playbook
## Optional
- [Privacy policy](https://shmeer.com/privacy)On WordPress, several SEO plugins can now generate the file for you. On other platforms, upload it as a static file at the root.
The bottom line
llms.txt is a sensible idea with real value for documentation and AI coding tools, and a five-minute job for most sites. It isn’t a shortcut to AI visibility. If you want ChatGPT, Perplexity and Google’s AI to name you, the work is in your content, your reputation across the web, and measuring what AI actually says about you. That is what our generative engine optimization services focus on.