| TL;DR |
| llms.txt is a plain-text file at your root domain that tells AI models what your site is, what matters, and what to skip. Same concept as robots.txt different audience, different purpose. 73% of top 10K sites don’t have one yet. Early adopters see 2.4× better AI citation accuracy. Takes 20 minutes to implement. The window is open now. |
A quick history lesson
In 1994, Martijn Koster invented robots.txt to tell Webcrawler which pages not to index. It became the de facto standard within months not because a standards body ratified it, but because it solved a real problem simply. One file, at a predictable location, with a readable format. Every crawler adopted it. Thirty years later it’s still running on virtually every website on the internet.
We’re at the same inflection point again. AI systems ChatGPT Search, Perplexity, Claude, Gemini are crawling and reading your site to generate answers. But unlike traditional search engines, they don’t just want to know where to go. They want to understand what you are. Your about page, your structured data, and your meta descriptions were never written with that question in mind. They were written for humans and for keyword matching not for a language model trying to form an accurate, confident summary of your business.
llms.txt fills that gap. It’s a brief, human-readable markdown file that gives AI systems the context they need to represent your brand accurately in generated answers. Same location logic as robots.txt root domain, plain text, instantly accessible but the content is entirely different.
| Key insight: robots.txt controls access. llms.txt controls understanding. Both matter but only one of them exists on most sites right now. |
What’s different from robots.txt
The comparison is intuitive but the distinction matters. robots.txt is an access control file. It tells crawlers: go here, don’t go there. It communicates nothing about meaning, priority, or context. A crawler that obeys your robots.txt still has no idea what your company actually does.
| robots.txt | llms.txt | |
| Purpose | Controls crawler access | Controls AI understanding |
| Format | Allow / Disallow rules | Plain markdown prose |
| Audience | Search engine bots | Large language models |
| What it says | Where to go or not go | What your site means |
| Age | 30+ years, universal standard | Emerging early adopter edge |
| Adoption | ~100% of serious websites | ~27% of top 10K sites |
llms.txt is descriptive rather than directive. You’re not restricting anything you’re providing a curated self-description that helps AI systems skip the guesswork and represent you accurately. Think of it as the brief you’d hand a journalist before an interview: here’s who we are, here’s what matters, here’s what not to confuse us with.
| 73%of top 10K websites have no llms.txt as of April 2026 | 2.4×higher AI citation accuracy for sites with llms.txt vs without | 20 minaverage time to write and deploy a basic llms.txt file |
What it looks like in practice
The format is intentionally simple plain markdown, no schema, no JSON-LD, no developer required. Here’s a minimal but effective example:
That’s the whole file. Drop it at your root domain (yourdomain.com/llms.txt), make sure it’s publicly accessible with no authentication wall, and you’re done with the technical implementation. The harder part is writing it well.
A good llms.txt is honest, specific, and brief. The “About” section should be two to three sentences that describe what you do and who you do it for the kind of description you’d give at a conference in thirty seconds. The “Key pages” section tells AI retrieval systems which URLs carry your most important, citable content. The “Do not summarise” section is optional but useful for pages where AI-generated summaries could be misleading pricing pages, legal terms, anything time-sensitive or context-dependent.
| Pro tip: Write your llms.txt description the same way you’d brief a journalist assume they’re smart, short on time, and need to get your positioning right on the first read. |
Why the window is open right now
In 2004, adding a sitemap.xml gave you a measurable edge over competitors who hadn’t yet. That edge lasted about 18 months before it became table stakes. The same curve applied to schema markup, to mobile optimisation, to Core Web Vitals. There’s always a window between “early adopter advantage” and “everyone has it.”
llms.txt is at the start of that curve. AI answer engines are still in the phase where they weight explicit guidance heavily because implicit signals like links, domain authority, and content age haven’t fully translated to their retrieval models yet. A clear, well-written llms.txt is currently one of the highest-signal inputs you can give them.
Our audit data across 4,200 domains shows sites with a well-structured llms.txt receive 2.4 times more accurate brand representation in AI-generated answers compared to sites without one. That gap will narrow as AI systems get better at inferring context from existing content but right now, explicit beats implicit every time.
The 73% of top 10K sites that don’t have an llms.txt aren’t making a deliberate choice. Most simply haven’t heard of it yet. That’s your window.
FAQ
Is llms.txt an official standard?
Not yet it was proposed by Answer.AI in late 2024 and has been adopted informally by a growing number of sites and AI platforms. It doesn’t require ratification to be useful; robots.txt wasn’t an official standard for years either, and it became universal through adoption, not decree.
Will AI models actually read it?
Perplexity and several other retrieval-augmented AI systems have confirmed they check for llms.txt. Claude and ChatGPT’s retrieval systems are expected to follow as the format gains traction. Even if a model doesn’t read it directly today, publishing one signals intentionality and that matters as these systems evolve and look for structured signals.
Does llms.txt replace structured data or schema markup?
No, they’re complementary. Schema helps traditional search engines understand your content at a granular, structured level. llms.txt gives AI systems narrative context and priorities that schema can’t express. Use both: schema for precision, llms.txt for orientation.
How do I know if my llms.txt is working?
The signal is in your AI citation accuracy whether AI systems describe your product, services, and positioning correctly when answering queries about your space. Run an AI citation audit before you publish your llms.txt to establish a baseline, then re-audit four to six weeks after to measure the delta.
| Check your AI visibility nowFind out if your site has an llms.txt, how AI systems currently describe your brand, and what to fix in under 2 minutes.→ Run your own audit at serp.fyi/register |
