llms.txt is the most oversold file in AI search. It is genuinely worth publishing — it takes an hour and costs nothing — but it is not why ChatGPT will start naming your brand, and any agency telling you otherwise is selling you the cheap part of a hard job.
This guide covers what the file is, what it demonstrably does and does not do, exactly how to write one, where the honest limits sit, and how it fits alongside the work that actually drives citations. Written for UAE brands being sold llms.txt as a shortcut.
What is llms.txt?
The reasoning behind it is sound. Models work within a limited context window, and a modern web page is mostly navigation, scripts and markup wrapped around a small amount of actual content. llms.txt offers a clean, pre-digested version — here is what this organisation does, here are the pages that matter, here is a one-line summary of each.
Think of it as an executive summary written for machines. Useful, cheap, sensible. Not magic.
llms.txt vs robots.txt vs sitemap.xml
These three get conflated constantly, and only one of them is actually enforced:
| File | Question it answers | Honoured by |
|---|---|---|
robots.txt | May you crawl this? | Widely respected, including by major AI crawlers |
sitemap.xml | What pages exist here? | Universally supported by search engines |
llms.txt | What matters here, and what is it about? | No formal commitment from any major provider |
That last row is the one vendors leave out. If someone sells you llms.txt as the reason ChatGPT will start recommending you, they are either misinformed or hoping you are. robots.txt is the file that actually determines whether you are visible — get that wrong and nothing else matters.
How to write a good one
The format is deliberately simple: an H1 with your organisation name, a blockquote summary, then H2 sections containing Markdown links with short descriptions.
What it will not do
It will not make you rank — there is no ranking to enter. It will not override crawler blocks: if robots.txt or your WAF refuses the bot, the file is never read. It will not manufacture authority, because authority comes from what other sites say about you. And it will not fix contradictory facts elsewhere; models weigh corroboration across many sources, not one self-published summary.
So is it worth doing?
Yes — with correct expectations. It costs an hour, it forces you to articulate your entity precisely (valuable in itself), it may help retrieval systems that do read it, and if adoption grows you are already there. Just place it correctly in the hierarchy of what matters:
| Priority | Work | Impact on AI visibility |
|---|---|---|
| 1 | Crawler access — robots.txt, WAF, bot mitigation | Decisive. Blocked means invisible. |
| 2 | Entity consistency across every source | Very high. Contradictions cause hedging. |
| 3 | Extractable answer blocks | Very high. Unquotable content is never quoted. |
| 4 | Third-party corroboration | High, and the hardest to fake. |
| 5 | Structured data | Moderate. Removes excuses to skip you. |
| 6 | llms.txt | Low but positive. Cheap, so do it — last. |
Our own llms.txt is public if you want a working reference. The rest of the stack is covered in our AEO service and GEO agency pages.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed standard — a plain-text, Markdown-formatted file placed at your website's root that gives language models a curated summary of your site and links to your most important pages. It was proposed by Jeremy Howard of Answer.AI in September 2024. It is a convention rather than an enforced specification: no major AI provider has formally committed to honouring it, so it should be treated as a low-cost supporting measure, not a ranking mechanism.
Does llms.txt actually improve AI visibility?
Modestly at best, and only when the fundamentals are already in place. No major AI provider guarantees it is read. It cannot override a crawler block, it cannot create authority, and it cannot resolve contradictory facts published elsewhere. It is worth doing because it costs an hour and forces you to articulate your entity precisely — but crawler access, entity consistency, extractable content and third-party corroboration matter far more.
What is the difference between llms.txt and robots.txt?
They answer different questions. robots.txt tells crawlers whether they may access your site, and it is widely respected including by major AI crawlers — get it wrong and you become invisible. llms.txt tells language models what your site is about and which pages matter, and no provider is obliged to read it. robots.txt is enforcement; llms.txt is a suggestion.
How do I write an llms.txt file?
Use Markdown: an H1 with your organisation name, a blockquote summarising what you do and where, then H2 sections listing your important pages as Markdown links, each with a one-sentence description of the question that page answers. State your core facts — address, founding year, services — exactly once and make sure they match your schema, site copy and Google Business Profile. Curate ruthlessly; a dump of every URL defeats the purpose.
Where should llms.txt be placed?
At the root of your domain, served as plain text — for example https://www.dubaiprod.com/llms.txt. Some sites also publish an llms-full.txt containing expanded content. Keep it current: a stale file is worse than none, because it teaches models outdated facts with unwarranted confidence.
Will llms.txt make ChatGPT recommend my business?
No, not by itself. Being recommended depends on ChatGPT being able to crawl you, finding content it can quote cleanly, and — most importantly — seeing your brand described consistently and favourably across independent third-party sources. llms.txt supports the first two marginally and does nothing for the third. Treat it as the last item on the checklist, not the first.