llms.txt: What It Is, and Whether It Actually Does Anything

‍ llms.txt is a plain Markdown file placed at the root of a domain that gives AI systems a curated map of a site’s most useful content. It does not improve visibility in AI search results, and Google has confirmed it plays no part in AI Overviews or AI Mode. It does have a real and growing use: making a site legible to AI agents and coding assistants.

The gap between what llms.txt is sold as and what it does is now one of the widest in technical SEO. Search volume for the term has climbed steadily, and most of the articles answering that demand still promise better AI search visibility. The data says otherwise, and there is enough of it now to be definite rather than cautious. This piece covers what the file is, what the evidence shows, the one case where it genuinely earns its keep, and how to add it to a Webflow site without creating a duplicate content problem. It sits inside Team 4’s wider guide to generative engine optimisation.

What is llms.txt?

llms.txt is a Markdown file served at the root of a domain, at /llms.txt, listing a site’s most important pages with short descriptions so an AI system can find the useful content without parsing the whole site. It was proposed in September 2024 by Jeremy Howard of Answer.AI, borrowing the placement convention from robots.txt and the intent from a sitemap.

The technical argument behind it is sound. HTML is noisy to a language model: navigation, scripts, cookie banners and tracking pixels all consume context window before the model reaches the sentence that matters. Serving clean Markdown instead cuts that waste substantially, with some companies reporting token reductions of up to 10x on documentation content.

There is a companion format, llms-full.txt, which contains a site’s full content in a single Markdown document rather than a curated index. Most sites have no need for it. Documentation-heavy SaaS products are the exception.

One thing worth being clear about: llms.txt is a community convention, not a standard. It has no backing from the W3C, the IETF or any recognised standards body, and no enforcement mechanism. Providers who support it do so on their own terms, which is exactly why the picture is so inconsistent.

Does llms.txt improve AI search visibility?

No, and this is now documented rather than debated. If your reason for adding llms.txt is to get cited more often by ChatGPT, Claude, Perplexity or Google AI Mode, the evidence does not support it.

Google has said so directly. Its AI optimisation guidance, published in May 2026, tells site owners that llms.txt is not needed for AI Overviews, AI Mode or any other generative search feature, and groups it with content chunking and AI-specific rewriting as tactics that do not help (Google Search, 2026). John Mueller has been blunter, comparing the file to the old keywords meta tag and pointing out that server logs show AI services do not even request it.

His underlying objection is the interesting part, because it is architectural rather than tactical. A model that has already retrieved your actual pages has no particular reason to trust a separate file describing those pages, especially when nothing stops a site listing one set of content in llms.txt while serving something different to visitors. That is cloaking with a new name, and any provider building retrieval infrastructure has to assume it.

The measurement data agrees. A study of 300,000 domains found an adoption rate of 10.13%, and a machine learning model built to predict AI citation frequency became more accurate when llms.txt was removed as a variable, meaning the file contributed noise rather than signal (SE Ranking, 2026). Crawler monitoring points the same way: across more than 500 million AI bot visits in a 90-day window, only 408 requests targeted llms.txt.

Roughly one site in ten has the file, and no measurable citation benefit has been found for any of them. If you have added llms.txt and seen no change in AI visibility, nothing went wrong with your implementation.

So why would you add llms.txt at all?

llms.txt earns its place as agent infrastructure rather than as a search tactic, and that distinction is where nearly all the confusion sits. Discoverability by an agent and visibility in an answer are two different jobs, and the file only does the first.

The agent-facing case is real and being actively built on. Anthropic recommends llms.txt in its guidance on writing for agents. OpenAI uses it for the Agents SDK and its Agentic Commerce Protocol. Chrome’s Lighthouse added an agentic browsing audit in May 2026 that checks whether a site provides the file. Coding assistants including Claude Code, Cursor, Copilot and Windsurf fetch documentation in real time, and a good llms.txt gets them to the right page with far less token waste.

For a B2B SaaS company, that translates fairly directly. If you have product documentation, an API reference, a help centre or integration guides, developers evaluating you are increasingly reading them through an assistant rather than a browser. Making those pages cheap and accurate to retrieve is a product experience question as much as a marketing one.

If you sell to non-technical buyers and have no documentation to speak of, the honest answer is that llms.txt will do very little for you. Add it because it takes half a day and costs nothing to maintain, not because it will change your pipeline.

How to set up llms.txt properly

1. Decide what belongs in it

Curate, do not dump. The file should list the pages that answer real questions about your product: documentation, integration guides, pricing, security and compliance information, and your main category explainers. Each entry gets a link and a one-line description of what the page covers. A file listing 400 blog posts defeats the purpose, since the whole point is reducing what the model has to wade through.

2. Write it as Markdown, not HTML

The format is an H1 with your site name, an optional blockquote summary, then Markdown link lists grouped under H2 headings by section. Keep the descriptions factual and specific. This is not a place for positioning copy, and a model reading it has no interest in your value proposition.

3. Serve it at the root, uncached and as plain text

It has to resolve at yourdomain.com/llms.txt with a text/plain or text/markdown content type. On Webflow, the usual route is a Cloudflare Worker or a reverse proxy rule, since Webflow does not let you publish arbitrary root files directly. Team 4 handles this the same way as other root-level files on client sites, and the Webflow development service page covers the wider setup.

4. Keep the Markdown copies out of the index

This is the mistake that does actual damage. A popular implementation generates an individual Markdown copy of every page on the site. If those files are indexable, you have created large-scale duplicate content, diluting crawl budget and potentially suppressing the original pages. Block them in robots.txt, or serve them with a noindex header, and check the result rather than assuming.

5. Diarise a review

An llms.txt describing last year’s documentation is worse than none, because it points agents confidently at the wrong pages. Add it to whatever cadence you already use for sitemap and redirect checks.

Common llms.txt mistakes to avoid

• Expecting a citation lift. The evidence says there is none. Judge it on agent retrieval quality instead.
• Listing everything. A curated 30-link file beats an exhaustive 400-link one, because reducing noise is the entire mechanism.
• Letting the Markdown copies get indexed. The one implementation choice that can actively harm your organic performance.
• Treating it as a substitute for the work that does matter. Direct-answer content structure, verifiable specifics, consistent entity naming and technical retrievability are what change citation rates. None of them involve this file.
• Publishing it once and forgetting it. A stale file misdirects the agents it was meant to help.

How Team 4 approaches llms.txt

Team 4 treats llms.txt as a half-day infrastructure task, not a service line. On client sites it goes in alongside the technical retrievability work that actually moves AI citations: server-rendered content, accurate structured data, sensible AI crawler handling and consistent entity descriptions across the site and third-party sources.

We will tell a client when something is unlikely to move a number. This is one of those cases, and it is a useful test of any supplier pitching AI search work. If a proposal leads with llms.txt as a visibility tactic, ask them for the citation data behind it. The honest version of the recommendation is that the file is cheap, occasionally useful for agents, and not where the pipeline comes from. For where it does come from, see how to track AI search visibility and improving your brand presence in AI.

Read more: generative engine optimisation

This article is part of Team 4’s complete guide to generative engine optimisation.

• Back to: Generative Engine Optimisation: The Definitive Guide for B2B SaaS
• Next: RAG in SEO, how retrieval actually works
• GEO vs AEO vs SEO vs LLM optimisation

About Team 4

Team 4 is a specialist B2B SaaS marketing agency based in London, working with SaaS start-ups and scale-ups globally. The agency builds Inbound Engines®: compounding organic growth systems that turn search and AI visibility into pipeline. Core services include SEO, GEO, PPC, Webflow development and content. No account managers. The strategists do the work.