This article is my data-informed answer and opinion on:<\/p>\n
- \n
- What llms.txt actually is (and what it isn\u2019t).<\/li>\n
- How it fits with robots.txt<\/a> and sitemap.xml<\/a>.<\/li>\n
- What real-world logs and public statements tell us so far.<\/li>\n
- Where it genuinely adds value (and where it doesn\u2019t).<\/li>\n
- How I\u2019d decide whether to implement it.<\/li>\n<\/ul>\n
Let’s dive in.<\/p>\n
1. What is llms.txt? The short, non-hype definition<\/h2>\n
llms.txt is a small Markdown file<\/b> served at https:\/\/yourdomain.com\/llms.txt that lists a curated set of your most important pages<\/b>, each with a short description, in a tightly defined format.
\n![\"Llms.txt](\"https:\/\/www.similarweb.com\/blog\/wp-content\/uploads\/2025\/12\/attachment-llms-txt-definition.png\")
<\/p>\nThe llms.txt file is designed primarily for inference time <\/b>(i.e., when an LLM or agent is actively answering a user\u2019s question), not for training or broad web indexing.<\/p>\n
If you like analogies:<\/p>\n
- \n
- robots.txt: \u201cHere is where you can\u2019t<\/b> crawl.\u201d<\/li>\n
- sitemap.xml: \u201cHere is everything you could<\/b> crawl.\u201d<\/li>\n
- llms.txt: \u201cIf you\u2019re trying to answer questions about us, start with these pages<\/b>.\u201d<\/li>\n<\/ul>\n
That last line is the key.<\/p>\n
As SEOs, we\u2019re used to influencing discovery and indexing. Llms.txt tries to influence prioritization<\/b> under a hard constraint: an LLM can\u2019t load and understand your entire site on every query. You can think of it as a type of “crawl budget”.<\/p>\n
1.1. How llms.txt is structured, and why that matters<\/h3>\n
The official spec<\/a> (proposed by Jeremy Howard\/Answer.AI) defines a strict, simple structure:<\/p>\n
- \n
- H1 title (#)<\/b>\n
- \n
- The name of your site (the only required element).<\/li>\n
- For an LLM, this is the top-level entity label: \u201cEverything below is about this<\/i>.\u201d<\/li>\n<\/ol>\n<\/li>\n
- Blockquote summary (>)<\/b>\n
- \n
- A one\/two-sentence summary of what you do and who you serve.<\/li>\n
- Gives models immediate context even if they only skim the file.<\/li>\n<\/ol>\n<\/li>\n
- Optional explanatory paragraphs<\/b>\n
- \n
- Used to clarify scope or versioning (e.g., \u201cThis file covers v2 of our public API docs only\u201d).<\/li>\n
- Helpful when your docs, products, or sites have multiple generations or audiences.<\/li>\n<\/ol>\n<\/li>\n
- H2 sections (##) that group links by theme<\/b>\n
- \n
- For example: ## Documentation, ## Guides, ## Product, ## Support.<\/li>\n
- Mirrors how LLMs tend to reason in topical \u201cbuckets,\u201d not in random URL lists.<\/li>\n<\/ol>\n<\/li>\n
- Bulleted lists under each H2 with Markdown links<\/b>\n
- \n
- Each bullet looks like:
\n– [API reference](https:\/\/example.com\/docs\/api): Endpoints, parameters, and examples<\/li>\n - That combination of anchor text<\/b>, URL<\/b>, and short factual description<\/b> gives the model strong hints about what each page is for and when to use it.<\/li>\n<\/ol>\n<\/li>\n
- Optional ## Optional section<\/b>\n
- \n
- <\/b>A reserved section name. Links here are explicitly lower priority.<\/li>\n
- Tools are free to drop these first if the context is tight.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
Example of a simple llm.txt file:<\/h4>\n
![\"Llms.txt](\"https:\/\/www.similarweb.com\/blog\/wp-content\/uploads\/2025\/12\/attachment-example-llmstxt.png\")
<\/p>\nFrom an SEO and AEO<\/a> point of view, this structure matters because it forces you to:<\/p>\n
- \n
- Identify your real \u201csource of truth\u201d pages<\/b>, instead of assuming AI will magically find them.<\/li>\n
- Describe pages in concise, factual language<\/b>, which is exactly the kind of text LLMs are good at classifying and routing with.<\/li>\n<\/ul>\n
You\u2019re not \u201coptimizing the llms.txt file\u201d for rankings, you\u2019re making your content model legible to machines<\/b>.<\/p>\n
1.2. llms-full.txt and .md page variants<\/h3>\n
Around this spec, a small ecosystem has emerged:<\/p>\n
- \n
- llms-full.txt<\/b>\n
- \n
- A single Markdown file containing the full text of your documentation corpus.<\/li>\n
- Used as a convenient ingestion endpoint for AI tools and coding agents.<\/li>\n
- Docs platforms (like Mintlify) auto-generate this so agents can pull an entire docs set from one URL.<\/li>\n<\/ul>\n<\/li>\n
- .md versions of docs pages<\/b>\n
- \n
- Each docs URL also has a Markdown variant, typically by appending .md:\n
- \n
- https:\/\/example.com\/docs\/api \u2192 HTML for humans<\/li>\n
- https:\/\/example.com\/docs\/api.md \u2192 Markdown for AI<\/li>\n<\/ul>\n<\/li>\n
- The llms.txt spec encourages this pattern. nbdev-based projects under Answer.AI already generate Markdown docs by default.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Why this matters:<\/strong><\/p>\n
- \n
- Markdown strips layout noise (nav, ads, tracking junk) that wastes tokens.<\/li>\n
- That reduces cost<\/b> and truncation risk<\/b>, which is essential when agents operate inside finite context windows.<\/li>\n<\/ul>\n
If you\u2019re a dev platform<\/b> or docs-heavy product<\/b>, this ecosystem already improves agent reliability and developer experience, regardless of whether Google or OpenAI ever officially \u201cuse\u201d llms.txt.<\/p>\n
2. llms.txt vs robots.txt vs sitemap.xml<\/h2>\n
Because llms.txt lives at the domain root and ends in .txt, it\u2019s easy for stakeholders to assume it works like robots.txt. It doesn\u2019t.<\/p>\n
2.1. Three files, three different jobs<\/h3>\n
In plain language:<\/p>\n
- \n
- robots.txt: access control<\/b>\n
- \n
- \u201cWhere can crawlers go?\u201d<\/li>\n
- Manages what is and isn\u2019t crawlable (including some AI-training opt-outs).<\/li>\n<\/ul>\n<\/li>\n
- sitemap.xml: discovery and coverage<\/b>\n
- \n
- \u201cWhat exists on this site?\u201d<\/li>\n
- Lists indexable URLs plus metadata like last-modified dates.<\/li>\n<\/ul>\n<\/li>\n
- llms.txt: curation and interpretation<\/b>\n
- \n
- \u201cIf you only have room for a subset of pages, which ones matter most, and what are they for?\u201d<\/li>\n
- Doesn\u2019t block anything, and it doesn\u2019t list everything. It highlights a small set of high-signal pages and labels them clearly.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
The spec explicitly states that llms.txt coexists with <\/b>(not replaces) robots.txt and sitemap.xml.<\/p>\n
2.2. Why the distinction matters<\/h3>\n
Mis-framing llms.txt leads to bad expectations:<\/p>\n
- \n
- It does not<\/b> control crawling or training behavior.<\/li>\n
- It does not<\/b> override robots.txt or meta directives.<\/li>\n
- It does not<\/b> serve as an \u201cAI XML sitemap\u201d.<\/li>\n<\/ul>\n
From a strategy perspective:<\/p>\n
- \n
- robots.txt<\/b> is your guardrail<\/b>.<\/li>\n
- sitemap.xml<\/b> is your catalog<\/b>.<\/li>\n
- llms.txt<\/b> is your curated reading list with notes, written for AI<\/b>.<\/li>\n<\/ul>\n
If you treat llms.txt like a second sitemap and dump 200 URLs into it, you\u2019ve missed the point, and you\u2019re actively making life harder for any agent that tries to use it.<\/p>\n
3. What problem is llms.txt actually trying to solve?<\/h2>\n
To decide if llms.txt is worth your attention, you need to be clear on the friction it addresses.<\/p>\n
3.1. How LLM-based systems \u201csee\u201d your site<\/h3>\n
Classic search engine crawlers:<\/p>\n
- \n
- Continuously crawl and update large slices of the web.<\/li>\n
- Render JavaScript where needed.<\/li>\n
- Maintain long-lived indexes used for ranking.<\/li>\n<\/ul>\n
Many LLM-based systems, especially in \u201cbrowsing\u201d modes or agent setups:<\/p>\n
- \n
- Fetch content at query time<\/b>, not as part of an ongoing crawl.<\/li>\n
- Often, generative engines don\u2019t execute complex JavaScript<\/b>, meaning JS-only content is invisible to them.<\/li>\n
- Operate within strict token windows<\/b>: they may only \u201csee\u201d a fraction of the HTML they fetch.<\/li>\n<\/ul>\n
The result:<\/p>\n
- \n
- Navigation, cookie banners, and clutter often appear before your core content.<\/li>\n
- Long, dense pages can get truncated<\/b>, cutting off key sections.<\/li>\n
- Important docs hidden behind JS-heavy nav structures are poorly understood or missed.<\/li>\n<\/ul>\n
Similarweb\u2019s research on GenAI<\/a> and publishers shows how these dynamics play out: AI Overviews increasingly satisfy user intent<\/a> on the SERP, driving \u201czero-click\u201d behavior and reducing visits to the underlying sites.<\/p>\n
That same \u201canswer-first\u201d pattern applies when LLMs browse your site directly: they\u2019re trying to get in, grab exactly what they need, and get out. Quickly.<\/p>\n
3.2. Typical failure modes from an AI visibility standpoint<\/h3>\n
Because of those constraints, we see failure patterns like:<\/p>\n
- \n
- AI answers that rely on old blog posts rather than updated ones.<\/li>\n
- Third-party explainers outranking your own product guides<\/b> as the cited source.<\/li>\n
- Misstated pricing, limits, or policies<\/b> because the canonical page is long, salesy, or buried.<\/li>\n
- Incomplete or hallucinated API behavior because reference docs are fragmented or noisy.<\/li>\n<\/ul>\n
Traditional SEO is mostly about being discoverable and indexable<\/b>. In the AI era, we also need to be legible and prioritized<\/b> within narrow context windows.<\/p>\n
3.3. llms.txt as a targeted mitigation<\/h3>\n
Llms.txt doesn\u2019t pretend to fix everything. But it does attack one narrow yet important question:<\/p>\n
If a model can only look at a handful of your pages, how do you help it pick the right<\/b> ones and understand what each is for?<\/p>\n
It does that by:<\/p>\n
- \n
- Offering a curated list of high-value URLs<\/b>.<\/li>\n
- Presenting them in Markdown<\/b> to minimize layout noise.<\/li>\n
- Including short, explicit descriptions<\/b> (e.g., \u201cAPI v2 reference,\u201d \u201ccurrent self-serve pricing,\u201d \u201cgetting started guide\u201d).<\/li>\n<\/ul>\n
From a GEO<\/a>\/AEO perspective, llms.txt is less about chasing a ranking factor and more about increasing the odds that, if AI tools use your site, they start with the right content<\/b>.<\/p>\n
\n\n\n\n Try Similarweb Now<\/a>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n\n4. How the SEO industry actually sees llms.txt<\/h2>\n
If you read across SEO blogs, dev docs, and product updates, four distinct positions show up.<\/p>\n
4.1. The skeptics: \u201cNot worth it (yet)\u201d.<\/h3>\n
This point of view is driven by the current adoption reality:<\/p>\n
- \n
- Googlers have publicly stated that no Gen AI system currently uses llms.txt<\/a>, and compared it to the old keywords meta tag<\/a>: something SEOs might obsess over, but that search engines ignore<\/b>.
\n![\"John](\"https:\/\/www.similarweb.com\/blog\/wp-content\/uploads\/2025\/12\/attachment-noai.png\")
<\/li>\n- Early log-file analyses show very few consumer-facing LLM crawlers<\/b> (GPTBot, Google-Extended, PerplexityBot, ClaudeBot) requesting \/llms.txt at any scale.<\/li>\n<\/ul>\n
From that vantage point:<\/p>\n
- \n
- There\u2019s no evidence<\/b> that llms.txt improves rankings, AI Overviews<\/a> presence, or traffic.<\/li>\n
- No major LLM vendor has said \u201cwe treat this as a signal.\u201d<\/li>\n<\/ul>\n
My SEO take:<\/h4>\n
If you\u2019re still dealing with crawl issues, thin content, weak website infrastructure, or fragile technical SEO<\/a>, this camp is right: llms.txt does not belong anywhere near the top of your backlog.<\/p>\n
4.2. The pragmatic futurists: \u201cCheap insurance\u201d.<\/h3>\n
This group agrees the impact is unproven, but looks at the cost differently:<\/p>\n
- \n
- A simple, 5-15 URL llms.txt file takes under an hour<\/b> to draft.<\/li>\n
- Updating it a couple of times a year is negligible compared with most content or dev projects.<\/li>\n
- It\u2019s essentially an option<\/b>: low downside, potential upside if\/when adoption grows.<\/li>\n<\/ul>\n
They also value the internal exercise:<\/p>\n
- \n
- To write a good llms.txt, you must agree on your canonical, source-of-truth pages<\/b>.<\/li>\n
- That often surfaces outdated docs, overlapping content, or internal misalignment (which are issues you should fix anyway).<\/li>\n<\/ul>\n
My SEO take:<\/h4>\n
If your fundamentals are in good shape and you already care about AI visibility and Generative Engine Optimization, I\u2019m aligned with this camp.<\/p>\n
Think of llms.txt as \u201cfuture insurance + content clarity,\u201d<\/b> not as a lever you report on quarterly.<\/p>\n
4.3. Docs & agent champions: \u201cIt\u2019s useful right now\u201d.<\/h3>\n
In developer ecosystems, the discussion is much less theoretical.<\/p>\n
- \n
- Docs platforms generate llms.txt, llms-full, and .md exports out of the box to help coding agents and AI tools<\/b> ingest docs.<\/li>\n
- Anthropic and others highlight LLM-friendly Markdown docs as a best practice for tools and agents to consume.<\/li>\n
- Benchmarks from teams working on code assistants show that AI agents guided by llms.txt-structured docs often outperform<\/b> those that rely solely on semantic search across unstructured HTML. This is another proof that AI agents are only as good as the data integrated with them<\/a>.<\/li>\n<\/ul>\n
Here, the upside is tangible:<\/p>\n
- \n
- Better agent reasoning over your docs.<\/li>\n
- Lower token usage and cost when AI agents<\/a> fetch Markdown instead of full HTML.<\/li>\n
- Fewer AI-related support tickets because \u201cthe AI\u201d is finally reading the right docs.<\/li>\n<\/ul>\n
My SEO take:<\/h4>\n
If your product is developer-first<\/b> or heavily API-based<\/b>, I\u2019d treat llms.txt and its ecosystem as a DX\/docs requirement<\/b>, not an SEO experiment.<\/p>\n
The value shows up in developer adoption and retention, even if it never shows up in \u201corganic sessions\u201d reports.<\/p>\n
4.4. AI SEO\/GEO enthusiasts: \u201cBe the answer\u201d.<\/h3>\n
This camp is focused on the bigger shift from clicks to answers<\/b>.<\/p>\n
- \n
- Similarweb\u2019s GenAI research shows AI chatbot traffic growing fast: in June 2025, major AI platforms generated over 1.1 billion referral visits<\/b>, up 357% year-over-year.<\/li>\n
- At the same time, Similarweb\u2019s reports on publishers show substantial traffic drops for many news sites<\/b> as Google\u2019s AI Overviews satisfy more queries without clicks.<\/li>\n<\/ul>\n
In that world, GEO (Generative Engine Optimization) is about:<\/p>\n
- \n
- Being selected and cited<\/b> inside AI answers, not just ranking in classic SERPs.<\/li>\n
- Making sure generative AI engines describe your brand accurately when they answer in your space.<\/li>\n<\/ul>\n
For this group, llms.txt is one more supporting signal<\/b>:<\/p>\n
- \n
- It doesn\u2019t replace schema, internal links<\/a>, or entity work.<\/li>\n
- It does give you a clean way to say, \u201cThese are our authoritative answers.\u201d<\/li>\n<\/ul>\n
My SEO take:<\/h4>\n
Strategically, I agree with the GEO direction. Tactically, llms.txt is a small, aligned tactic<\/b>, not the core of your AI optimization strategy.<\/p>\n
Strong, straightforward, authoritative content still does most of the heavy lifting.<\/p>\n
5. What the data says (so far)<\/h2>\n
Strip out the hype, and you get a reasonably consistent picture.<\/p>\n
5.1. Adoption is growing, but still niche<\/h3>\n
- \n
- The llms.txt spec has stable documentation and a growing tool ecosystem (CLI tools, plugins for popular doc generators, CMS integrations).<\/li>\n
- Adoption is heavily concentrated in developer tools, SaaS docs, AI-aware agencies, and early GEO experiments<\/b>.<\/li>\n<\/ul>\n
Relative to the entire web, it\u2019s still an early adopter pattern, not a mainstream standard.<\/p>\n
5.2. For classic SEO and AI Overviews, llms.txt is neutral today<\/h3>\n
Across public statements and independent experiments:<\/p>\n
- \n
- There is no evidence<\/b> that llms.txt:\n
- \n
- Improves organic rankings<\/li>\n
- Increases AI Overview inclusion<\/li>\n
- Moves traditional SEO KPIs in a repeatable way<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Today, classic ranking systems still respond to technical health, relevance, authority, and user signals. Not to llms.txt.<\/p>\n
5.3. For agents, tools, and AI-native docs, it already has jobs<\/h3>\n
On the other hand:<\/p>\n
- \n
- Tool builders and docs platforms are already using llms.txt and llms-full.txt as ingestion endpoints<\/b> for LLM-based tools and MCP servers<\/a>.<\/li>\n
- That doesn\u2019t register as \u201cmore organic traffic,\u201d but it does show up as:\n
- \n
- Better agent-driven onboarding<\/li>\n
- More accurate AI-generated examples<\/li>\n
- Less friction when developers use AI assistants<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
If your product\u2019s success depends on developers understanding your docs via AI tools, this matters.<\/p>\n
6. Should you implement llms.txt? A practical framework<\/h2>\n
With so many pros and cons to llms.txt, SEOs need to weigh the benefits it can bring to their site vs. the potential time waste of a 0-impact project. We don\u2019t always have all the resources we need to carry out SEO tasks, so why add tasks that don\u2019t result in more traffic or revenue for us?<\/p>\n
Let\u2019s get concrete, here\u2019s how I\u2019d prioritize it:<\/p>\n
![\"Llms.txt](\"https:\/\/www.similarweb.com\/blog\/wp-content\/uploads\/2025\/12\/attachment-llms-txt-implementation-priorities.png\")
<\/p>\n6.1. High priority: dev platforms and docs-heavy products<\/h3>\n
You should strongly consider<\/b> llms.txt (plus llms-full and .md docs) if:<\/p>\n
- \n
- You\u2019re API-first or developer-first.<\/li>\n
- You maintain a substantial public docs site.<\/li>\n
- Your users already rely on:\n
- \n
- IDE assistants<\/li>\n
- Embedded AI in docs<\/li>\n
- Agents that fetch docs via HTTP<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
In that context:<\/p>\n
- \n
- llms.txt is part of building AI-native documentation<\/b>, not a speculative traffic play.<\/li>\n
- It\u2019s aligned with how LLM-aware dev tooling is evolving.<\/li>\n<\/ul>\n
My SEO take:<\/h4>\n
Own this as a product\/docs initiative<\/b>, with the SEO team as a stakeholder.<\/p>\n
Measure success via developer outcomes (time-to-first-success, reduced support burden), not ranking charts.<\/p>\n
6.2. Medium priority: mature, content-rich sites exploring GEO<\/h3>\n
You should consider a lightweight llms.txt<\/b> if:<\/p>\n
- \n
- Your technical SEO and content fundamentals are solid.<\/li>\n
- You have clear \u201cpillar<\/a>\u201d content and stable product docs.<\/li>\n
- You\u2019re actively tracking AI behavior:\n
- \n
- Using Similarweb\u2019s AI traffic tracker<\/b> ( in the AI Search Intelligence<\/b><\/a> suite) to see which chatbots send traffic to which URLs.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Here, a small llms.txt is:<\/p>\n
- \n
- A low-cost experiment<\/b>.<\/li>\n
- A forcing function to clarify your 5-15 actual source-of-truth pages<\/b>.<\/li>\n
- A nice complement to your GEO\/AEO<\/a> initiatives.<\/li>\n<\/ul>\n
My SEO take:<\/h4>\n
Invest ~45-60 minutes to create a curated llms.txt, then review it a few times a year alongside your normal content audits. Don\u2019t sell it internally as a \u201cgrowth lever\u201d, present it as readiness + modeling clarity<\/b>.<\/p>\n
6.3. Low priority: sites still fighting the basics<\/h3>\n
You should not prioritize<\/b> llms.txt yet if:<\/p>\n
- A forcing function to clarify your 5-15 actual source-of-truth pages<\/b>.<\/li>\n
- A low-cost experiment<\/b>.<\/li>\n
- You\u2019re actively tracking AI behavior:\n
- It\u2019s aligned with how LLM-aware dev tooling is evolving.<\/li>\n<\/ul>\n
- llms.txt is part of building AI-native documentation<\/b>, not a speculative traffic play.<\/li>\n
- That doesn\u2019t register as \u201cmore organic traffic,\u201d but it does show up as:\n
- Tool builders and docs platforms are already using llms.txt and llms-full.txt as ingestion endpoints<\/b> for LLM-based tools and MCP servers<\/a>.<\/li>\n
- There is no evidence<\/b> that llms.txt:\n
- It does give you a clean way to say, \u201cThese are our authoritative answers.\u201d<\/li>\n<\/ul>\n
- Making sure generative AI engines describe your brand accurately when they answer in your space.<\/li>\n<\/ul>\n
- At the same time, Similarweb\u2019s reports on publishers show substantial traffic drops for many news sites<\/b> as Google\u2019s AI Overviews satisfy more queries without clicks.<\/li>\n<\/ul>\n
- Fewer AI-related support tickets because \u201cthe AI\u201d is finally reading the right docs.<\/li>\n<\/ul>\n
- That often surfaces outdated docs, overlapping content, or internal misalignment (which are issues you should fix anyway).<\/li>\n<\/ul>\n
- No major LLM vendor has said \u201cwe treat this as a signal.\u201d<\/li>\n<\/ul>\n
- Early log-file analyses show very few consumer-facing LLM crawlers<\/b> (GPTBot, Google-Extended, PerplexityBot, ClaudeBot) requesting \/llms.txt at any scale.<\/li>\n<\/ul>\n
- Presenting them in Markdown<\/b> to minimize layout noise.<\/li>\n
- Misstated pricing, limits, or policies<\/b> because the canonical page is long, salesy, or buried.<\/li>\n
- Important docs hidden behind JS-heavy nav structures are poorly understood or missed.<\/li>\n<\/ul>\n
- Often, generative engines don\u2019t execute complex JavaScript<\/b>, meaning JS-only content is invisible to them.<\/li>\n
- Fetch content at query time<\/b>, not as part of an ongoing crawl.<\/li>\n
- sitemap.xml<\/b> is your catalog<\/b>.<\/li>\n
- It does not<\/b> override robots.txt or meta directives.<\/li>\n
- It does not<\/b> control crawling or training behavior.<\/li>\n
- robots.txt: access control<\/b>\n
- Each docs URL also has a Markdown variant, typically by appending .md:\n
- Describe pages in concise, factual language<\/b>, which is exactly the kind of text LLMs are good at classifying and routing with.<\/li>\n<\/ul>\n
- Tools are free to drop these first if the context is tight.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
- Optional ## Optional section<\/b>\n
- Each bullet looks like:
- Blockquote summary (>)<\/b>\n
- sitemap.xml: \u201cHere is everything you could<\/b> crawl.\u201d<\/li>\n