- llms.txt is a Markdown file at your site's root that gives AI a curated, structured map of your most important content. Proposed by Jeremy Howard in September 2024, it is used at inference time when users ask AI tools about your site, not for training.
- It is not a Google ranking signal, and no major AI company (OpenAI, Google, Anthropic) has officially confirmed they read it during crawls. Semrush's server log test showed zero visits from major AI bots from mid-August to late October 2025.
- WordPress has the easiest path: Yoast SEO ships native llms.txt generation in Site Features. Webflow allows direct upload on CMS and Business plans. Shopify has no root-directory access, so requires a CDN-upload + redirect workaround, or a dedicated app.
- Audit before deploying: do not dump your sitemap into llms.txt. Include homepage, About, core product pages, pricing, key docs, and your 3-5 best-performing posts. Exclude legal pages, outdated content, login flows, and aggressive sales copy.
- Update on a quarterly cadence and after any major site-structure change. An outdated llms.txt with deprecated URLs is worse than no file at all.
What llms.txt actually is, and what it isn't
llms.txt is a Markdown file at the root of your domain that gives large language models a curated, expert-level summary of your site, plus structured links to your most important pages. It is more like a curated sitemap.xml than a robots.txt.
Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety. Converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise. The /llms.txt file - llmstxt.org
The distinction from robots.txt matters because many articles conflate the two. Robots.txt is the bouncer at the door: it tells automated tools what access to your site is acceptable, such as for search indexing. llms.txt is the concierge handing AI visitors a personalized guide to the building. It does not block crawlers, dictate indexing behavior, or restrict access.
The expectation per the official specification is that llms.txt is mainly useful for inference, at the time a user is seeking assistance, rather than for training. When someone asks ChatGPT "what does Company X do?" or "which product should I choose?", the AI reaches for content it can parse quickly and trust. Your llms.txt tells it where to start.
One related file in the spec: llms-full.txt. Where llms.txt points to sources, llms-full.txt contains the entire content of a website's documentation in a single Markdown file. If llms.txt is the executive brief, llms-full.txt is the full-length book. For most business sites, start with just llms.txt and add llms-full.txt later only if you have extensive documentation or an API reference where exact wording matters.
When it landed and how strong the adoption signals are
Jeremy Howard of Answer.AI introduced llms.txt in September 2024. As of late 2025 it remains a proposed standard rather than an officially adopted one. None of the major LLM companies (OpenAI, Google, Anthropic) have publicly confirmed they read llms.txt during crawls.
- Proposed: September 2024 by Jeremy Howard (Answer.AI co-founder)
- Adoption signal 1: Anthropic specifically asked Mintlify to implement llms.txt and llms-full.txt for their documentation
- Adoption signal 2: Google included an llms.txt file in their new A2A (Agent-to-Agent) protocol
- Adoption signal 3: Vercel reports roughly 10% of their signups now come from ChatGPT, the same retrieval surface llms.txt targets
- Skeptic data point: Semrush's server log test showed zero visits from Google-Extended, GPTbot, PerplexityBot, or ClaudeBot from mid-August to late October 2025
- Platform momentum: Yoast SEO, WordPress.com, Webflow, Mintlify, and multiple WordPress plugins now ship native llms.txt generation
The implementation cost is near zero on every supported platform. The potential upside is substantial. That asymmetry is the reason most agencies are deploying llms.txt for clients now even before formal AI-vendor confirmation, similar to how robots.txt was once optional and is now essential for managing search crawlers.
Who benefits most by platform and content type
llms.txt value depends on what platform you run and what content you have. The implementation paths and limitations differ meaningfully across WordPress, Shopify, and Webflow.
| Platform / Site Type | Difficulty | Notes |
|---|---|---|
| WordPress (Yoast / dedicated plugins / manual) | Easy | Yoast SEO includes llms.txt generation free in Site Features (one-click toggle). For more control, the Website LLMs.txt plugin supports Yoast, Rank Math, SEOPress, and AIOSEO. AIOSEO has built-in support enabled by default. Manual upload via FTP works for total control over every line. |
| Webflow (CMS / Business plans only) | Easy | Native upload in SEO settings. Webflow handles root placement and prevents search engines from indexing the file. Starter and Basic plans do not expose the file even if uploaded. Re-publishing the site after upload is required, or the file is not served live. |
| Shopify (CDN + redirect or dedicated app) | Medium | No direct root-directory access. Workaround: upload file to Shopify CDN via Settings > Files, then create a URL redirect from /llms.txt to the CDN URL. Dedicated apps (LLMs.txt Generator, Arc, Revhope) automate this and keep files in sync with the product catalog. |
| Sites with extensive documentation or API references | High value | These are the cases the standard was originally designed for. Deploy both llms.txt (executive brief) and llms-full.txt (full content in one Markdown file). Anthropic and Mintlify deploy this pattern across their docs. |
| Sites with public pricing | High value | AI tools frequently get asked "how much does X cost?" If pricing is in your llms.txt, the AI will pull from it. If not, it will guess, and guesses are often |