--- title: "llms.txt: What It Is, Who Actually Fetches It, and Is It Worth It?" description: "What llms.txt is, how to generate it in Rank Math, and why Google Search calls it unnecessary while Lighthouse checks it. Plus data from my server logs on who actually fetches the file." date: 2026-11-01 author: "Ewelina Podrez-Siama" url: "https://podrez.pl/en/llms-txt/" categories: - "SEO Blog" --- Originally published [in Polish](https://podrez.pl/plik-llms-txt/): 15 October 2025 · Updated: 1 November 2026 · English translation: 1 November 2026 Does AI need a new robots.txt? People talk more and more about **llms.txt**, a new, experimental format meant to help large language models (LLMs) such as ChatGPT, Gemini or Claude understand your website. But does it actually work? And is it worth implementing today? In short **Google Search:** ignores llms.txt, including in AI Overviews and AI Mode. Google’s guide says outright that the file neither helps nor hurts visibility. **Lighthouse:** since version 13.3 it checks llms.txt in the experimental Agentic Browsing category, with agents working in the browser in mind, not rankings. **My logs:** llms.txt is fetched mostly by SEO tools and Lighthouse. AI bots with verified addresses account for 14 of 420 fetches. Let me say it upfront: in my view llms.txt changes little when it comes to [brand visibility in AI](https://podrez.pl/en/ai-search-visibility/). Agents that complete tasks in the browser are a different story: that’s where Lighthouse has started checking the file. It’s good to know what the file does and what it doesn’t before you add it to your SEO strategy. Table of contents [Toggle](#) - [What is an llms.txt file?](#what-is-an-llmstxt-file) - [What does an llms.txt file look like?](#what-does-an-llmstxt-file-look-like) - [Why create an llms.txt file?](#why-create-an-llmstxt-file) - [Google Search vs Lighthouse: two different messages](#google-search-vs-lighthouse-two-different-messages) - [Who actually fetches llms.txt? Data from podrez.pl logs](#who-actually-fetches-llmstxt-data-from-podrezpl-logs) - [Limitations and the actual status of llms.txt](#limitations-and-the-actual-status-of-llmstxt) - [How to generate llms.txt in Rank Math (WordPress)](#how-to-generate-llmstxt-in-rank-math-wordpress) - [llms.txt best practices](#llmstxt-best-practices) - [Is llms.txt worth implementing?](#is-llmstxt-worth-implementing) - [Summary](#summary) - [Questions about llms.txt](#questions-about-llmstxt) What is an llms.txt file? ------------------------- llms.txt is an experimental standard designed with large language models in mind, such as ChatGPT, Claude, Gemini or Perplexity. It is meant to help models understand the structure of your site by pointing to the most important content and the context worth reading first. Just as robots.txt guides search engine crawlers, llms.txt is meant to act as a context map for AI. For comparison: - ### robots.txt Tells crawlers what not to fetch. - ### sitemap.xml Lists the URLs worth fetching and indexing. - ### llms.txt Suggests to language models and agents what matters most on the site. What does an llms.txt file look like? ------------------------------------- The file is plain text (Markdown), readable by humans and machines. A typical layout: ``` # Your Brand > SEO blog and agency – we build online visibility strategies and teach how algorithms work. ## Key resources - [SEO blog](https://yoursite.com/blog): Articles on SEO, search intent and AI in search. - [Services](https://yoursite.com/services): SEO services and online training. - [Contact](https://yoursite.com/contact): Contact details and form. ## Optional - [About us](https://yoursite.com/about): The team and the company’s mission. ``` Elements of an llms.txt file and what they doElementWhat it’s for`#`main title (brand or project name)`>`a short description of the mission or context`##`topical sections, e.g. Blog, Products, Contact`- [Title](URL): description`lists of links with short explanations`## Optional`secondary content a model may skip You’ll find my file at [podrez.pl/llms.txt](https://podrez.pl/llms.txt). Why create an llms.txt file? ---------------------------- Why consider llms.txt at all? Because it may: - **1**### Help models understand the site faster Models work with a limited context and can’t read a whole website. llms.txt hands them a concise list of key content. - **2**### Make correct citation easier A well-prepared file may increase the chance that chatbots and AI assistants refer to your site as a source. - **3**### Give you control over what AI sees You choose which pages are priorities and which can be skipped. That matters especially for premium content or content that’s easy to take out of context. - **4**### Make life easier for browser agents The Lighthouse documentation assumes that without the file an agent may spend more time crawling the site before it understands its structure. Google Search vs Lighthouse: two different messages --------------------------------------------------- In May 2026 Google sent two signals about llms.txt that seem to contradict each other at first glance. The search team, in its guide to generative AI features, put llms.txt in the bag of things you don’t need. In the same month, Lighthouse 13.3 got a new audit category in which llms.txt is one of the tests. What Google Search and Lighthouse say about llms.txtGoogle SearchLighthouse (Chrome)**Document**[guide to optimizing for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) (May 2026, updated 10 July 2026)[llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) in the Agentic Browsing category (documentation from 5 May 2026)**What it says about the file**Google Search ignores it: the file neither helps nor hurts, but you can keep it for other services“emerging convention”, a concise Markdown summary of the site for agents**Who it’s for**the search engine and its AI featuresagents working in the browser**No file**no effect on visibilityaudit “not applicable” (404); a server error while fetching is flagged**Status**official search documentationexperimental category, testing requires Chrome 150 or newer **It’s not a contradiction, just two different uses.** Search decides what to show in results. Lighthouse checks whether an agent will understand the site easily. The Agentic Browsing category is about more than llms.txt, too. It also checks WebMCP integration, how accessible the page is to agents (the accessibility tree) and layout stability. It doesn’t give a 0–100 score, it shows how many tests the page passed. My take If your site is meant to be convenient for agents, e.g. a shop where an agent should find a product and place an order, llms.txt is fifteen minutes of work and does no harm. A result in this category says nothing about your position in Google, though. Who actually fetches llms.txt? Data from podrez.pl logs ------------------------------------------------------- My llms.txt file is generated by Rank Math. I checked my server logs to see who fetched it from 2 February to 5 August 2026 and from 17 to 25 October 2026, the periods for which I have complete archives. That came to 420 fetches. I checked bot IP addresses against lists published by the operators and, where there are no lists, with reverse DNS. SEO tools210browsers, scripts, scanners80Lighthouse (PageSpeed Insights)72other AI indexes and assistants29AI and search bots14my own tests12posing as known bots3SEO tools210browsers, scripts, scanners80Lighthouse (PageSpeed Insights)72other AI indexes and assistants29AI and search bots14my own tests12posing as known bots3 Who fetched podrez.pl/llms.txt: 2 Feb – 5 Aug and 17–25 Oct 2026, 420 fetches. Darker color: AI and search bots with verified IP addresses. Source: server logs, my own analysis. - ### SEO tools: 210 More than half of all fetches. Mostly AhrefsSiteAudit, the site audit crawler in Ahrefs, followed by Barkrowler, Screaming Frog and the Semrush audit, among others. Tools check whether the file exists. - ### Lighthouse: 72 Requests from Google addresses tagged Chrome-Lighthouse, i.e. PageSpeed Insights tests. The first ones in my logs appeared on 28 July. That’s the new llms.txt audit in practice. - ### AI and search bots: 14 GPTBot 5, Amazonbot 4, bingbot 3, Applebot 1 and CCBot (Common Crawl) 1, all with verified addresses. GPTBot looked at the file 5 times in more than half a year. - ### Other AI indexes and assistants: 29 Smaller search engines and AI services, including LinkupBot, ibou and ElevenLabs, plus a scanner that collects llms.txt files from across the web. My take Today llms.txt is read mostly by tools checking whether you have one, not by models that would use it. A dozen or so AI bot visits in more than half a year is too few to talk about real impact. But it’s enough reason to keep the file, given it takes fifteen minutes. A side note: Markdown versions of pages Not the same thing as llms.txt, but from the same family. Since 16 October 2026 I’ve been testing the Markdown for Agents plugin on podrez.pl, which gives every page a Markdown version (a URL ending in `index.md` or `?format=md`). In the first two weeks, bots fetched these versions about 1,770 times, led by ibou (703), Exa (518) and the ElevenLabs voice assistant (201). Some AI systems clearly prefer to take the content of pages in a machine format rather than their index from llms.txt. How to check who really visits your site, and whether it’s a bot in disguise, I show in my free tool [AI bots in your logs](https://podrez.pl/en/ai-bots-in-logs/). Limitations and the actual status of llms.txt --------------------------------------------- Here’s the key point: as of today (November 2026), llms.txt is only a **proposal**, not a standard supported by the largest models. - ### Google Gary Illyes compared llms.txt to the 1990s keywords meta tag, and the guide to generative AI features says outright that Google Search ignores such files. Lighthouse in Chrome checks it, though, in a category for agents. - ### OpenAI Hasn’t officially confirmed support. In my logs GPTBot fetched llms.txt 5 times in more than half a year, but there’s no evidence it interprets it. - ### Anthropic and others No confirmed implementation on the model side. In the logs I analyzed, there was no llms.txt fetch by a verified Anthropic crawler. - ### SEO tools Rank Math, Yoast, WordLift and other SEO tools have started adding support, preparing for the future. For now, llms.txt is a **leading signal**. Adding it won’t hurt, but there’s no guarantee any model actually “reads” it today. One more thing. The authors of the specification assumed the file would be most useful at answer time, when a system fetches a page for someone’s question. A model without web access won’t fetch it mid-conversation. It only knows its training data from before its [knowledge cutoff](https://podrez.pl/en/ai-knowledge-cutoff-dates/) and whatever you paste in. How to generate llms.txt in Rank Math (WordPress) ------------------------------------------------- Since August 2025, Rank Math can generate an **llms.txt** file automatically. Here’s how: 1. 1**Turn on the module**In the WordPress dashboard go to **Rank Math → Dashboard → Modules** and turn on the **LLMS Txt** module. 2. 2**Choose content**In **Rank Math → Titles & Meta → LLMS Txt**, select content types (posts, pages, categories), the maximum number of posts (100 by default) and whether to add an “Optional” section. 3. 3**Add links**You can add important URLs by hand, e.g. a landing page or documentation. 4. 4**Check the file**Save your changes and open `yoursite.com/llms.txt`. Rank Math adds headings, links and descriptions in Markdown, following the llmstxt.org specification. llms.txt in WordPress If you want an **SEO command center in WordPress** without buying expensive add-ons, Rank Math SEO is the best choice. That’s also where you’ll find `llms.txt`. The free version offers more than paid Yoast, and a PRO license, once bought, works on multiple sites. Ideal if you run a blog, a shop and a few projects at once. [Go to Rank Math](https://rankmath.com/?ref=epodrez) llms.txt best practices ----------------------- - **1**### Write concisely and like a human Models understand plain language better. Avoid marketing fluff and jargon. - **2**### Add a short description in the blockquote That paragraph will often be the context for how your brand is interpreted. - **3**### Don’t overdo the length Keep the file light, around 20–50 links at most. - **4**### Update it every few months Especially after major changes to your site structure. - **5**### Make sure the server serves it Lighthouse doesn’t flag a missing file, but it does flag a server error when fetching it. - **6**### Combine it with classic SEO llms.txt won’t replace a sitemap, internal linking or content optimization. It’s an add-on that may help. Is llms.txt worth implementing? ------------------------------- - ### Yes, if… you want to be a step ahead, or you care about agents in the browser. It’s a simple way to prepare your site for a future in which chatbots and agents get things done instead of search engines. It certainly won’t hurt. - ### No, if… you expect an immediate effect in Google. Neither Google, OpenAI nor Anthropic treats llms.txt as a ranking signal. Just don’t treat it as a ticket to a world where GEO (*Generative Engine Optimization*) replaces SEO. AI features in Google, AI Overviews and AI Mode, use the same systems and the same index as regular results, so their foundation is still SEO. llms.txt is a cheap experiment for the future, not the foundation of AI visibility. More on that in [GEO vs SEO](https://podrez.pl/en/geo-vs-seo/). Summary ------- llms.txt is a new way for a site to talk to AI models. It doesn’t guarantee visibility, but it organizes your content and gives you control over what AI “reads”. It also makes life easier for agents, which Lighthouse has already started to take into account. With Rank Math you can generate it in a few clicks. A small step that may turn into an advantage once models and agents start using it every day. Read next --------- - ![llms.txt: What It Is, Who Actually Fetches It, and Is It Worth It?](https://podrez.pl/wp-content/uploads/2026/11/geo-vs-seo-768x432.webp "llms.txt: What It Is, Who Actually Fetches It, and Is It Worth It?")GEO ### [GEO vs SEO](https://podrez.pl/en/geo-vs-seo/) Where GEO is just SEO, and where separate work begins. - ![llms.txt: What It Is, Who Actually Fetches It, and Is It Worth It?](https://podrez.pl/wp-content/uploads/2026/08/co-to-za-bot-768x403.jpg "llms.txt: What It Is, Who Actually Fetches It, and Is It Worth It?")List with search ### [What bot is this?](https://podrez.pl/en/bots/) Check who is behind a bot from your logs and how to block it. - ![llms.txt: What It Is, Who Actually Fetches It, and Is It Worth It?](https://podrez.pl/wp-content/uploads/2026/08/analiza-logow-768x403.jpg "llms.txt: What It Is, Who Actually Fetches It, and Is It Worth It?")Server logs ### [Do AI bots actually reach your content?](https://podrez.pl/en/server-log-analysis-ai-visibility/) What my server logs showed when the host started blocking AI bots. Questions about llms.txt ------------------------ ### Does llms.txt help in Google? Not in search. Google’s guide to generative AI features says plainly that Google Search ignores such files, so they neither help nor hurt visibility, including in AI Overviews and AI Mode. Lighthouse checks llms.txt only in an experimental category for agents working in the browser. ### What happens if I don’t have an llms.txt file? In Google Search, nothing. In Lighthouse, the audit is marked “not applicable”, because the file is optional for now. You only get a flag when the server returns an error while fetching it. ### Do ChatGPT or Claude read llms.txt? Neither OpenAI nor Anthropic has confirmed using the file in answers. In my logs GPTBot fetched it 5 times in more than half a year, and there was no fetch at all by a verified Anthropic crawler. The file is fetched mostly by SEO tools and Lighthouse. ### How do I generate llms.txt in WordPress? The simplest way is Rank Math: turn on the LLMS Txt module, choose content types and the post limit, save your changes and check the file at yoursite.com/llms.txt. ### Sources Google Search Central, [guide to optimizing for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) (May 2026, updated 10 July 2026). Chrome for Developers, Lighthouse: [llms.txt](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) and [Agentic browsing scoring](https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring) (5 May 2026). Search Engine Journal, Matt G. Southern, [Google’s llms.txt Guidance Depends On Which Product You Ask](https://www.searchenginejournal.com/googles-llms-txt-guidance-depends-on-which-product-you-ask/575431/) (20 May 2026). Search Engine Roundtable, [Gary Illyes on llms.txt and the keywords meta tag](https://www.seroundtable.com/google-parallel-meta-keywords-llmstxt-39862.html) (1 August 2025). Jeremy Howard, [llmstxt.org](https://llmstxt.org/) (specification). Rank Math, [llms.txt Documentation](https://rankmath.com/kb/llms-txt/). Yoast, [LLM SEO optimization techniques (including llms.txt)](https://yoast.com/llm-seo-optimization-techniques-including-llms-txt/). My own data: podrez.pl server logs (2 Feb – 5 Aug 2026 and 17–25 Oct 2026, bot IP addresses checked against operator lists and reverse DNS) and the Markdown for Agents plugin log (16–30 Oct 2026), excluding requests from my own tests. An LLM may have supported me in preparing this text – most often at the translation, research, proofreading or code-styling stage. The responsibility for the decisions, the claims made and the arguments cited is fully mine. More on [how I work with AI](https://podrez.pl/en/authorship-and-ai/).