--- title: "Can AI bots read your site?" description: "Tool · robots.txt and AI bots Can AI bots read your site? robots.txt tells bots what they may do. Check whether it says what you think it says. Enter a domain or paste your robots.txt. You will see what your file lets 57 bots and tokens from my list do: from GPTBot and ClaudeBot to" date: 2026-11-12 author: "Ewelina Podrez-Siama" url: "https://podrez.pl/en/robots-txt-checker/" --- Tool · robots.txt and AI bots Can AI bots read your site? =========================== robots.txt tells bots what they may do. Check whether it says what you think it says. Enter a domain or paste your robots.txt. You will see what your file lets 57 bots and tokens from my list do: from GPTBot and ClaudeBot to the new \_\_usage\_ai-training token, with Googlebot and Bingbot for comparison. Plus meta robots, X-Robots-Tag, TDMRep, Content-Signal and llms.txt, and for every bot the line that decided it. This tool needs JavaScript. [Something not working? Let me know →](#feedback) [How it works](#how-it-works)[Statuses](#statuses)[AI signals](#signals)[A request, not a lock](#caveats)[Privacy](#privacy)[What to do next](#what-to-do)[Tools and lists](#tools)[Further reading](#further-reading)[FAQ](#faq) The tool at a glance -------------------- **57 bots and tokens**from my bot list: training, AI search, visits on a user’s request and tokens without a bot of their own. Googlebot and Bingbot for comparison. **A parser like Google’s**logic and tests from Google’s open-source robotstxt library: the longest rule wins, and Allow wins a tie. **5 signals and llms.txt**robots.txt, the \_\_usage\_ai-training token, meta robots and X-Robots-Tag, TDMRep and Content-Signal. Plus llms.txt. **0 stored domains**the server fetches the files, your browser computes the result. I store no domains or files. How it works What the tool checks and how it decides --------------------------------------- When you enter a domain, the podrez.pl server fetches four public files: `/robots.txt`, the home page (headers and `` only), `/llms.txt` and `/.well-known/tdmrep.json`. Your browser does the rest. You can also paste your robots.txt, and then the file never leaves your computer. I read the file the way Google does: following RFC 9309 and the open-source robotstxt library, whose tests I ported as well. A bot looks for the group with its name. If there is none, it takes the `*` group. When several rules match an address, the longest one wins, and Allow wins a tie. Groups with the same name are merged, and Google only reads the first 500 KiB of the file. Some bots have a fallback rule. Without a group of their own, Google-CloudVertexBot and Googlebot-News follow the rules for Googlebot. The shared `__usage_ai-training` token does not decide access. It only signals an opt-out of training. Anthropic has taken it into account since October 30, 2026 (rolling out gradually) when the file has no ClaudeBot group, so for ClaudeBot I show it separately, under the result. `__usage_ai-training` itself is not covered by the `*` group: without its own group, the site expresses no preference. For every bot you see the line that decided, and in the tester you can check any address and any user-agent. That is the feature Google removed from Search Console in December 2023. Only the robots.txt report and URL Inspection remain, and URL Inspection works for Googlebot only. Result What the statuses mean ---------------------- - ✓### Can read Your robots.txt lets the bot fetch the checked address. Exceptions shared by all bots, e.g. /wp-admin/, are shown as a note under the bot. For tokens: “No opt-out”. - ◐### Partly Rules just for this bot close some addresses to it or open only part of the site. The list is under the bot. For tokens: “Partial opt-out”. - ✕### Blocked Your robots.txt asks the bot not to fetch this address and no rule reopens it. For tokens: “Opted out”. When robots.txt returns a server error (5xx or 429), you will see “Blocked (server error)”. - –### No preference Only \_\_usage\_ai-training: there is no group for it and the \* group does not cover it, so the site says neither yes nor no. - ·### Not applicable The operator gives no robots.txt name for this bot (e.g. ChatGPT-User). The file will not stop it; only your server can. The statuses show what your robots.txt asks for, not what a bot will actually do. A server firewall or Cloudflare settings can turn away bots that the file lets in. You will see “Can read” here while the bot gets a 403 or 429. Your logs will show it. Enter a domain and I check the home page. Enter the full address of a page and I check that page. You can also enter another address in the tester below and recalculate the whole table for it. AI signals Allow and Disallow are not everything ------------------------------------- - **`__usage_ai-training`.** A shared token against AI training, announced on October 30, 2026 by Google, Microsoft, OpenAI and Anthropic. I explain who applies it and since when in my [post on \_\_usage\_ai-training](https://podrez.pl/en/usage-ai-training/). - **Google-Extended.** Consent to Gemini training and grounding, i.e. Gemini answers based on your pages, in Gemini apps and Vertex AI. Blocking it does not remove the site from AI Overviews or AI Mode. The Search Console setting does that (Settings → [Search generative AI](https://support.google.com/webmasters/answer/16908024?hl=en), since August 31, 2026). At page level, you can limit it with `nosnippet`, `max-snippet` or `noindex`. - **Bing.** From February 1, 2027, Microsoft reads training (`ai-training`) and grounding (`ai-grounding`) separately in a meta tag or the X-Robots-Tag header. Until March 1, 2027 it also reads `noarchive` and `nocache` as an opt-out of both. The tool shows both settings. - **TDMRep.** A text and data mining reservation under Article 4 of EU Directive 2019/790: a `tdmrep.json` file, a header or a meta tag. It is a W3C Community Group report, not a W3C standard. - **Content-Signal and Content-Usage.** Declarations in robots.txt: the first from Cloudflare, the second from an IETF draft. I show which group they belong to, because a blank line does not close a group and a line can end up applying to a different bot than the author meant. - **llms.txt.** I check whether it exists, its status, title and links, similar to the experimental Lighthouse “Agentic Browsing” audit. Google Search does not use it. More in my [post on llms.txt](https://podrez.pl/en/llms-txt/). Before you draw conclusions robots.txt is a request, not a lock ----------------------------------- Honest bots follow it. Spoofed and malicious ones do not. According to their operators, some bots acting on a user’s request (e.g. ChatGPT-User or Perplexity-User) may skip robots.txt, and Google’s feed fetchers ignore it entirely. The tool shows what each operator states. Your server logs show what bots actually do on your site. I have a separate tool for that: [AI bots in logs](https://podrez.pl/en/ai-bots-in-logs/). Privacy What happens to the domain you enter ------------------------------------ The files are fetched by the podrez.pl server, so the checked site does not see your IP address, only my server and the tool’s name in the user-agent. The server connects only to public addresses and follows at most five redirects. It stores no domains and no file content. For rate limits it keeps hashes (HMAC) of your IP address, network and the checked domain for at most a day, never the values themselves. A pasted robots.txt stays in your browser. Its content never leaves it. **What I measure.** Google Analytics only learns that someone used the tool: in which mode (domain, pasted file, example), the type of robots.txt response (e.g. “2xx” or “5xx”), the error code if something failed, and whether the tester checked a listed bot or a typed one and came out “allowed” or “disallowed”. No domain, address or user-agent, and only in line with your cookie settings. Before you change anything What to do with the result -------------------------- - **No training, but you want to stay in AI search.** Block the bots in the “Model training” group (GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider and others), the Google-Extended and Applebot-Extended tokens, and add a `__usage_ai-training` group. Leave AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) open. What models already know is marked by their knowledge cutoff (see my [AI knowledge cutoff dates](https://podrez.pl/en/ai-knowledge-cutoff-dates/) table). - **You want out of AI Overviews.** Google-Extended will not do it. The simplest way is the [Search generative AI](https://support.google.com/webmasters/answer/16908024?hl=en) setting in Search Console (Settings). It excludes the site from AI Overviews, AI Mode and AI features in Discover, with no effect on regular results. For single pages you can use `nosnippet` or `max-snippet`, but that also shortens their description in regular results. - **Everything is blocked when it shouldn’t be.** Look for `Disallow: /` in the `*` group. It is a common leftover from a staging site. - **robots.txt returns a 5xx error.** For the first 12 hours Google stops crawling the site, then for up to 30 days it uses the last good copy of the file. Fix the server before you fine-tune the rules. - **“Can read”, but the bot is missing from your logs or gets a 403 or 429.** That is your server, not robots.txt. Ask your host about the firewall (WAF) or check the bot settings in Cloudflare. My own host once quietly switched on a firewall that turned AI bots away with a 429. robots.txt let them all in, and I only saw it in my logs. - **Notes on the file.** Google forgives typos in directive names and a missing colon; other bots may not. Rules before the first User-agent and patterns without a slash are skipped by every bot. Better to fix them. - On my own site I let all AI bots in and do not add `__usage_ai-training`. I care about visibility in AI answers, so this is a deliberate choice, not an oversight. Tools and lists More tools and lists -------------------- Free, no sign-up. Each one answers one specific question. - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/11/obserwatorium-encji-hero-kolaz-768x432.webp "Can AI bots read your site?")Tool ### [Entity Observatory](https://podrez.pl/en/entity-observatory/) See a page’s structured data as a relation map and add the Wikidata record next to it. - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/05/knowledge-graph-768x512.jpg "Can AI bots read your site?")Tool ### [Are you in Google’s Knowledge Graph?](https://podrez.pl/en/knowledge-graph-checker/) Type a name and see whether Google knows you as an entity and whether that matches Wikidata. - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/05/common-crawl-hero-768x512.jpg "Can AI bots read your site?")Tool ### [Is your site in Common Crawl?](https://podrez.pl/en/common-crawl-checker/) See how many of your pages made it into the Common Crawl archive and whether anything blocks CCBot. - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/08/analiza-logow-ai-768x403.jpg "Can AI bots read your site?")Tool ### [AI bots in logs](https://podrez.pl/en/ai-bots-in-logs/) Drop in a server log and see which AI bots actually visit and whether your server turns any of them away. - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/10/spadek-wyswietlen-search-console-boty-768x432.webp "Can AI bots read your site?")Tool ### [Who typed these Search Console queries?](https://podrez.pl/en/search-console-prompts/) Drop your Search Console export and see which queries look like prompts, operators and tool footprints. - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/08/co-to-za-bot-768x403.jpg "Can AI bots read your site?")Searchable list ### [List of bots and user agents](https://podrez.pl/en/bots/) A searchable list of bots: who runs them, what they do and how to handle them in robots.txt. The robots.txt file is a declaration. To see how your server responds to requests posing as GPTBot or ClaudeBot, try [High Five, AI!](https://high5ai.app/) by Maciej Chmurkowski. Further reading On AI bots, robots.txt and llms.txt ----------------------------------- - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/11/usage-ai-training-robots-txt-768x432.webp "Can AI bots read your site?")Post ### [One rule against AI training](https://podrez.pl/en/usage-ai-training/) What \_\_usage\_ai-training changes and who already applies it. - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/11/plik-llms-txt-768x404.webp "Can AI bots read your site?")Post ### [llms.txt](https://podrez.pl/en/llms-txt/) What it is, who actually fetches it and whether it is worth it. - ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/08/analiza-logow-ai-768x403.jpg "Can AI bots read your site?")Post ### [Server log analysis and AI visibility](https://podrez.pl/en/server-log-analysis-ai-visibility/) Do AI bots actually reach your content? Frequently asked questions -------------------------- ### How do I check whether GPTBot can read my site? Enter your domain in the field at the top of the page. GPTBot is in the “Model training” group, and OAI-SearchBot (ChatGPT search) in “AI search engines”. For each you see the status and the robots.txt line that decided. Enter a page address straight away or check it in the tester below. ### Is this a robots.txt validator? Yes, and more. It reads your file the way Google’s parser does and flags typos, rules before the first User-agent, patterns without a slash, a file over 500 KiB and an HTML page served as robots.txt. On top of that, it shows what the file means for 57 bots and tokens from my list. ### Is there an alternative to Google’s robots.txt tester? Google removed the robots.txt tester from Search Console in December 2023. This tool covers what it did: enter any address and any user-agent and see whether robots.txt allows it and which line decided. The parser and its tests come from Google’s open-source robotstxt library. ### Can it check my llms.txt? Yes. When you enter a domain, the tool fetches `/llms.txt` and shows its status, size, title and summary, plus the number of sections and links. It flags a missing “# title” line, an HTML page instead of the file and a server error, which the experimental Lighthouse “Agentic Browsing” audit counts as a problem. Google Search does not use llms.txt. ### How do I block AI training but stay in AI search? Block training bots by name (e.g. `User-agent: GPTBot` with `Disallow: /`), add the Google-Extended and Applebot-Extended tokens and a `__usage_ai-training` group with `Disallow: /`. Leave AI search bots such as OAI-SearchBot, Claude-SearchBot or PerplexityBot open. After the change, check the file with this tool. ### Does blocking Google-Extended remove my site from AI Overviews? No. Google-Extended covers Gemini training and grounding in Gemini apps and Vertex AI. AI Overviews and AI Mode use the search index. You can exclude the whole site from them in Search Console (Settings → [Search generative AI](https://support.google.com/webmasters/answer/16908024?hl=en)), with no effect on regular results. For single pages, use `nosnippet`, `max-snippet` or `noindex`. ### What is \_\_usage\_ai-training? It is a shared robots.txt token that opts out of AI model training at several companies at once. The `*` group does not cover it: unless you add a separate group for it, the site expresses no preference. It is a training opt-out, not a block: whether a bot comes in at all is still decided by the rules for its name or the `*` group. Details and rollout dates: [post on \_\_usage\_ai-training](https://podrez.pl/en/usage-ai-training/). ### Does robots.txt really stop bots? Only the ones that choose to follow it. It is a request, not a lock. Spoofed bots and scrapers ignore it, and some fetches on a user’s request may skip it by design. Real traffic shows up in your server logs. ### Why is ChatGPT-User “Not applicable”? Because OpenAI does not document ChatGPT-User as a robots.txt control token and says the rules may not apply to it. If your file has a group for it, the tool shows it, with a caveat. The same goes for other bots acting on a user’s request. ### What does “Partly” mean? That your robots.txt has rules just for this bot, and they close some addresses to it or open only part of the site. The paths are listed under the bot. Exceptions shared by all bots, e.g. `/wp-admin/` or search results in the `*` group, do not change the status. I show them as a note under the bot. ### Do I need llms.txt? It is not mandatory, and Google Search does not use it. The experimental Lighthouse “Agentic Browsing” audit treats a missing file (404) as “not applicable”, and only a server error counts as a problem. I discuss whether it is worth adding in my [post on llms.txt](https://podrez.pl/en/llms-txt/). ### How is this different from the robots.txt tester in Search Console? Search Console now only has the robots.txt report, which shows the state of the file, and URL Inspection, which shows a block but only for Googlebot. Here you can check any address with any user-agent, plus the whole list of AI bots at once. ### How often can I check? A domain 4 times a minute, 20 times an hour and 60 times a day from one IP address. The tool checks the same domain at most 12 times an hour and 40 times a day, so as not to flood other people’s servers. A pasted robots.txt has no limit, because it never leaves your browser. ### Do you store the domains I check? No. The server stores no domains and no fetched files. For rate limits it keeps only hashes of the IP address and the checked domain, which disappear within a day at most, and its statistics count checks only, without domains. Google Analytics gets only the mode, the type of robots.txt response, the error code and the tester result, without the domain. ![Can AI bots read your site?](https://podrez.pl/wp-content/uploads/2026/11/ewelina-podrez-siama-awatar-bez-tla.webp "Can AI bots read your site?")Built by **Ewelina Podrez-Siama**, an independent SEO consultant, AI search visibility strategist and trainer, in SEO since 2009. [About me](https://podrez.pl/en/about-me/) Version 1.0.4, November 13, 2026 Data: my bot list (as of November 8, 2026). Parser: ported from Google’s open-source robotstxt library (Apache License 2.0) together with its tests. The tool is not affiliated with Google, Microsoft, OpenAI, Anthropic or Cloudflare. What next Not sure which bots to let in? ------------------------------ Training, AI search, user-triggered visits and regular search engines are four separate decisions. I help you make them without losing visibility where you need it. Want me to take a look? Send a short brief: your domain and what the tool showed. I’ll tell you what to set up first. [Send a short brief](https://podrez.pl/en/contact/) [AI search visibility](https://podrez.pl/en/ai-search-visibility/) Feedback Something not working or got an idea? ------------------------------------- I build these tools myself and keep improving them, so every note helps. Tell me what happened or what you’re missing. If something broke, add your browser and device (phone or computer) and I’ll find it faster. What’s not working, or what should I add? \* E-mail (if you’d like a reply) Only your message and the address of this page reach me by e-mail. The data controller is Ms. Fox Ewelina Podrez-Siama (Wrocław, Poland). Your note (message, e-mail if given, page address, IP address, browser, date and reCAPTCHA result) is also stored in the site’s admin panel; I’ll use your e-mail only to reply ([privacy policy, in Polish](/polityka-prywatnosci/)). This site is protected by reCAPTCHA and the Google [Privacy Policy](https://policies.google.com/privacy) and [Terms of Service](https://policies.google.com/terms) apply.