What bot is this? List of AI crawlers, search bots and user-agents

Looking at your logs and seeing a name you don’t recognize? This page answers one question: what is this bot and what should you do about it. Check the operator, what the bot does, its robots.txt rules and the number of requests I recorded on my own site.

The “podrez.pl logs” column shows how many requests carrying a given bot name in the user-agent field I recorded on my site between 2 February and 5 August 2026. Read these numbers as a hint of what to expect, not as a market average.

Bot and user-agent list [searchable]

The table covers AI crawlers, classic search engines, SEO tools, link previews from messaging apps, monitoring services and plain HTTP libraries. Narrow the category or type a name – the table filters as you type, and when a single row matches you get a ready answer, plus a robots.txt entry wherever the bot has a usable token.

Why doesn't robots.txt block every AI bot?

An entry in robots.txt does not work the same way on every bot. Crawlers that move around the web on their own initiative usually follow it. With fetchers triggered by a user things get complicated, because here every operator has a policy of its own.

• Automated crawlers

  They move around the web on the operator's initiative: GPTBot, ClaudeBot, PerplexityBot, Googlebot. They honor robots.txt and this file was created with them in mind.

• User-triggered fetchers

  They fetch a page because a specific user action required it. Operators differ here. Anthropic and Mistral provide separate tokens (Claude-User, MistralAI-User) and declare that they honor them. Google, OpenAI, Perplexity and Amazon note that the rules may not apply, because the request is initiated by a user.

• Control tokens without a bot

  Google-Extended and Applebot-Extended send no requests at all. You put them in robots.txt, but you will never see them in your logs. Google-Extended covers training of Gemini models and grounding, meaning content passed to the model at the moment of answering. Applebot-Extended covers training of Apple's models.

The practical conclusion: before you block a fetcher triggered by a user, check the policy of that particular operator. A Disallow for Claude-User or MistralAI-User genuinely closes the door, because Anthropic and Mistral declare that they honor those tokens. With Google, OpenAI, Perplexity and Amazon the same entry is a request, so if you need certainty, add a rule at the server or WAF level. How to check this on your own site, I described in the article on server log analysis.

Suspicious user-agents: impossible names and names to verify

Five names turned up in my logs that cannot be reconciled with the documentation of their supposed operators. They are worth knowing, because they look credible and slip through a filter easily.

Two of these names cannot come from the operator, because its own documentation says those tokens do not crawl at all. Three more do not appear in the current documentation of their supposed operators – that does not prove impersonation, but it does mean you should check the IP address before treating them as genuine.

Addresses come from the podrez.pl logs, February–August 2026. Number of occurrences: Google-Extended 31, anthropic-ai 28, Claude-Web 20, BingIndexCrawler 10, Applebot-Extended 5.

The pattern is clear: suspicious requests reach for tokens that are not user-agents, and for names absent from operators' current documentation. It is worth distinguishing intent here. Requests signed as Google-Extended or anthropic-ai were looking for password and configuration files on my site, so that is a scanner. A name that does not match the documentation need not mean bad intent, though – which is why the IP address decides, not the name alone. The problem is not limited to my site either: Common Crawl explicitly warns in its documentation that it knows of crawlers falsely claiming to be CCBot, and recommends verifying the user-agent.

How to check whether a bot is real: verification by IP address

A user-agent can be faked in ten seconds, so the name alone proves nothing. Certainty comes only from matching the name with the IP address and the operator's official ranges, and where possible with a reverse and forward DNS check. Google is also testing cryptographic authentication of bots, that is Web Bot Auth.

How to block AI bots in robots.txt? Ready-made examples.

The first variant limits the main crawlers used for model training. The second blocks separately identified AI bots and fetchers, leaving access open to Googlebot, bingbot and Applebot. Both are a starting point, not a universal configuration for every site.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
User-agent: cohere-ai
User-agent: CohereBot
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: Amzn-SearchBot
User-agent: ShapBot
Allow: /

# search engines stay
User-agent: Googlebot
User-agent: bingbot
User-agent: Applebot
Allow: /

# model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
User-agent: cohere-ai
User-agent: CohereBot
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# indexes behind generative answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: MistralAI-Index
User-agent: Meta-WebIndexer
User-agent: DuckAssistBot
User-agent: Amzn-SearchBot
User-agent: LinkupBot
User-agent: ShapBot
User-agent: YouBot
User-agent: Diffbot
Disallow: /

# fetches on user request
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
Disallow: /

The second variant has a hole and it is better to know about it right away. The last group are fetchers triggered by a user, and some of them may ignore robots.txt. Gemini Notebook and Google-Agent do not even have a token of their own. If the block is to be airtight, it has to be added at the server level. An example for Apache and the .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Google-GeminiNotebook|Google-NotebookLM|Google-Agent|GoogleProducer|ChatGPT-User|Perplexity-User|Amzn-User|Meta-ExternalFetcher|Shap-User) [NC]
RewriteRule ^ - [F,L]

One caveat that is easy to forget: robots.txt is not a tool for protecting data. The file is public, the directives are not access control, and a blocked URL can still end up in the index on the strength of external links. Genuinely private sections are secured with server-side authentication.

How to limit AI Overviews? On nosnippet, data-nosnippet and max-snippet

robots.txt tells compliant bots whether they should fetch a page. Meta directives, by contrast, control indexing and which fragments can reach the results and the AI features. These are two different layers, and with AI Overviews the second one is often the only one you have left.

Apple honors nosnippet as an opt-out from the use of content in generative answers based on current information. Amazon declares that it honors noarchive, noindex and none, but does not support crawl-delay.

What AI bots take from your site: content or files?

The number of visits alone says little. What is more interesting is how many of them land on a page with content, and how many on CSS files, images and API endpoints.

podrez.pl logs, 2 Feb – 5 Aug 2026. Perplexity-User, ChatGPT-User and Claude-User land mostly straight on content, but the class of bot alone does not determine its behavior: CCBot has a high share of hits on content, and Gemini Notebook a low one.

Frequently asked questions

The name in the log is only the beginning. Far more interesting is what that bot got in response – and whether it got anything at all. How to check that and what I found on my own site, I described in the article on server log analysis.