Ewelina Podrez-Siama · tools and lists

What bot is this? List of AI crawlers, search bots and user-agents

Q: Why don't I see Google-Extended in my logs?

Because Google-Extended is not a bot. It is a control token in robots.txt with which you decide about the use of your content for training Gemini models and for grounding. The fetching itself is done by Google's ordinary robots. If you do see this name in your log, you are looking at an impersonator.

Q: How do I opt out of AI Overviews?

Through robots.txt you can't. AI Overviews and AI Mode use Google's ordinary index, built by Googlebot, and the Google-Extended token covers only the training of Gemini models and grounding. The only available tool is the preview directives: nosnippet, data-nosnippet, max-snippet and noindex. Each of them also limits your ordinary search result, so leaving AI Overviews means partly leaving Google.

This page answers one question: what is this bot and what should you do about it.

Looking at your logs and seeing a name you don’t recognize? Check the operator, what the bot does, its robots.txt rules and the number of requests I recorded on my own site.

Go to the search Check bots in your logs

120

bots and user-agents
in the table

operator documents
behind the table

2.6M

requests from
podrez.pl logs

Search · 120 entries

Bot and user-agent list [searchable]

The table covers AI crawlers, classic search engines, SEO tools, link previews from messaging apps, monitoring services and plain HTTP libraries. Narrow the category or type a name – the table filters as you type. Click a bot’s name to see a card with its description and notes, plus a ready-made robots.txt entry wherever the bot has a usable token. A bot with an “i” icon has additional notes.

The “Requests” column shows how many requests carrying a given bot name in the user-agent field I recorded on my site between 2 February and 6 October 2026. Read these numbers as a hint of what to expect, not as a market average – and remember that they count names, not bots: scanners impersonate some of the names.

1. Narrow the category

What do the bot categories mean?Eleven classes of bots, one sentence each.

AI: training: collects content so that models can learn from it
AI: indexes and answers: builds an index that a generative system quotes from
User-triggered: fetches a page because someone has just asked about something
Tokens without a bot: send no requests, they exist only as robots.txt entries
Search engines: classic search engines and their helper bots
Link previews: generate the link thumbnail in a messenger or on social media
SEO and marketing: SEO tools, audits and checks on ad landing pages
Monitoring and security: site availability, brand mentions, abuse scanning
Research and archives: academic corpora, datasets and page archiving
Tools and libraries: traffic triggered by a browser, plugin, app or someone’s own script
Unknown or suspicious: operator, purpose or user-agent authenticity has not been confirmed

The table scrolls sideways →

Bot and operator	Purpose	robots.txt token	Honors it	Requests podrez.pl, 2 Feb – 6 Oct 2026
OpenAI	Model training	GPTBot	Yes	10,218 of which 9,925 verified
Amazon	Improving Amazon products; may also be used for model training	Amazonbot	Yes	8,111 of which 7,819 verified
Anthropic	Model training	ClaudeBot	Yes	6,789 of which 6,518 verified
Meta	Content collection, including for training	Meta-ExternalAgent	Stated by operator	4,212
ByteDance	Collects content for ByteDance models	Bytespider	No clear documentation	968
Common Crawl	Open corpus of web pages, used among other things for training	CCBot	Yes	718 of which 647 verified
Mistral AI	Model training	MistralAI-Training	Yes	0
OpenAI	ChatGPT search index	OAI-SearchBot	Yes	5,891 of which 5,604 verified
Parallel	Parallel’s search index, feeds its API for AI agents. All unverified requests date from February⁠–⁠July, before Parallel published its address list on 14 July 2026. Since August, 100% verify.	ShapBot	Yes	5,106 of which 3,391 verified
Linkup (linkup.so)	Stated index for a search API used by AI applications. Has not appeared in my logs since July 2026.	LinkupBot	No clear documentation	4,935
Perplexity	Search index	PerplexityBot	Yes	3,161 of which 2,925 verified
Exa (exa.ai)	Stated index of the Exa search engine for AI applications. In my logs almost exclusively since August 2026.	ExaSearchBot	No clear documentation	756
Meta	Index behind Meta AI search results	Meta-WebIndexer	Yes	401
You.com	Search index	YouBot	Yes	251
Anthropic	Search index. No request under this name in my logs came from the operator’s addresses. Some of them were looking for password files – I describe this in the section on suspicious user-agents.	Claude-SearchBot	Yes	213 0 verified
DuckDuckGo	Generative answers in DuckDuckGo	DuckAssistBot	Yes	74
Google	Crawl ordered by the site owner for Vertex AI. No request under this name in my logs came from the operator’s addresses.	Google-CloudVertexBot Googlebot	Yes	37 0 verified
Diffbot	Diffbot’s search engine and Knowledge Graph, not used for training	Diffbot	Yes, with possible contractual exceptions	3
Mistral AI	Search index for Vibe	MistralAI-Index	Yes	0
OpenAI	Fetch on user request. Fewer requests verify than you might expect, because OpenAI only publishes its current address list. Most unverified requests date from February⁠–⁠July and came from the Microsoft Azure network, so they are more likely older OpenAI address pools than impersonators. Since August, 79% verify.	OpenAI does not document it as a control token	May not apply	8,396 of which 3,249 verified
Google	Fetches sources added by the user. All 4,064 requests arrived under the old name `Google-NotebookLM`, all from Google addresses. The new name has not appeared in my logs even once – as of 6 October 2026. Blocking rules should cover both names.	none, server-side blocking only	Usually not	4,064 all verified
Anthropic	Fetch on user request. The unverified part is not only a scanner looking for password files. Claude-User also runs from Google Cloud addresses that were missing from Anthropic’s 2 October list: in my logs from 17⁠–⁠26 October, 51 of 81 unverified requests came from addresses that are on its 20 October list.	Claude-User	Yes	1,902 of which 966 verified
Google	Fetches RSS and Atom feeds for Google News and WebSub	none, user fetcher	Usually not	1,571 of which 1,522 verified
Google	Reads a page aloud on user request	none, user fetcher	Usually not	280 all verified
Perplexity	Fetch on user request. Since August 2026 no request under this name in my logs has come from Perplexity addresses.	Perplexity-User	Usually not	206 of which 51 verified
Mistral AI	Fetch on request of a Vibe user, with a link to the source	MistralAI-User	Yes	180 of which 142 verified
Diffbot	Fetch on request of a tool user	Diffbot-User	Yes	158
Google	Page fetches by Gemini Deep Research on user request. Google does not list it in its fetcher documentation. In my logs the requests came from Google addresses.	none, undocumented fetcher	No clear documentation	93
Parallel	Fetches a page on request of a Parallel user	none – Parallel describes it as a visibility signal, not a control mechanism	Not applicable	7 0 verified
Google	Google agents performing tasks on user request	none, server-side blocking only	Usually not	0
Amazon	Fetch on user request, e.g. a question put to Alexa	Amzn-User	May not apply	0
Meta	Fetch on user request, including by Meta AI agents performing tasks	Meta-ExternalFetcher	Sometimes ignored	0
Google	Fetches sources added by a Pinpoint user	none, user fetcher	Usually not	0
Google	Fetches URLs that developers provide in the metadata of Chrome extensions and themes	none, user fetcher	Usually not	0
Google	Feeds configured by a publisher in Google Publisher Center	none, user fetcher	Usually not	0
Apple	Fetches URLs tied to content registered in Apple Podcasts	none	No – Apple states that iTMS does not apply robots.txt	0
Google	Consent for Gemini training and grounding. Every appearance of this name in your logs is an impersonation	Google-Extended	Control token, no user-agent	176 impersonation only
Apple	Consent for Apple Intelligence training. Every appearance of this name in your logs is an impersonation	Applebot-Extended	Control token, no user-agent	5 impersonation only
shared IETF token	Shared opt-out from generative AI training across providers. Announced on 30 Oct 2026 by Google, Microsoft and OpenAI, not honored by anyone yet (Microsoft from 1 Feb 2027, Google in early 2027). At Google it works like blocking Google-Extended. User-agent: * never matches it (what this token changes)	__usage_ai-training	Control token, no user-agent	0 not a crawler
Google	Search engine, also feeds AI Overviews and AI Mode. The unverified part is largely impersonation, including 1,000 requests from two DigitalOcean addresses in March. Since August, 92% verify.	Googlebot	Yes	13,595 of which 11,241 verified
Microsoft	Bing search engine, also feeds Copilot	bingbot	Yes	10,079 of which 9,809 verified
Apple	Siri, Spotlight and Safari; also feeds Apple’s generative answers, and the data collected may be used for model training	Applebot	Yes	4,563 of which 4,407 verified
Huawei	Petal Search engine	PetalBot	Yes	4,426
Baidu	Chinese search engine; the -render variant renders JavaScript	Baiduspider	Yes	3,159
Yandex	Yandex search engine	YandexBot	Yes	2,255
Google	Image indexing	Googlebot-Image Googlebot	Yes	1,924 of which 1,923 verified
Seznam	Czech search engine	SeznamBot	Yes	1,503
Google	General-purpose crawler	GoogleOther	Yes	1,146 of which 1,119 verified
DuckDuckGo	Klasyczne wyniki wyszukiwania	DuckDuckBot	Yes	995
Yandex	JS rendering, images, favicons and landing pages for Yandex Direct ads. Yandex documentation distinguishes whether a robot follows the general rules in robots.txt. YandexImages does. YandexFavicons ignores the rules for `User-agent: *` and has to be named explicitly. YandexRenderResourcesBot fetches resources of pages available to YandexBot regardless of robots.txt, and YaDirectFetcher, which checks ad landing pages, does not use robots.txt at all.	separate tokens e.g. YandexImages	Partly – fully only YandexImages	825
Amazon	Search across Amazon products, including Alexa. No request under this name in my logs came from the operator’s addresses. Some of them were looking for password files – I describe this in the section on suspicious user-agents.	Amzn-SearchBot	Yes	244 0 verified
Microsoft	Page snapshots for previews in Bing. User-agent observed in my logs; Microsoft’s current documentation shows this traffic as bingbot.	no separate token; bingbot rules apply	Per bingbot rules	95 of which 27 verified
Qwant	French privacy-focused search engine; the -news variant serves Qwant News	Qwantbot	Yes, including Crawl-delay	93 all verified
Yahoo Japan (LY Corporation)	Agent Yahoo Japan. In my logs all requests came from the LY Corporation network, with a confirmed reverse DNS of `*.yahoo.co.jp`.	none	No clear documentation	87
Brave – partly confirmed	User-agent przypisywany indeksowi Brave Search. Brave says its crawler does not identify itself with a separate user-agent and will not fetch a page that Googlebot cannot fetch – so robots.txt controls it through the rules for Googlebot. Yet 6 requests signed `Bravebot` arrived in March and April 2026 from addresses with a confirmed reverse DNS of `*.searchbot.brave.app`. The other 80 came from hosting providers and Google Cloud customer machines, without confirmation, so the name alone is not worth trusting.	no separate token; Brave follows the rules for Googlebot	Per Googlebot rules	86 of which 6 verified
Google	GoogleOther variants for public images and video	GoogleOther-Image GoogleOther-Video GoogleOther	Yes	44 all verified
Yahoo	Yahoo search and partner content for Yahoo News, Finance and Sports	Slurp	Yes	13
Google	Google Shopping. No request under this name in my logs came from the operator’s addresses.	Storebot-Google	Yes	12 0 verified
Sogou	Chinese search engine	Sogou	Yes	9
Mojeek	Independent search engine with its own index	MojeekBot	Yes; does not support Crawl-delay	4
Google	Video indexing	Googlebot-Video Googlebot	Yes	2 of which 1 verified
Microsoft	Video previews in Bing	BingVideoPreview	Yes	0
Ahrefs / Yep	The Yep search engine, currently only for IndexNow requests. The index itself is built by AhrefsBot	YepBot	Yes, also honors Crawl-delay	0
Pinterest	Indexes content and products for Pins, updates prices and dead links	Pinterestbot	Yes; honors Crawl-delay up to a value of 1, treats higher values as 1	0
Meta	Preview of links shared across Meta services	facebookexternalhit	Sometimes ignored	6,694
Slack	Link previews, image fetching and Slack’s remaining requests	none	No – Slack states that it does not process robots.txt	907
Microsoft	Link previews in Skype	none	No clear documentation	439
Meta	Preview of links sent in WhatsApp	none	No clear documentation	426
LinkedIn	Preview of links shared on LinkedIn	LinkedInBot	No clear documentation	411
X (formerly Twitter)	Link previews in X cards	Twitterbot	No clear documentation	56
Discord	Link previews on Discord	Discordbot	No clear documentation	41
Google	Preview of links sent in Google Messages	none, user fetcher	Usually not	15
Microsoft	Page previews in Microsoft products	MicrosoftPreview	Yes	0
Ahrefs	Technical audit ordered by the site owner	AhrefsSiteAudit	Yes by default; the site owner can switch this off in the audit settings	167,450 all verified
whoever runs the program	Manual crawl, usually your own or your agency’s	Screaming Frog SEO Spider	Depends on the settings	16,013
Ahrefs	Index of links and content for Ahrefs and the Yep search engine	AhrefsBot	Yes	6,671 all verified
Babbar	Link index	Barkrowler	Yes	6,615
Semrush	Index of links and visibility data	SemrushBot	Yes	5,790
Serpstat	Backlink index	serpstatbot	Yes	3,406
Majestic	Link index	MJ12bot	Yes	2,076
DataForSEO	Index of SEO data sold through an API	DataForSeoBot	Yes	1,450
SE Ranking	Backlink index	SERankingBacklinksBot	Yes	1,091
SE Ranking	SE Ranking’s technical audit crawler (Website Audit). A user of the tool can change the user-agent in the audit settings.	SEBot-WA	Yes by default	1,082
Semrush	Technical audit on request	SiteAuditBot	Yes	962
Google	Quality assessment of ad landing pages	AdsBot-Google	Yes, but ignores User-agent: *	228 of which 227 verified
Google	Crawler for Google’s advertising products, including AdSense and Google Ad Manager. No request under this name in my logs came from the operator’s addresses.	Mediapartners-Google	Yes, but ignores User-agent: *	103 0 verified
Google	Quality assessment of mobile ad landing pages	AdsBot-Google-Mobile	Yes, but ignores User-agent: *	39 all verified
Meta	Improving Meta’s ads and other business products	meta-externalads	Yes	0
Microsoft	Quality control of landing pages for Microsoft Advertising	adidxbot	Yes	0
Moz	Crawler building the Moz Link Explorer link index	DotBot	No current clear documentation from the operator	0
Moz	Audits of sites added to Moz Pro campaigns by their owners	rogerbot	No current clear documentation from the operator	0
OpenAI	Verification of ad landing pages in ChatGPT; the data collected is not used for model training. OpenAI’s developer documentation describes OAI-AdsBot with its full user-agent and IP address list, but as robots.txt tags it still lists only `GPTBot` and `OAI-SearchBot`. The help article for advertisers says outright that OpenAI’s crawlers honor robots.txt and that a `Disallow` stops fetching immediately. Blocking this bot prevents your ads in ChatGPT from being verified.	OAI-AdsBot	Yes	0
UptimeRobot	Site availability monitoring, usually set up by the owner	UptimeRobot	Not applicable	69,895
Awario	Monitoring wzmianek o markach. In robots.txt, Awario gives the tokens AwarioSmartBot and AwarioRssBot; the request count refers to the AwarioBot user-agent.	AwarioSmartBot AwarioRssBot	Yes	1,797
security research and domain inventory	Scanning the public surface of websites, domain statistics	separate tokens	No clear documentation	71
Google	Detecting malware and abuse behind publicly shared links	none	No – Google states that it ignores robots.txt	61 all verified
media monitoring	Tracking mentions of brands and topics	separate tokens	No clear documentation	47
various operators, including NVIDIA	Tools for building image and document datasets, including for model training. In my logs under the names ImageBot, nvidiabot and cc-download, almost exclusively images and PDFs.	none	No clear documentation	3,591
Internet Archive	Archiving pages in the Wayback Machine	archive.org_bot	No current clear documentation from the operator	1,504
OpenWebSearch.eu	Crawler europejskiego projektu otwartego indeksu OpenWebSearch.eu. In my logs from the network of the Leibniz-Rechenzentrum computing center.	OWLer	No clear documentation	211
Carnegie Mellon University	Building the public ClueWeb corpus for search research. I found no current documentation for this crawler; the project’s earlier crawlers honored robots.txt.	ClueWeb-Crawler	No clear documentation	174
Slovak Academy of Sciences	Building public language corpora	none	No clear documentation	79
WP Rocket	Cache plugin: Preload builds the cache, Pre-fetch fetches links in advance. Your own traffic. On my site this is the largest entry in the whole table.	none	Not applicable	418,899
WP Rocket	A WP Rocket service that fetches pages and files from the operator’s servers for optimization. Also your own traffic if you use WP Rocket	none	Not applicable	183,835
any script	Not bots, just libraries for fetching pages. Behind each one stands a person or someone else’s program	none	Not applicable – depends on the script, the library enforces nothing	25,903
Google	Lighthouse and PageSpeed Insights tests, run manually by someone	none	Not applicable	14,508
Google	Search Console testing tools	Google-InspectionTool Googlebot	Yes	7,264 of which 7,262 verified
RSS readers	Fetching an RSS feed on behalf of a subscriber	separate tokens	Usually yes	2,437
Rank Math	SEO plugin checking links. If you run Rank Math, this is your own traffic	none	Not applicable	2,148
Google	Page prefetching for Chrome users. Looks like a bot, but a browser stands behind it	none	Not applicable	2,064 all verified
Microsoft (Bing app)	Browser built into the Bing app on Android. A person stands behind this traffic, not a crawler. It is not on Microsoft’s list of crawlers.	none	Not applicable	76
Google	Checking site ownership verification in Search Console	none, user fetcher	Usually not	18 all verified
any user	A Python library for building datasets. Someone was pulling content into a set of their own	none	Not applicable – depends on the script	14
Google	Delivering notifications from Google APIs, e.g. PubSubHubbub	APIs-Google	Yes, but ignores User-agent: *	0
impersonation – home networks in India	Login attacks through /xmlrpc.php under the name of the Jetpack plugin. If you do not use Jetpack, every such request is suspicious	none	Not applicable – block it on the server	12,405
not established	No documentation and no way to establish the operator. Judge them by behavior	none	Unknown	4,608
Exalead? – to verify	Historically the Exalead search engine. In my logs 97 of 98 requests are a bare `Exabot` user-agent from hosting providers, fetching only robots.txt.	Exabot	No clear documentation	98
Cohere? – to verify	Names attributed to Cohere. Cohere states that it currently does not use bots to collect content for training, and gives CohereBot only as an example of a future rule. In my logs all requests come from a scanner.	cohere-ai CohereBot	Not applicable – Cohere declares no active bot	60 none from Cohere
2ip.io	Service for checking IP addresses and websites	2ip bot	No clear documentation	51
unknown	Harvesting email addresses. The one entry where blocking needs no thought	EmailCrawler	Unknown; if you want certainty, block it on the server	21

If you block NotebookLM, block both names. On 16 July 2026 Google changed the user-agent name in its documentation from Google-NotebookLM to Google-GeminiNotebook and announced support for the old name until August 2026. In practice, up to 6 October 2026 all 4,064 requests from this fetcher reached my site under the old name and from Google addresses, while the new one did not appear even once. A rule in .htaccess or in your WAF therefore has to cover both: the old name, because it still works, and the new one, because according to the documentation it can take over at any moment.

How this table was built

I checked purposes and tokens in the operators’ documentation everywhere such documentation exists. Entries with no official source are described on the basis of the full user-agent, the address given in its string and its behavior in the logs – a missing confirmation is clearly marked. The frequency data comes from over 2.6 million requests in the podrez.pl logs for the period from 2 February to 6 October 2026.

I count each request in a single row – the one whose name appears earliest in the user-agent – so, for example, Googlebot-Image is not counted a second time as Googlebot. Some entries group several related user-agents from one operator, so there are more names than rows.

The table counts names, not bots. Scanners impersonate some AI bots, which is why for some entries I give how many requests passed IP verification.

Compiled by Ewelina Podrez-Siama, working in SEO since 2009. Last updated: 24 October 2026.

Sources: bot operator documentationForty-four documents in which I checked tokens, purposes and rules. You can check each one yourself.

OpenAI
Overview of OpenAI Crawlers – GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot
Advertiser Guidance for Allowing OpenAI Web Crawlers – the OAI-AdsBot token and rule compliance

Anthropic
Does Anthropic crawl data from the web… – ClaudeBot, Claude-User, Claude-SearchBot

Perplexity
Perplexity Crawlers – PerplexityBot, Perplexity-User, IP address lists

Google
Google’s common crawlers – Googlebot, GoogleOther, Google-Extended, Google-CloudVertexBot
Google special-case crawlers – AdsBot, Mediapartners, APIs-Google, Google-Safety
Google user-triggered fetchers – Gemini Notebook, Google-Agent, GoogleProducer
Verify requests from Google – three classes of bots, three JSON files, reverse DNS
Crawler documentation changelog – change dates, including 17.09.2026, 16.07.2026 and 20.03.2026
AI Features and Your Website – preview directives with regard to AI Overviews and AI Mode

Apple
About Applebot – Applebot, Applebot-Extended, iTMS, IP address list, inheriting rules from Googlebot

Meta
Meta Web Crawlers – meta-externalagent, meta-webindexer, meta-externalfetcher, facebookexternalhit; the last two may skip robots.txt

Amazon
About Amazonbot – Amazonbot, Amzn-SearchBot, Amzn-User, three IP address lists

Mistral AI
Mistral crawlers – MistralAI-Training, MistralAI-Index, MistralAI-User

Microsoft
Overview of Bing crawlers – bingbot, adidxbot, BingPreview, MicrosoftPreview, BingVideoPreview

Other search engines
DuckDuckGo – DuckDuckBot and DuckAssistBot – rule compliance, IP address lists
YouBot – robots.txt, Crawl-delay, reverse DNS
Yandex – robots in server logs – which robots follow the general rules, reverse DNS
Seznam – Crawling control
Petal Search – PetalBot
Baidu – Baiduspider – statement that it follows the robots protocol
Sogou – spider and robots.txt
Yahoo – Why is Slurp crawling my page?
Mojeek – MojeekBot

SEO tools
Semrush – SemrushBot and SiteAuditBot
Majestic – MJ12bot
Serpstat – serpstatbot
DataForSEO – DataForSeoBot
SE Ranking – SERankingBacklinksBot
Babbar – Barkrowler
SE Ranking – SEBot-WA – the Website Audit crawler, reverse DNS
Awario Crawlers – AwarioSmartBot and AwarioRssBot

Other operators
Common Crawl – CCBot
Parallel – ShapBot and Shap-User – version of 11.08.2026
Cohere Web Crawlers – statement that Cohere currently does not use bots to collect content for training
Qwant – QwantBot and robots.txt – user-agent variants, IP address list, Crawl-delay
Ahrefs Bots – AhrefsBot, AhrefsSiteAudit, IP address lists
Slack Robots – three agents and a statement about not honoring the rules
Yep – YepBot
Pinterest – Pinterestbot
Diffbot – Does Diffbot respect robots.txt?
Brave Search Crawler

Context
Cloudflare – Your site, your rules: new AI traffic options – the Search, Agent and Training categories, default settings from 15.09.2026 and blocking of multi-purpose crawlers under a Training block

All links checked on 7 and 8 October 2026. Entries marked in the table as “no clear documentation” have no counterpart on this list – I describe them solely on the basis of the user-agent and behavior in the logs.

Verification. For some entries I give, under the number, how many requests passed verification: an IP address from the operator’s official list, or a reverse DNS in its domain confirmed by a forward lookup. I downloaded the lists on 7 October 2026, so older requests from addresses the operator no longer uses may come out as unverified. That is why I don’t mark ChatGPT-User and ShapBot this way – their unverified requests come mainly from February⁠–⁠July, and since August 79% and 100% of them verify, respectively.

The “honors robots.txt” column. It is based on operator documentation, not on my own tests. “Stated by operator” means the operator says so, though reports of deviations circulate. “No clear documentation” means exactly what it says – I found no reliable source and I am not guessing.

robots.txt

Why doesn’t robots.txt block every AI bot?

An entry in robots.txt does not work the same way on every bot. Crawlers that move around the web on their own initiative usually follow it. With fetchers triggered by a user things get complicated, because here every operator has a policy of its own.

Automated crawlers
They move around the web on the operator’s initiative: GPTBot, ClaudeBot, PerplexityBot, Googlebot. They honor robots.txt and this file was created with them in mind.
User-triggered fetchers
They fetch a page because a specific user action required it. Operators differ here. Anthropic and Mistral provide separate tokens (Claude-User, MistralAI-User) and declare that they honor them. Google, OpenAI, Perplexity, Amazon and Meta note that the rules may not apply, because the request is initiated by a user.
Control tokens without a bot
Google-Extended and Applebot-Extended send no requests at all. You put them in robots.txt, but you will never see them in your logs. Google-Extended covers training of Gemini models and grounding, meaning content passed to the model at the moment of answering. Applebot-Extended covers training of Apple’s models.

The practical conclusion: before you block a fetcher triggered by a user, check the policy of that particular operator. A Disallow for Claude-User or MistralAI-User genuinely closes the door, because Anthropic and Mistral declare that they honor those tokens. With Google, OpenAI, Perplexity, Amazon and Meta the same entry is a request, so if you need certainty, add a rule at the server or WAF level. I described how to check this on your own site in the article on server log analysis.

Impersonation

Suspicious user-agents: impossible names and names to verify

Since June 2026 a scanner has been active in my logs that signs itself with the names of AI bots and looks for files with passwords and keys: .env, .aws/credentials, Firebase configuration. It comes mainly from Google Cloud customer machines and cheap hosting providers, and it rotates the bot names. These names are worth knowing, because they look credible and slip through a filter easily.

Some of the names cannot come from the operator, because its own documentation says those tokens do not crawl at all. Others are real bot names, but not a single request under them came from the operator’s addresses. Still others appear in no documentation whatsoever. The same scanner also signs itself with names that have their own rows in the table – ChatGPT-User, Claude-User, GPTBot or Amazonbot – and inflates their counts.

Name in the log	Status	Why	What it was looking for on my site
`Google-Extended`	Impossible	A control token in `robots.txt`; Google does not send requests with it	`/.aws/credentials`, `/.env.test`
`Applebot-Extended`	Impossible	Apple states outright that this token does not crawl pages	`/privatekey.key`, `/secrets.json`
`Claude-SearchBot`	Impersonation	A real Anthropic bot name, but not a single request came from addresses on Anthropic’s list	files with passwords and keys
`Amzn-SearchBot`	Impersonation	A real Amazon bot name, but not a single request came from Amazon addresses	files with passwords and keys
`anthropic-ai`	Scanner	A name outside Anthropic’s current documentation	`/__/firebase/init.json`, `/env.js`
`Claude-Web`	Scanner	A name outside Anthropic’s current documentation	`/secrets.json`, `/.env.old`
`cohere-ai`, `CohereBot`	Scanner	Cohere states that it currently does not use bots to collect content for training	`/.aws/credentials`, `/api/v1/env`
`GrokBot`, `Kimi-SearchBot`, `DeepSeekBot` and others	Scanner	Names outside the operators’ documentation, the same traffic pattern	`.env`, `credentials`, Firebase configuration
`BingIndexCrawler`	To verify	Microsoft’s network, but without a `search.msn.com` reverse DNS; Bing crawls as `bingbot`	images from `/wp-content/uploads/`

podrez.pl logs, 2 Feb – 6 Oct 2026. Number of occurrences: Amzn-SearchBot 244, Claude-SearchBot 213, Google-Extended 176, cohere-ai and CohereBot 60, anthropic-ai 28, Claude-Web 20, BingIndexCrawler 10, Applebot-Extended 5. The other fake AI names – GrokBot, xAI-Grok, xAI-SearchBot, Kimi-SearchBot, MoonshotBot, Qwenbot, DeepSeekBot, ChatGLM-Spider, Hunyuan, YiBot and MistralBot – between 20 and 38 occurrences each.

The pattern is clear. Suspicious requests reach for:

tokens that are not user-agents,
names absent from documentation,
the names of real bots, counting on every filter to let those through.

A name that does not match the documentation need not mean bad intent, though – BingIndexCrawler came from Microsoft’s network and fetched only images – which is why the IP address decides, not the name alone. The problem is not limited to my site either: Common Crawl explicitly warns in its documentation that it knows of crawlers falsely claiming to be CCBot, and recommends verifying the user-agent.

Verification

How to check whether a bot is real: verification by IP address

A user-agent can be faked in ten seconds, so the name alone proves nothing. Certainty comes only from matching the name with the IP address and the operator’s official ranges, and where possible with a reverse and forward DNS check. Google is also testing cryptographic authentication of bots, that is Web Bot Auth.

Operator	Source of addresses
OpenAI	`openai.com/gptbot.json`, `openai.com/searchbot.json`, `openai.com/chatgpt-user.json`, `openai.com/adsbot.json`
Anthropic	`claude.com/crawling/bots.json`
Perplexity	`perplexity.com/perplexitybot.json`, `perplexity.com/perplexity-user.json`
Google – common crawlers	`common-crawlers.json`; reverse DNS ends in `googlebot.com`
Google – special-case crawlers	`special-crawlers.json`; reverse DNS ends in `google.com`
Google – user fetchers and agents	separate JSON files; reverse DNS ends in `google.com` or `gae.googleusercontent.com`
Apple	`search.developer.apple.com/applebot.json`; reverse DNS ends in `applebot.apple.com`
Mistral	`mistral.ai/mistralai-user-ips.json`, `mistral.ai/mistralai-index-ips.json`; for `MistralAI-Training` the list is not published
Amazon	three separate lists: `/ip-addresses/`, `/searchbot-ip-addresses/`, `/live-ip-addresses/`
Parallel	docs.parallel.ai/resources/shapbot.json
Microsoft	the Verify Bingbot tool in Bing Webmaster Tools; reverse DNS ends in search.msn.com
Common Crawl	`index.commoncrawl.org/ccbot.json`; reverse DNS ends in `crawl.commoncrawl.org`
Qwant	`help.qwant.com/qwantbot.json`; reverse DNS ends in `qwant.com`
Meta	addresses from autonomous system AS32934, retrievable with `whois -h whois.radb.net -- '-i origin AS32934'`; the list changes often
Baidu	reverse DNS ends in `baidu.com` or `baidu.jp`; Baidu does not publish an address list
DuckDuckGo	`duckduckgo.com/duckassistbot.json` for DuckAssistBot
You.com	reverse DNS ends in `search.you.com`; cryptographic signatures as well
Yandex	reverse DNS ends in `yandex.ru`, `yandex.net` or `yandex.com`; Yandex does not publish an address list
SE Ranking	reverse DNS ends in `seranking.com` (SERankingBacklinksBot) or `sr-srv.net` (SEBot-WA)
Babbar (Barkrowler)	reverse DNS ends in `babbar.eu`
Ahrefs (also YepBot)	ranges and single addresses via `api.ahrefs.com`; reverse DNS ends in `ahrefs.com` or `ahrefs.net`
Pinterest	reverse DNS ends in `pinterest.com` or `pinterestcrawler.com`

How many of these are real bots? Verifying my logs

The table at the top counts names in the logs, not bots. So for every bot whose addresses can be checked, I also show how many requests came from the operator’s official ranges or passed a reverse and forward DNS check. Same period as the table: 2 Feb – 6 Oct 2026, podrez.pl.

Bot	Requests	Verified	Share
GPTBot	10,218	9,925	97%
OAI-SearchBot	5,891	5,604	95%
ChatGPT-User	8,396	3,249	39%*
ClaudeBot	6,789	6,518	96%
Claude-User	1,902	966	51%
Claude-SearchBot	213	0	0%
PerplexityBot	3,161	2,925	93%
Perplexity-User	206	51	25%*
Amazonbot	8,111	7,819	96%
Applebot	4,563	4,407	97%
Googlebot	13,595	11,241	83%
bingbot	10,079	9,809	97%

* Since August 2026: ChatGPT-User 79%, Perplexity-User 0%. OpenAI only publishes its current address list, so older requests from the Microsoft Azure network do not match it.

To check that this is not a quirk of one site, I compared the week of 17⁠–⁠23 October 2026 in the logs of nine sites I look after. The picture is the same, and for some names even starker:

Perplexity-User: 169 requests, not a single one from Perplexity’s addresses,
DuckAssistBot: 207 requests, 23 genuine,
PerplexityBot: every third request fake,
ChatGPT-User: every sixth fake,
OAI-SearchBot and GPTBot: every tenth and every fourteenth fake.

The moral? Before you boast that “ChatGPT visits you 200 times a week”, check the addresses. A report that counts AI bots by name alone can show a scanner looking for a .env file as your most loyal Perplexity reader.

Blocking

How to block AI bots in robots.txt? Ready-made examples.

Before you block anything: in most cases I do not recommend it. A block cuts you off from generative answers, and so from a channel where your brand can be mentioned and cited. For most companies, experts and shops that is a loss, not a gain.

The exceptions are real and concern publishers, portals and media above all – those who make a living from selling content or from traffic on their own site. There, a block can be a justified business decision – but still a decision, not a default setting.

So treat the configurations below as a starting point for an informed choice, not as a recommendation.

The first variant limits the main crawlers used for model training. The second blocks separately identified AI bots and fetchers, leaving access open to Googlebot, bingbot and Applebot. Both are a starting point, not a universal configuration for every site. Keep in mind that a block does not work retroactively. A model that has already been trained will not forget what it learned up to its knowledge cutoff.

Variant 1 · No training

I protect my content

I let in everything that can cite me and cut off content collection for model training.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
User-agent: cohere-ai
User-agent: CohereBot
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: Amzn-SearchBot
User-agent: ShapBot
Allow: /

The cost: with multi-purpose crawlers such as Amazonbot or CCBot, a block also switches off uses other than training.

Variant 2 · Separate AI bots

Google and Apple stay

I block separate training crawlers, AI indexes and user fetchers, but leave classic search engines open. This does not remove the site from AI Overviews, AI Mode or Apple’s generative answers.

# search engines stay
User-agent: Googlebot
User-agent: bingbot
User-agent: Applebot
Allow: /

# model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
User-agent: cohere-ai
User-agent: CohereBot
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# indexes behind generative answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: MistralAI-Index
User-agent: Meta-WebIndexer
User-agent: DuckAssistBot
User-agent: Amzn-SearchBot
User-agent: LinkupBot
User-agent: ShapBot
User-agent: YouBot
User-agent: Diffbot
Disallow: /

# fetches on user request
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
Disallow: /

The effect: you limit access for systems that have bots of their own – ChatGPT, Claude, Perplexity, Mistral. This is not a full exit from generative answers: AI Overviews and AI Mode are fed by Googlebot, Copilot by bingbot, and Applebot can be a source of context for Apple’s answers.

The second variant has a hole and it is better to know about it right away. The last group are fetchers triggered by a user, and some of them may ignore robots.txt. Gemini Notebook and Google-Agent do not even have a token of their own. If the block is to be airtight, it has to be added at the server level. An example for Apache and the .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Google-GeminiNotebook|Google-NotebookLM|Google-Agent|GoogleProducer|ChatGPT-User|Perplexity-User|Amzn-User|Meta-ExternalFetcher|Shap-User) [NC]
RewriteRule ^ - [F,L]

One caveat that is easy to forget: robots.txt is not a tool for protecting data. The file is public, the directives are not access control, and a blocked URL can still end up in the index on the strength of external links. Genuinely private sections are secured with server-side authentication.

robots.txt is not the only layer.

If a more specific matching group exists for a bot, it takes precedence over User-agent: *, and the wildcard applies only when nothing more specific matches.
Fetchers triggered by a user are a separate case: some honor dedicated tokens, some may skip them and need a rule at the server or WAF level.
If your site sits behind Cloudflare, check its AI bot settings too: since 15 September 2026 a block on the Training category – in the new settings or through the old “Block AI bots” toggle – also covers Googlebot, bingbot and Applebot, because these bots collect data both for search and for training. Your robots.txt can be set up perfectly and Googlebot will still be refused at the Cloudflare level.

Google

How to limit AI Overviews? On nosnippet, data-nosnippet and max-snippet

robots.txt tells compliant bots whether they should fetch a page. Meta directives, by contrast, control indexing and which fragments can reach the results and the AI features. These are two different layers, and with AI Overviews the second one is often the only one you have left.

The key point

There is no separate opt-out from AI Overviews and AI Mode that would leave your ordinary presence in Google untouched. They are fed by the same Googlebot and the same index, and Google-Extended covers only Gemini training and grounding, so blocking it changes nothing here.

You can, however, limit how much content reaches those features. Google lists four directives: nosnippet, data-nosnippet, max-snippet and noindex. Each of them works on ordinary results and on AI features at the same time – you cannot separate one from the other.

Directive	What it does	What it costs you
`nosnippet`	Blocks the display of a page fragment in every form of results, including AI Overviews and AI Mode	The ordinary description under the title also disappears, along with the chance of a direct answer
`data-nosnippet`	The same, but only for the marked HTML fragment	The rest of the page works normally – the most precise tool in this set
`max-snippet:N`	Limits the fragment length to N characters and with it the amount of content passed to the model	At a low value the effect comes close to `nosnippet`
`noindex`	Removes the page from every form of Google results, including AI Overviews and AI Mode	The page also disappears from ordinary search results
`noarchive`	At Amazon it means “do not use this page to train models”	Works only with operators that honor this directive

Apple honors nosnippet – also in the X-Robots-Tag: applebot: nosnippet header – as an opt-out from the use of content as additional context in generative answers. Marking a page as isAccessibleForFree: false in structured data has the same effect, and the page can still appear in Apple’s search results. Amazon declares that it honors noarchive, noindex and none, but does not support crawl-delay.

The most precise tool is data-nosnippet. It excludes only the marked fragment of a page from previews and from direct use by AI features. Use it deliberately: the very paragraphs that would most easily replace a visit to your site – definitions, step lists, tables – are usually the ones that get a site cited in the first place.

Data from my logs

What AI bots take from your site: content or files?

The number of visits alone says little. What is more interesting is how many of them land on a page with content, and how many on CSS files, images and API endpoints.

Bot	Successful requests	Share of hits on content
Perplexity-User	61	91.8%
CCBot	569	81.5%
ChatGPT-User	7,805	74.2%
Claude-User	984	63.7%
PerplexityBot	2,749	62.1%
OAI-SearchBot	4,145	33.5%
GPTBot	5,843	30.2%
ClaudeBot	4,289	21.8%
Google-GeminiNotebook	3,686	19.9%
Applebot	4,051	17.7%
Amazonbot	5,415	15.1%

podrez.pl logs, 2 Feb – 6 Oct 2026; a successful request is a response with status 200. Content means any URL other than the home page, robots.txt, sitemaps, the /wp-content/, /wp-includes/ and /wp-json/ directories and .php files. Perplexity-User, ChatGPT-User and Claude-User land mostly straight on content, but the class of bot alone does not determine its behavior: CCBot has a high share of hits on content, and Gemini Notebook a low one.

The definition of content changes the conclusions here more than you might think. One in four successful ChatGPT-User hits is the home page – if I counted it as content, its share would rise to 98.6%. That looks more like checking who the author of the site is than reading a particular article. The second observation concerns the last two months: since 6 August GPTBot fetched content in only 13% of its successful hits, and on average visited the site almost four times less often than before.

Tools and lists

More tools and lists

Free tool
Entity Observatory
See a page’s structured data as a relation map and add the Wikidata record next to it.
Free tool
Are you in Google’s Knowledge Graph?
Enter a name and check whether Google knows you as an entity and whether it matches Wikidata.
Free tool
AI bots in your logs
Drop a server log and see which AI bots visit, which come from confirmed IP addresses and which ones your server turns away. Your file stays on your computer.
Free tool
Is your site in Common Crawl?
Enter a domain to see how many pages CCBot saved in the last 6 crawls and whether anything blocks it.
Table
AI knowledge cutoff dates
How current is the data in ChatGPT, Claude, Gemini and Grok? Cutoff dates checked against provider documentation.

Bots don’t come to your site? Maybe they can’t! Check it in High Five, AI! by Maciej Chmurkowski.

Questions

Frequently asked questions

How do I check which bot visited my site?

The bot name sits in the user-agent field of your server’s access log, usually in access.log or domain.com.log in the logs directory. Copy the name and search for it in the table above. If you want to be sure the bot is genuine, compare its IP address with the ranges published by the operator. The quickest way is my AI bots in logs tool: you load the log in your browser, the file never leaves your computer, and the tool recognises the bots and checks their IP addresses.

Does blocking GPTBot remove me from ChatGPT results?

No. GPTBot is responsible for collecting content for model training, while presence in ChatGPT search results is handled by OAI-SearchBot. These are two independent tokens and you can set them differently: block training, leave search open. Anthropic and Mistral apply an analogous three-way split. Amazon also divides its traffic across three agents, but its Amazonbot is a multi-purpose crawler that only may be used for training. Perplexity, by contrast, separates just the search index from fetches on user request and has no documented training crawler.

Why don’t I see Google-Extended in my logs?

Because Google-Extended is not a bot. It is a control token in robots.txt with which you decide about the use of your content for training Gemini models and for grounding. The fetching itself is done by Google’s ordinary robots. If you do see this name in your log, you are looking at an impersonator.

How do I opt out of AI Overviews?

Through robots.txt you can’t. AI Overviews and AI Mode use Google’s ordinary index, built by Googlebot, and the Google-Extended token covers only the training of Gemini models and grounding. The only available tool is the preview directives: nosnippet, data-nosnippet, max-snippet and noindex. Each of them also limits your ordinary search result, so leaving AI Overviews means partly leaving Google.

Will a robots.txt block stop every AI bot?

No. Fetchers triggered by a user, such as Gemini Notebook or ChatGPT-User, may skip robots.txt, because the request is initiated by a person, not a machine. To stop them you need a rule at the server level, in your WAF configuration or in the .htaccess file.

What changed with NotebookLM?

On 16 July 2026 Google updated its documentation and changed the user-agent name from Google-NotebookLM to Google-GeminiNotebook, after the product itself was renamed Gemini Notebook. It announced support for the old name until August 2026. In my logs, up to 6 October all requests from this fetcher still arrived under the old name, while the new one did not appear even once, so blocking rules should cover both.

Last updated: 24 October 2026 I check bot tokens and purposes in the operators’ documentation (listed in the sources above; last full review on 7⁠–⁠8 October 2026, including Cloudflare). Entries marked as “no clear documentation” are described on the basis of observation. Log data comes from the podrez.pl access logs for 2 Feb – 6 Oct 2026, over 2.6 million requests, and I count each request in a single row. I verify IP addresses against operator lists downloaded on 7 October 2026.

Page history and timeline of changes at the operators

History of this page 08.10.2026 – log data extended to 6 October 2026, new entries (including ExaSearchBot, Gemini-Deep-Research, OWLer and the fake Jetpack), verified-request counts for some bots and an expanded section on bot impersonation.
06.08.2026 – first version.

Timeline of changes at the operators 17.09.2026 – Google extended the Mediapartners-Google documentation to cover various advertising products, including AdSense and Google Ad Manager.
15.09.2026 – the date Cloudflare announced on 1 July for new default settings: on ad-supported pages of newly onboarded domains the Training and Agent categories are blocked, and crawlers that combine search with training – including Googlebot, bingbot and Applebot – fall under a Training block.
August 2026 – the end of support for the old Google-NotebookLM name announced by Google; in my logs it still works (as of 6 October).
16.07.2026 – Google changed the NotebookLM user-agent name to Google-GeminiNotebook; the old name supported for a transition period.
01.07.2026 – Cloudflare introduced a split of bots into the Search, Agent and Training categories.
20.03.2026 – Google documented the Google-Agent user-agent.

Next steps

Not sure which bots your site lets in?

The name in the log is only the beginning. Far more interesting is what that bot got in response – and whether it got anything at all. I described how to check that, and what I found on my own site, in the article on server log analysis.

In my AI search visibility projects I go through server logs whenever they are available, because robots.txt alone won’t tell you what a bot actually received.

Log analysis covering AI bots, search engine crawlers, SEO tools and suspicious traffic is part of my audit. If you would rather start with a conversation, book a short call.

Book a consultation