--- title: "What bot is this? List of AI crawlers, search bots and user-agents" description: "Ewelina Podrez-Siama · tools and lists What bot is this? List of AI crawlers, search bots and user-agents This page answers one question: what is this bot and what should you do about it. Looking at your logs and seeing a name you don't recognize? Check the operator, what the bot does, its robots.txt rules" date: 2026-08-06 author: "Ewelina Podrez-Siama" url: "https://podrez.pl/en/bots/" --- ![Paper collage: a bot directory with crawler icons, a spider, a magnifying glass and servers](https://podrez.pl/wp-content/uploads/2026/08/co-to-za-bot.jpg "What bot is this? List of AI crawlers, search bots and user-agents")Ewelina Podrez-Siama · tools and lists What bot is this? List of AI crawlers, search bots and user-agents ================================================================== This page answers one question: what is this bot and what should you do about it. Looking at your logs and seeing a name you don’t recognize? Check the operator, what the bot does, its robots.txt rules and the number of requests I recorded on my own site. [Go to the search ](#table)[Check bots in your logs](https://podrez.pl/en/ai-bots-in-logs/) [Search](#table)[robots.txt](#classes)[Suspicious bots](#impersonators)[IP check](#verification)[Blocking](#robots)[AI Overviews](#meta)[What they fetch](#data)[FAQ](#faq) **120**bots and user-agents in the table **44**operator documents behind the table **2.6M**requests from podrez.pl logs Search · 120 entries Bot and user-agent list \[searchable\] -------------------------------------- The table covers AI crawlers, classic search engines, SEO tools, link previews from messaging apps, monitoring services and plain HTTP libraries. Narrow the category or type a name – the table filters as you type. Click a bot’s name to see a card with its description and notes, plus a ready-made robots.txt entry wherever the bot has a usable token. A bot with an “i” icon has additional notes. The “Requests” column shows how many requests carrying a given bot name in the user-agent field I recorded on my site between 2 February and 6 October 2026. Read these numbers as a hint of what to expect, not as a market average – and remember that they count names, not bots: scanners impersonate some of the names. 1\. Narrow the category All AI: training AI: indexes and answers User-triggered Tokens without a bot Search engines Link previews SEO and marketing Monitoring and security Research and archives Tools and libraries Unknown or suspicious 2. Or type the bot name straight away, or paste the whole user-agent from your logs Clear Popular: [GPTBot](#) · [Googlebot](#) · [ChatGPT-User](#) · [AhrefsBot](#) · [SemrushBot](#) **I couldn’t find that user-agent.** Check that the name you pasted is complete. If it isn’t here, judge the bot by its IP address and by what it requested – I describe that in the section [How to check whether a bot is real](#verification). ### What do the bot categories mean?Eleven classes of bots, one sentence each. AI: trainingcollects content so that models can learn from it AI: indexes and answersbuilds an index that a generative system quotes from User-triggeredfetches a page because someone has just asked about something Tokens without a botsend no requests, they exist only as robots.txt entries Search enginesclassic search engines and their helper bots Link previewsgenerate the link thumbnail in a messenger or on social media SEO and marketingSEO tools, audits and checks on ad landing pages Monitoring and securitysite availability, brand mentions, abuse scanning Research and archivesacademic corpora, datasets and page archiving Tools and librariestraffic triggered by a browser, plugin, app or someone’s own script Unknown or suspiciousoperator, purpose or user-agent authenticity has not been confirmed The table scrolls sideways → Bot and operatorPurposerobots.txt tokenHonors itRequests podrez.pl, 2 Feb – 6 Oct 2026 GPTBotOpenAIModel trainingGPTBotYes10,218 of which 9,925 verified AmazonbotAmazonImproving Amazon products; may also be used for model trainingAmazonbotYes8,111 of which 7,819 verified ClaudeBotAnthropicModel trainingClaudeBotYes6,789 of which 6,518 verified meta-externalagentMetaContent collection, including for trainingMeta-ExternalAgentStated by operator4,212 BytespiderByteDanceCollects content for ByteDance modelsBytespiderNo clear documentation968 CCBotCommon CrawlOpen corpus of web pages, used among other things for trainingCCBotYes718 of which 647 verified MistralAI-TrainingMistral AIModel trainingMistralAI-TrainingYes0 OAI-SearchBotOpenAIChatGPT search indexOAI-SearchBotYes5,891 of which 5,604 verified ShapBotParallelParallel’s search index, feeds its API for AI agents. All unverified requests date from February–July, before Parallel published its address list on 14 July 2026. Since August, 100% verify.ShapBotYes5,106 of which 3,391 verified LinkupBotLinkup (linkup.so)Stated index for a search API used by AI applications. Has not appeared in my logs since July 2026.LinkupBotNo clear documentation4,935 PerplexityBotPerplexitySearch indexPerplexityBotYes3,161 of which 2,925 verified ExaSearchBotExa (exa.ai)Stated index of the Exa search engine for AI applications. In my logs almost exclusively since August 2026.ExaSearchBotNo clear documentation756 meta-webindexerMetaIndex behind Meta AI search resultsMeta-WebIndexerYes401 YouBotYou.comSearch indexYouBotYes251 Claude-SearchBotAnthropicSearch index. No request under this name in my logs came from the operator’s addresses. Some of them were looking for password files – I describe this in the section on suspicious user-agents.Claude-SearchBotYes213 0 verified DuckAssistBotDuckDuckGoGenerative answers in DuckDuckGoDuckAssistBotYes74 Google-CloudVertexBotGoogleCrawl ordered by the site owner for Vertex AI. No request under this name in my logs came from the operator’s addresses.Google-CloudVertexBot GooglebotYes37 0 verified DiffbotDiffbotDiffbot’s search engine and Knowledge Graph, not used for trainingDiffbotYes, with possible contractual exceptions3 MistralAI-IndexMistral AISearch index for VibeMistralAI-IndexYes0 ChatGPT-UserOpenAIFetch on user request. Fewer requests verify than you might expect, because OpenAI only publishes its current address list. Most unverified requests date from February–July and came from the Microsoft Azure network, so they are more likely older OpenAI address pools than impersonators. Since August, 79% verify.OpenAI does not document it as a control tokenMay not apply8,396 of which 3,249 verified Google-GeminiNotebook still logged as Google-NotebookLMGoogleFetches sources added by the user. All 4,064 requests arrived under the old name `Google-NotebookLM`, all from Google addresses. The new name has not appeared in my logs even once – as of 6 October 2026. Blocking rules should cover both names.none, server-side blocking onlyUsually not4,064 all verified Claude-UserAnthropicFetch on user request. The unverified part is not only a scanner looking for password files. Claude-User also runs from Google Cloud addresses that were missing from Anthropic’s 2 October list: in my logs from 17–26 October, 51 of 81 unverified requests came from addresses that are on its 20 October list.Claude-UserYes1,902 of which 966 verified FeedFetcher-GoogleGoogleFetches RSS and Atom feeds for Google News and WebSubnone, user fetcherUsually not1,571 of which 1,522 verified Google-Read-AloudGoogleReads a page aloud on user requestnone, user fetcherUsually not280 all verified Perplexity-UserPerplexityFetch on user request. Since August 2026 no request under this name in my logs has come from Perplexity addresses.Perplexity-UserUsually not206 of which 51 verified MistralAI-UserMistral AIFetch on request of a Vibe user, with a link to the sourceMistralAI-UserYes180 of which 142 verified Diffbot-UserDiffbotFetch on request of a tool userDiffbot-UserYes158 Gemini-Deep-ResearchGooglePage fetches by Gemini Deep Research on user request. Google does not list it in its fetcher documentation. In my logs the requests came from Google addresses.none, undocumented fetcherNo clear documentation93 Shap-UserParallelFetches a page on request of a Parallel usernone – Parallel describes it as a visibility signal, not a control mechanismNot applicable7 0 verified Google-AgentGoogleGoogle agents performing tasks on user requestnone, server-side blocking onlyUsually not0 Amzn-UserAmazonFetch on user request, e.g. a question put to AlexaAmzn-UserMay not apply0 meta-externalfetcherMetaFetch on user request, including by Meta AI agents performing tasksMeta-ExternalFetcherSometimes ignored0 Google-PinpointGoogleFetches sources added by a Pinpoint usernone, user fetcherUsually not0 Google-CWSGoogleFetches URLs that developers provide in the metadata of Chrome extensions and themesnone, user fetcherUsually not0 GoogleProducerGoogleFeeds configured by a publisher in Google Publisher Centernone, user fetcherUsually not0 iTMSAppleFetches URLs tied to content registered in Apple PodcastsnoneNo – Apple states that iTMS does not apply robots.txt0 Google-ExtendedGoogleConsent for Gemini training and grounding. Every appearance of this name in your logs is an impersonationGoogle-ExtendedControl token, no user-agent176 impersonation only Applebot-ExtendedAppleConsent for Apple Intelligence training. Every appearance of this name in your logs is an impersonationApplebot-ExtendedControl token, no user-agent5 impersonation only \_\_usage\_ai-trainingshared IETF tokenShared opt-out from generative AI training across providers. Announced on 30 Oct 2026 by Google, Microsoft and OpenAI, not honored by anyone yet (Microsoft from 1 Feb 2027, Google in early 2027). At Google it works like blocking Google-Extended. User-agent: \* never matches it ([what this token changes](https://podrez.pl/en/usage-ai-training/))\_\_usage\_ai-trainingControl token, no user-agent0 not a crawler GooglebotGoogleSearch engine, also feeds AI Overviews and AI Mode. The unverified part is largely impersonation, including 1,000 requests from two DigitalOcean addresses in March. Since August, 92% verify.GooglebotYes13,595 of which 11,241 verified bingbotMicrosoftBing search engine, also feeds CopilotbingbotYes10,079 of which 9,809 verified ApplebotAppleSiri, Spotlight and Safari; also feeds Apple’s generative answers, and the data collected may be used for model trainingApplebotYes4,563 of which 4,407 verified PetalBotHuaweiPetal Search enginePetalBotYes4,426 BaiduspiderBaiduChinese search engine; the -render variant renders JavaScriptBaiduspiderYes3,159 YandexBotYandexYandex search engineYandexBotYes2,255 Googlebot-ImageGoogleImage indexingGooglebot-Image GooglebotYes1,924 of which 1,923 verified SeznamBotSeznamCzech search engineSeznamBotYes1,503 GoogleOtherGoogleGeneral-purpose crawlerGoogleOtherYes1,146 of which 1,119 verified DuckDuckBotDuckDuckGoKlasyczne wyniki wyszukiwaniaDuckDuckBotYes995 YandexRenderResourcesBot YandexImages YandexFavicons YaDirectFetcherYandexJS rendering, images, favicons and landing pages for Yandex Direct ads. Yandex documentation distinguishes whether a robot follows the general rules in robots.txt. YandexImages does. YandexFavicons ignores the rules for `User-agent: *` and has to be named explicitly. YandexRenderResourcesBot fetches resources of pages available to YandexBot regardless of robots.txt, and YaDirectFetcher, which checks ad landing pages, does not use robots.txt at all.separate tokens e.g. YandexImagesPartly – fully only YandexImages825 Amzn-SearchBotAmazonSearch across Amazon products, including Alexa. No request under this name in my logs came from the operator’s addresses. Some of them were looking for password files – I describe this in the section on suspicious user-agents.Amzn-SearchBotYes244 0 verified BingPreview / bingbotMicrosoftPage snapshots for previews in Bing. User-agent observed in my logs; Microsoft’s current documentation shows this traffic as bingbot.no separate token; bingbot rules applyPer bingbot rules95 of which 27 verified Qwantbot Qwantbot-newsQwantFrench privacy-focused search engine; the -news variant serves Qwant NewsQwantbotYes, including Crawl-delay93 all verified Y!J-DLCYahoo Japan (LY Corporation)Agent Yahoo Japan. In my logs all requests came from the LY Corporation network, with a confirmed reverse DNS of `*.yahoo.co.jp`.noneNo clear documentation87 BravebotBrave – partly confirmedUser-agent przypisywany indeksowi Brave Search. Brave says its crawler does not identify itself with a separate user-agent and will not fetch a page that Googlebot cannot fetch – so robots.txt controls it through the rules for Googlebot. Yet 6 requests signed `Bravebot` arrived in March and April 2026 from addresses with a confirmed reverse DNS of `*.searchbot.brave.app`. The other 80 came from hosting providers and Google Cloud customer machines, without confirmation, so the name alone is not worth trusting.no separate token; Brave follows the rules for GooglebotPer Googlebot rules86 of which 6 verified GoogleOther-Image GoogleOther-VideoGoogleGoogleOther variants for public images and videoGoogleOther-Image GoogleOther-Video GoogleOtherYes44 all verified Yahoo! SlurpYahooYahoo search and partner content for Yahoo News, Finance and SportsSlurpYes13 Storebot-GoogleGoogleGoogle Shopping. No request under this name in my logs came from the operator’s addresses.Storebot-GoogleYes12 0 verified SogouSogouChinese search engineSogouYes9 MojeekBotMojeekIndependent search engine with its own indexMojeekBotYes; does not support Crawl-delay4 Googlebot-VideoGoogleVideo indexingGooglebot-Video GooglebotYes2 of which 1 verified BingVideoPreviewMicrosoftVideo previews in BingBingVideoPreviewYes0 YepBotAhrefs / YepThe Yep search engine, currently only for IndexNow requests. The index itself is built by AhrefsBotYepBotYes, also honors Crawl-delay0 PinterestbotPinterestIndexes content and products for Pins, updates prices and dead linksPinterestbotYes; honors Crawl-delay up to a value of 1, treats higher values as 10 facebookexternalhitMetaPreview of links shared across Meta servicesfacebookexternalhitSometimes ignored6,694 Slackbot-LinkExpanding Slack-ImgProxy SlackbotSlackLink previews, image fetching and Slack’s remaining requestsnoneNo – Slack states that it does not process robots.txt907 SkypeUriPreviewMicrosoftLink previews in SkypenoneNo clear documentation439 WhatsAppMetaPreview of links sent in WhatsAppnoneNo clear documentation426 LinkedInBotLinkedInPreview of links shared on LinkedInLinkedInBotNo clear documentation411 TwitterbotX (formerly Twitter)Link previews in X cardsTwitterbotNo clear documentation56 DiscordbotDiscordLink previews on DiscordDiscordbotNo clear documentation41 Google MessagesGooglePreview of links sent in Google Messagesnone, user fetcherUsually not15 MicrosoftPreviewMicrosoftPage previews in Microsoft productsMicrosoftPreviewYes0 AhrefsSiteAuditAhrefsTechnical audit ordered by the site ownerAhrefsSiteAuditYes by default; the site owner can switch this off in the audit settings167,450 all verified Screaming Frog SEO Spiderwhoever runs the programManual crawl, usually your own or your agency’sScreaming Frog SEO SpiderDepends on the settings16,013 AhrefsBotAhrefsIndex of links and content for Ahrefs and the Yep search engineAhrefsBotYes6,671 all verified BarkrowlerBabbarLink indexBarkrowlerYes6,615 SemrushBotSemrushIndex of links and visibility dataSemrushBotYes5,790 serpstatbotSerpstatBacklink indexserpstatbotYes3,406 MJ12botMajesticLink indexMJ12botYes2,076 DataForSeoBotDataForSEOIndex of SEO data sold through an APIDataForSeoBotYes1,450 SERankingBacklinksBotSE RankingBacklink indexSERankingBacklinksBotYes1,091 SEBot-WASE RankingSE Ranking’s technical audit crawler (Website Audit). A user of the tool can change the user-agent in the audit settings.SEBot-WAYes by default1,082 SiteAuditBotSemrushTechnical audit on requestSiteAuditBotYes962 AdsBot-GoogleGoogleQuality assessment of ad landing pagesAdsBot-GoogleYes, but ignores User-agent: \*228 of which 227 verified Mediapartners-GoogleGoogleCrawler for Google’s advertising products, including AdSense and Google Ad Manager. No request under this name in my logs came from the operator’s addresses.Mediapartners-GoogleYes, but ignores User-agent: \*103 0 verified AdsBot-Google-MobileGoogleQuality assessment of mobile ad landing pagesAdsBot-Google-MobileYes, but ignores User-agent: \*39 all verified meta-externaladsMetaImproving Meta’s ads and other business productsmeta-externaladsYes0 AdIdxBotMicrosoftQuality control of landing pages for Microsoft AdvertisingadidxbotYes0 DotBotMozCrawler building the Moz Link Explorer link indexDotBotNo current clear documentation from the operator0 rogerbotMozAudits of sites added to Moz Pro campaigns by their ownersrogerbotNo current clear documentation from the operator0 OAI-AdsBotOpenAIVerification of ad landing pages in ChatGPT; the data collected is not used for model training. OpenAI’s developer documentation describes OAI-AdsBot with its full user-agent and IP address list, but as robots.txt tags it still lists only `GPTBot` and `OAI-SearchBot`. The help article for advertisers says outright that OpenAI’s crawlers honor robots.txt and that a `Disallow` stops fetching immediately. Blocking this bot prevents your ads in ChatGPT from being verified.OAI-AdsBotYes0 UptimeRobotUptimeRobotSite availability monitoring, usually set up by the ownerUptimeRobotNot applicable69,895 AwarioBotAwarioMonitoring wzmianek o markach. In robots.txt, Awario gives the tokens AwarioSmartBot and AwarioRssBot; the request count refers to the AwarioBot user-agent.AwarioSmartBot AwarioRssBotYes1,797 NetcraftSurveyAgent CheckMarkNetwork StormIntelCrawler OI-Crawlersecurity research and domain inventoryScanning the public surface of websites, domain statisticsseparate tokensNo clear documentation71 Google-SafetyGoogleDetecting malware and abuse behind publicly shared linksnoneNo – Google states that it ignores robots.txt61 all verified Buck (Hypefactors) RecordedFuturemedia monitoringTracking mentions of brands and topicsseparate tokensNo clear documentation47 img2dataset pdf2datasetvarious operators, including NVIDIATools for building image and document datasets, including for model training. In my logs under the names ImageBot, nvidiabot and cc-download, almost exclusively images and PDFs.noneNo clear documentation3,591 archive.org\_botInternet ArchiveArchiving pages in the Wayback Machinearchive.org\_botNo current clear documentation from the operator1,504 OWLerOpenWebSearch.euCrawler europejskiego projektu otwartego indeksu OpenWebSearch.eu. In my logs from the network of the Leibniz-Rechenzentrum computing center.OWLerNo clear documentation211 ClueWeb-CrawlerCarnegie Mellon UniversityBuilding the public ClueWeb corpus for search research. I found no current documentation for this crawler; the project’s earlier crawlers honored robots.txt.ClueWeb-CrawlerNo clear documentation174 Aranea Web-Crawled CorporaSlovak Academy of SciencesBuilding public language corporanoneNo clear documentation79 WP Rocket Preload, Pre-fetchWP RocketCache plugin: Preload builds the cache, Pre-fetch fetches links in advance. Your own traffic. On my site this is the largest entry in the whole table.noneNot applicable418,899 WP-Rocket-SaaSWP RocketA WP Rocket service that fetches pages and files from the operator’s servers for optimization. Also your own traffic if you use WP RocketnoneNot applicable183,835 curl Wget python-requests Scrapy Go-http-client axios and relatedany scriptNot bots, just libraries for fetching pages. Behind each one stands a person or someone else’s programnoneNot applicable – depends on the script, the library enforces nothing25,903 Chrome-LighthouseGoogleLighthouse and PageSpeed Insights tests, run manually by someonenoneNot applicable14,508 Google-InspectionToolGoogleSearch Console testing toolsGoogle-InspectionTool GooglebotYes7,264 of which 7,262 verified Inoreader FeedlyRSS readersFetching an RSS feed on behalf of a subscriberseparate tokensUsually yes2,437 RankMath Link CheckerRank MathSEO plugin checking links. If you run Rank Math, this is your own trafficnoneNot applicable2,148 Chrome Privacy Preserving Prefetch ProxyGooglePage prefetching for Chrome users. Looks like a bot, but a browser stands behind itnoneNot applicable2,064 all verified BingSapphireMicrosoft (Bing app)Browser built into the Bing app on Android. A person stands behind this traffic, not a crawler. It is not on Microsoft’s list of crawlers.noneNot applicable76 Google-Site-VerificationGoogleChecking site ownership verification in Search Consolenone, user fetcherUsually not18 all verified datasets library (Hugging Face)any userA Python library for building datasets. Someone was pulling content into a set of their ownnoneNot applicable – depends on the script14 APIs-GoogleGoogleDelivering notifications from Google APIs, e.g. PubSubHubbubAPIs-GoogleYes, but ignores User-agent: \*0 Jetpack by WordPress.comimpersonation – home networks in IndiaLogin attacks through /xmlrpc.php under the name of the Jetpack plugin. If you do not use Jetpack, every such request is suspiciousnoneNot applicable – block it on the server12,405 br-crawler crawler\_eb\_germany Hermes-SVF-static-crawler VelenPublicWebCrawler and othersnot establishedNo documentation and no way to establish the operator. Judge them by behaviornoneUnknown4,608 ExabotExalead? – to verifyHistorically the Exalead search engine. In my logs 97 of 98 requests are a bare `Exabot` user-agent from hosting providers, fetching only robots.txt.ExabotNo clear documentation98 cohere-ai CohereBotCohere? – to verifyNames attributed to Cohere. Cohere states that it currently does not use bots to collect content for training, and gives CohereBot only as an example of a future rule. In my logs all requests come from a scanner.cohere-ai CohereBotNot applicable – Cohere declares no active bot60 none from Cohere 2ip bot2ip.ioService for checking IP addresses and websites2ip botNo clear documentation51 EmailCrawlerunknownHarvesting email addresses. The one entry where blocking needs no thoughtEmailCrawlerUnknown; if you want certainty, block it on the server21 **If you block NotebookLM, block both names.** On 16 July 2026 Google changed the user-agent name in its documentation from `Google-NotebookLM` to `Google-GeminiNotebook` and announced support for the old name until August 2026. In practice, up to 6 October 2026 all 4,064 requests from this fetcher reached my site under the old name and from Google addresses, while the new one did not appear even once. A rule in `.htaccess` or in your WAF therefore has to cover both: the old name, because it still works, and the new one, because according to the documentation it can take over at any moment. How this table was built **I checked purposes and tokens in the operators’ documentation everywhere such documentation exists.** Entries with no official source are described on the basis of the full user-agent, the address given in its string and its behavior in the logs – a missing confirmation is clearly marked. The frequency data comes from over 2.6 million requests in the podrez.pl logs for the period from 2 February to 6 October 2026. I count each request in a single row – the one whose name appears earliest in the user-agent – so, for example, Googlebot-Image is not counted a second time as Googlebot. Some entries group several related user-agents from one operator, so there are more names than rows. The table counts names, not bots. Scanners impersonate some AI bots, which is why for some entries I give how many requests passed IP verification. Compiled by Ewelina Podrez-Siama, working in SEO since 2009. Last updated: 24 October 2026. ### Sources: bot operator documentationForty-four documents in which I checked tokens, purposes and rules. You can check each one yourself. **OpenAI** [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) – GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot [Advertiser Guidance for Allowing OpenAI Web Crawlers](https://help.openai.com/en/articles/20001243-advertiser-guidance-for-allowing-openai-web-crawlers) – the OAI-AdsBot token and rule compliance **Anthropic** [Does Anthropic crawl data from the web…](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) – ClaudeBot, Claude-User, Claude-SearchBot **Perplexity** [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) – PerplexityBot, Perplexity-User, IP address lists **Google** [Google’s common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) – Googlebot, GoogleOther, Google-Extended, Google-CloudVertexBot [Google special-case crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers) – AdsBot, Mediapartners, APIs-Google, Google-Safety [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) – Gemini Notebook, Google-Agent, GoogleProducer [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) – three classes of bots, three JSON files, reverse DNS [Crawler documentation changelog](https://developers.google.com/crawling/docs/changelog) – change dates, including 17.09.2026, 16.07.2026 and 20.03.2026 [AI Features and Your Website](https://developers.google.com/search/docs/appearance/ai-features) – preview directives with regard to AI Overviews and AI Mode **Apple** [About Applebot](https://support.apple.com/en-us/119829) – Applebot, Applebot-Extended, iTMS, IP address list, inheriting rules from Googlebot **Meta** [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) – meta-externalagent, meta-webindexer, meta-externalfetcher, facebookexternalhit; the last two may skip robots.txt **Amazon** [About Amazonbot](https://developer.amazon.com/amazonbot) – Amazonbot, Amzn-SearchBot, Amzn-User, three IP address lists **Mistral AI** [Mistral crawlers](https://docs.mistral.ai/robots) – MistralAI-Training, MistralAI-Index, MistralAI-User **Microsoft** [Overview of Bing crawlers](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) – bingbot, adidxbot, BingPreview, MicrosoftPreview, BingVideoPreview **Other search engines** [DuckDuckGo – DuckDuckBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot) and [DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) – rule compliance, IP address lists [YouBot](https://docs.you.com/youbot) – robots.txt, Crawl-delay, reverse DNS [Yandex – robots in server logs](https://www.yandex.com/support/webmaster/en/robot-workings/check-yandex-robots) – which robots follow the general rules, reverse DNS [Seznam – Crawling control](https://o-seznam.cz/napoveda/vyhledavani/en/crawling-control/) [Petal Search – PetalBot](https://webmaster.petalsearch.com/site/petalbot) [Baidu – Baiduspider](https://ziyuan.baidu.com/college/articleinfo?id=1192) – statement that it follows the robots protocol [Sogou – spider and robots.txt](https://help.sogou.com/spider.html) [Yahoo – Why is Slurp crawling my page?](https://help.yahoo.com/kb/SLN22600.html) [Mojeek – MojeekBot](https://www.mojeek.com/bot.html) **SEO tools** [Semrush – SemrushBot and SiteAuditBot](https://www.semrush.com/bot/) [Majestic – MJ12bot](https://mj12bot.com/) [Serpstat – serpstatbot](https://serpstatbot.com/) [DataForSEO – DataForSeoBot](https://dataforseo.com/dataforseo-bot) [SE Ranking – SERankingBacklinksBot](https://help.seranking.com/hc/en-us/articles/23020353159964-SERankingBacklinksBot-Crawler) [Babbar – Barkrowler](https://babbar.tech/crawler) [SE Ranking – SEBot-WA](https://help.seranking.com/hc/en-us/articles/16335355751196-Verifying-SE-Ranking-Crawler-FAQ) – the Website Audit crawler, reverse DNS [Awario Crawlers](https://awario.com/bots/) – AwarioSmartBot and AwarioRssBot **Other operators** [Common Crawl – CCBot](https://commoncrawl.org/ccbot) [Parallel – ShapBot and Shap-User](https://parallel.ai/parallel-web-systems-bots) – version of 11.08.2026 [Cohere Web Crawlers](https://docs.cohere.com/v2/docs/cohere-web-crawlers) – statement that Cohere currently does not use bots to collect content for training [Qwant – QwantBot and robots.txt](https://help.qwant.com/hc/en-us/articles/51146433634833-The-Qwant-bot-QwantBot-and-the-robots-txt-file) – user-agent variants, IP address list, Crawl-delay [Ahrefs Bots](https://ahrefs.com/robot) – AhrefsBot, AhrefsSiteAudit, IP address lists [Slack Robots](https://api.slack.com/robots) – three agents and a statement about not honoring the rules [Yep – YepBot](https://yep.com/yepbot/) [Pinterest – Pinterestbot](https://help.pinterest.com/en/business/article/pinterestbot) [Diffbot – Does Diffbot respect robots.txt?](https://www.diffbot.com/docs/crawl/faq/robots-txt) [Brave Search Crawler](https://search.brave.com/help/brave-search-crawler) **Context** [Cloudflare – Your site, your rules: new AI traffic options](https://blog.cloudflare.com/content-independence-day-ai-options/) – the Search, Agent and Training categories, default settings from 15.09.2026 and blocking of multi-purpose crawlers under a Training block All links checked on 7 and 8 October 2026. Entries marked in the table as “no clear documentation” have no counterpart on this list – I describe them solely on the basis of the user-agent and behavior in the logs. **Verification.** For some entries I give, under the number, how many requests passed verification: an IP address from the operator’s official list, or a reverse DNS in its domain confirmed by a forward lookup. I downloaded the lists on 7 October 2026, so older requests from addresses the operator no longer uses may come out as unverified. That is why I don’t mark ChatGPT-User and ShapBot this way – their unverified requests come mainly from February–July, and since August 79% and 100% of them verify, respectively. **The “honors robots.txt” column.** It is based on operator documentation, not on my own tests. “Stated by operator” means the operator says so, though reports of deviations circulate. “No clear documentation” means exactly what it says – I found no reliable source and I am not guessing. Want to know which AI bots visit your site? Server logs show it; robots.txt doesn’t. Drop a log into the free tool and see which bots came and whether their IP addresses are on the operators’ lists. Full log analysis is part of my AI search visibility work. [Check AI bots in your logs](https://podrez.pl/en/ai-bots-in-logs/)[AI search visibility](https://podrez.pl/en/ai-search-visibility/)[Send a project brief](https://podrez.pl/en/contact/) robots.txt Why doesn’t robots.txt block every AI bot? ------------------------------------------ **An entry in `robots.txt` does not work the same way on every bot.** Crawlers that move around the web on their own initiative usually follow it. With fetchers triggered by a user things get complicated, because here every operator has a policy of its own. - ### Automated crawlers They move around the web on the operator’s initiative: GPTBot, ClaudeBot, PerplexityBot, Googlebot. They honor `robots.txt` and this file was created with them in mind. - ### User-triggered fetchers They fetch a page because a specific user action required it. Operators differ here. Anthropic and Mistral provide separate tokens (`Claude-User`, `MistralAI-User`) and declare that they honor them. Google, OpenAI, Perplexity, Amazon and Meta note that the rules may not apply, because the request is initiated by a user. - ### Control tokens without a bot `Google-Extended` and `Applebot-Extended` send no requests at all. You put them in `robots.txt`, but you will never see them in your logs. Google-Extended covers training of Gemini models *and* grounding, meaning content passed to the model at the moment of answering. Applebot-Extended covers training of Apple’s models. The practical conclusion: before you block a fetcher triggered by a user, check the policy of that particular operator. A `Disallow` for Claude-User or MistralAI-User genuinely closes the door, because Anthropic and Mistral declare that they honor those tokens. With Google, OpenAI, Perplexity, Amazon and Meta the same entry is a request, so if you need certainty, add a rule at the server or WAF level. I described how to check this on your own site in the article on [server log analysis](https://podrez.pl/en/server-log-analysis-ai-visibility/). Impersonation Suspicious user-agents: impossible names and names to verify ------------------------------------------------------------ Since June 2026 a scanner has been active in my logs that signs itself with the names of AI bots and looks for files with passwords and keys: `.env`, `.aws/credentials`, Firebase configuration. It comes mainly from Google Cloud customer machines and cheap hosting providers, and it rotates the bot names. These names are worth knowing, because they look credible and slip through a filter easily. Some of the names cannot come from the operator, because its own documentation says those tokens do not crawl at all. Others are real bot names, but not a single request under them came from the operator’s addresses. Still others appear in no documentation whatsoever. The same scanner also signs itself with names that have their own rows in the table – ChatGPT-User, Claude-User, GPTBot or Amazonbot – and inflates their counts. Name in the logStatusWhyWhat it was looking for on my site`Google-Extended`**Impossible**A control token in `robots.txt`; Google does not send requests with it`/.aws/credentials`, `/.env.test``Applebot-Extended`**Impossible**Apple states outright that this token does not crawl pages`/privatekey.key`, `/secrets.json``Claude-SearchBot`**Impersonation**A real Anthropic bot name, but not a single request came from addresses on Anthropic’s listfiles with passwords and keys`Amzn-SearchBot`**Impersonation**A real Amazon bot name, but not a single request came from Amazon addressesfiles with passwords and keys`anthropic-ai`ScannerA name outside Anthropic’s current documentation`/__/firebase/init.json`, `/env.js``Claude-Web`ScannerA name outside Anthropic’s current documentation`/secrets.json`, `/.env.old``cohere-ai`, `CohereBot`ScannerCohere states that it currently does not use bots to collect content for training`/.aws/credentials`, `/api/v1/env``GrokBot`, `Kimi-SearchBot`, `DeepSeekBot` and othersScannerNames outside the operators’ documentation, the same traffic pattern`.env`, `credentials`, Firebase configuration`BingIndexCrawler`To verifyMicrosoft’s network, but without a `search.msn.com` reverse DNS; Bing crawls as `bingbot`images from `/wp-content/uploads/` podrez.pl logs, 2 Feb – 6 Oct 2026. Number of occurrences: `Amzn-SearchBot` 244, `Claude-SearchBot` 213, `Google-Extended` 176, `cohere-ai` and `CohereBot` 60, `anthropic-ai` 28, `Claude-Web` 20, `BingIndexCrawler` 10, `Applebot-Extended` 5. The other fake AI names – `GrokBot`, `xAI-Grok`, `xAI-SearchBot`, `Kimi-SearchBot`, `MoonshotBot`, `Qwenbot`, `DeepSeekBot`, `ChatGLM-Spider`, `Hunyuan`, `YiBot` and `MistralBot` – between 20 and 38 occurrences each. The pattern is clear. Suspicious requests reach for: - tokens that are not user-agents, - names absent from documentation, - the names of real bots, counting on every filter to let those through. A name that does not match the documentation need not mean bad intent, though – `BingIndexCrawler` came from Microsoft’s network and fetched only images – which is why the IP address decides, not the name alone. The problem is not limited to my site either: Common Crawl explicitly warns in its documentation that it knows of crawlers falsely claiming to be CCBot, and recommends verifying the user-agent. Verification How to check whether a bot is real: verification by IP address -------------------------------------------------------------- A user-agent can be faked in ten seconds, so the name alone proves nothing. Certainty comes only from matching the name with the IP address and the operator’s official ranges, and where possible with a reverse and forward DNS check. Google is also testing cryptographic authentication of bots, that is Web Bot Auth. OperatorSource of addressesOpenAI`openai.com/gptbot.json`, `openai.com/searchbot.json`, `openai.com/chatgpt-user.json`, `openai.com/adsbot.json`Anthropic`claude.com/crawling/bots.json`Perplexity`perplexity.com/perplexitybot.json`, `perplexity.com/perplexity-user.json`Google – common crawlers`common-crawlers.json`; reverse DNS ends in `googlebot.com`Google – special-case crawlers`special-crawlers.json`; reverse DNS ends in `google.com`Google – user fetchers and agentsseparate JSON files; reverse DNS ends in `google.com` or `gae.googleusercontent.com`Apple`search.developer.apple.com/applebot.json`; reverse DNS ends in `applebot.apple.com`Mistral`mistral.ai/mistralai-user-ips.json`, `mistral.ai/mistralai-index-ips.json`; for `MistralAI-Training` the list is not publishedAmazonthree separate lists: `/ip-addresses/`, `/searchbot-ip-addresses/`, `/live-ip-addresses/`Paralleldocs.parallel.ai/resources/shapbot.jsonMicrosoftthe Verify Bingbot tool in Bing Webmaster Tools; reverse DNS ends in search.msn.comCommon Crawl`index.commoncrawl.org/ccbot.json`; reverse DNS ends in `crawl.commoncrawl.org`Qwant`help.qwant.com/qwantbot.json`; reverse DNS ends in `qwant.com`Metaaddresses from autonomous system AS32934, retrievable with `whois -h whois.radb.net -- '-i origin AS32934'`; the list changes oftenBaidureverse DNS ends in `baidu.com` or `baidu.jp`; Baidu does not publish an address listDuckDuckGo`duckduckgo.com/duckassistbot.json` for DuckAssistBotYou.comreverse DNS ends in `search.you.com`; cryptographic signatures as wellYandexreverse DNS ends in `yandex.ru`, `yandex.net` or `yandex.com`; Yandex does not publish an address listSE Rankingreverse DNS ends in `seranking.com` (SERankingBacklinksBot) or `sr-srv.net` (SEBot-WA)Babbar (Barkrowler)reverse DNS ends in `babbar.eu`Ahrefs (also YepBot)ranges and single addresses via `api.ahrefs.com`; reverse DNS ends in `ahrefs.com` or `ahrefs.net`Pinterestreverse DNS ends in `pinterest.com` or `pinterestcrawler.com` ### How many of these are real bots? Verifying my logs The table at the top counts names in the logs, not bots. So for every bot whose addresses can be checked, I also show how many requests came from the operator’s official ranges or passed a reverse and forward DNS check. Same period as the table: 2 Feb – 6 Oct 2026, podrez.pl. BotRequestsVerifiedShareGPTBot10,2189,92597%OAI-SearchBot5,8915,60495%ChatGPT-User8,3963,24939%\*ClaudeBot6,7896,51896%Claude-User1,90296651%Claude-SearchBot21300%PerplexityBot3,1612,92593%Perplexity-User2065125%\*Amazonbot8,1117,81996%Applebot4,5634,40797%Googlebot13,59511,24183%bingbot10,0799,80997% \* Since August 2026: ChatGPT-User 79%, Perplexity-User 0%. OpenAI only publishes its current address list, so older requests from the Microsoft Azure network do not match it. To check that this is not a quirk of one site, I compared the week of 17–23 October 2026 in the logs of nine sites I look after. The picture is the same, and for some names even starker: - **Perplexity-User**: 169 requests, not a single one from Perplexity’s addresses, - **DuckAssistBot**: 207 requests, 23 genuine, - **PerplexityBot**: every third request fake, - **ChatGPT-User**: every sixth fake, - **OAI-SearchBot** and **GPTBot**: every tenth and every fourteenth fake. The moral? Before you boast that “ChatGPT visits you 200 times a week”, check the addresses. A report that counts AI bots by name alone can show a scanner looking for a `.env` file as your most loyal Perplexity reader. Blocking How to block AI bots in robots.txt? Ready-made examples. -------------------------------------------------------- **Before you block anything: in most cases I do not recommend it.** A block cuts you off from generative answers, and so from a channel where your brand can be mentioned and cited. For most companies, experts and shops that is a loss, not a gain. The exceptions are real and concern publishers, portals and media above all – those who make a living from selling content or from traffic on their own site. There, a block can be a justified business decision – but still a decision, not a default setting. So treat the configurations below as a starting point for an informed choice, not as a recommendation. The first variant limits the main crawlers used for model training. The second blocks separately identified AI bots and fetchers, leaving access open to Googlebot, bingbot and Applebot. Both are a starting point, not a universal configuration for every site. Keep in mind that a block does not work retroactively. A model that has already been trained will not forget what it learned up to its [knowledge cutoff](https://podrez.pl/en/ai-knowledge-cutoff-dates/). ### Variant 1 · No training *I protect my content* I let in everything that can cite me and cut off content collection for model training. ``` User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Bytespider User-agent: Meta-ExternalAgent User-agent: MistralAI-Training User-agent: cohere-ai User-agent: CohereBot User-agent: Amazonbot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: DuckAssistBot User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: Amzn-SearchBot User-agent: ShapBot Allow: / ``` The cost: with multi-purpose crawlers such as Amazonbot or CCBot, a block also switches off uses other than training. ### Variant 2 · Separate AI bots *Google and Apple stay* I block separate training crawlers, AI indexes and user fetchers, but leave classic search engines open. This does not remove the site from AI Overviews, AI Mode or Apple’s generative answers. ``` # search engines stay User-agent: Googlebot User-agent: bingbot User-agent: Applebot Allow: / # model training User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Bytespider User-agent: Meta-ExternalAgent User-agent: MistralAI-Training User-agent: cohere-ai User-agent: CohereBot User-agent: Amazonbot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / # indexes behind generative answers User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: MistralAI-Index User-agent: Meta-WebIndexer User-agent: DuckAssistBot User-agent: Amzn-SearchBot User-agent: LinkupBot User-agent: ShapBot User-agent: YouBot User-agent: Diffbot Disallow: / # fetches on user request User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: MistralAI-User User-agent: Meta-ExternalFetcher User-agent: Amzn-User Disallow: / ``` The effect: you limit access for systems that have bots of their own – ChatGPT, Claude, Perplexity, Mistral. This is not a full exit from generative answers: AI Overviews and AI Mode are fed by Googlebot, Copilot by bingbot, and Applebot can be a source of context for Apple’s answers. The second variant has a hole and it is better to know about it right away. The last group are fetchers triggered by a user, and some of them may ignore `robots.txt`. Gemini Notebook and Google-Agent do not even have a token of their own. If the block is to be airtight, it has to be added at the server level. An example for Apache and the `.htaccess` file: ``` RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (Google-GeminiNotebook|Google-NotebookLM|Google-Agent|GoogleProducer|ChatGPT-User|Perplexity-User|Amzn-User|Meta-ExternalFetcher|Shap-User) [NC] RewriteRule ^ - [F,L] ``` One caveat that is easy to forget: `robots.txt` is not a tool for protecting data. The file is public, the directives are not access control, and a blocked URL can still end up in the index on the strength of external links. Genuinely private sections are secured with server-side authentication. **robots.txt is not the only layer.** - If a more specific matching group exists for a bot, it takes precedence over `User-agent: *`, and the wildcard applies only when nothing more specific matches. - Fetchers triggered by a user are a separate case: some honor dedicated tokens, some may skip them and need a rule at the server or WAF level. - If your site sits behind Cloudflare, check its AI bot settings too: since 15 September 2026 a block on the Training category – in the new settings or through the old “Block AI bots” toggle – also covers Googlebot, bingbot and Applebot, because these bots collect data both for search and for training. Your robots.txt can be set up perfectly and Googlebot will still be refused at the Cloudflare level. Your brand is missing from AI answers? Send a short brief with a few questions your customers ask. I’ll review it and propose a defined scope. [Send a project brief](https://podrez.pl/en/contact/)[AI search visibility](https://podrez.pl/en/ai-search-visibility/) Google How to limit AI Overviews? On nosnippet, data-nosnippet and max-snippet ----------------------------------------------------------------------- `robots.txt` tells compliant bots whether they *should* fetch a page. Meta directives, by contrast, control indexing and which fragments can reach the results and the AI features. These are two different layers, and with AI Overviews the second one is often the only one you have left. The key point **There is no separate opt-out from AI Overviews and AI Mode that would leave your ordinary presence in Google untouched.** They are fed by the same Googlebot and the same index, and `Google-Extended` covers only Gemini training and grounding, so blocking it changes nothing here. You can, however, limit how much content reaches those features. Google lists four directives: `nosnippet`, `data-nosnippet`, `max-snippet` and `noindex`. Each of them works on ordinary results and on AI features at the same time – you cannot separate one from the other. DirectiveWhat it doesWhat it costs you`nosnippet`Blocks the display of a page fragment in every form of results, including AI Overviews and AI ModeThe ordinary description under the title also disappears, along with the chance of a direct answer`data-nosnippet`The same, but only for the marked HTML fragmentThe rest of the page works normally – the most precise tool in this set`max-snippet:N`Limits the fragment length to N characters and with it the amount of content passed to the modelAt a low value the effect comes close to `nosnippet``noindex`Removes the page from every form of Google results, including AI Overviews and AI ModeThe page also disappears from ordinary search results`noarchive`At Amazon it means “do not use this page to train models”Works only with operators that honor this directive Apple honors `nosnippet` – also in the `X-Robots-Tag: applebot: nosnippet` header – as an opt-out from the use of content as additional context in generative answers. Marking a page as `isAccessibleForFree: false` in structured data has the same effect, and the page can still appear in Apple’s search results. Amazon declares that it honors `noarchive`, `noindex` and `none`, but does not support `crawl-delay`. **The most precise tool is `data-nosnippet`.** It excludes only the marked fragment of a page from previews and from direct use by AI features. Use it deliberately: the very paragraphs that would most easily replace a visit to your site – definitions, step lists, tables – are usually the ones that get a site cited in the first place. Data from my logs What AI bots take from your site: content or files? --------------------------------------------------- The number of visits alone says little. What is more interesting is how many of them land on a page with content, and how many on CSS files, images and API endpoints. BotSuccessful requestsShare of hits on contentPerplexity-User6191.8%CCBot56981.5%ChatGPT-User7,80574.2%Claude-User98463.7%PerplexityBot2,74962.1%OAI-SearchBot4,14533.5%GPTBot5,84330.2%ClaudeBot4,28921.8%Google-GeminiNotebook3,68619.9%Applebot4,05117.7%Amazonbot5,41515.1% podrez.pl logs, 2 Feb – 6 Oct 2026; a successful request is a response with status 200. Content means any URL other than the home page, `robots.txt`, sitemaps, the `/wp-content/`, `/wp-includes/` and `/wp-json/` directories and `.php` files. Perplexity-User, ChatGPT-User and Claude-User land mostly straight on content, but the class of bot alone does not determine its behavior: CCBot has a high share of hits on content, and Gemini Notebook a low one. The definition of content changes the conclusions here more than you might think. One in four successful ChatGPT-User hits is the home page – if I counted it as content, its share would rise to 98.6%. That looks more like checking who the author of the site is than reading a particular article. The second observation concerns the last two months: since 6 August GPTBot fetched content in only 13% of its successful hits, and on average visited the site almost four times less often than before. Tools and lists More tools and lists -------------------- - ![What bot is this? List of AI crawlers, search bots and user-agents](https://podrez.pl/wp-content/uploads/2026/11/obserwatorium-encji-hero-kolaz-768x432.webp "What bot is this? List of AI crawlers, search bots and user-agents")Free tool ### [Entity Observatory](https://podrez.pl/en/entity-observatory/) See a page’s structured data as a relation map and add the Wikidata record next to it. - ![What bot is this? List of AI crawlers, search bots and user-agents](https://podrez.pl/wp-content/uploads/2026/05/knowledge-graph-768x512.jpg "What bot is this? List of AI crawlers, search bots and user-agents")Free tool ### [Are you in Google’s Knowledge Graph?](https://podrez.pl/en/knowledge-graph-checker/) Enter a name and check whether Google knows you as an entity and whether it matches Wikidata. - ![What bot is this? List of AI crawlers, search bots and user-agents](https://podrez.pl/wp-content/uploads/2026/08/analiza-logow-ai-768x403.jpg "What bot is this? List of AI crawlers, search bots and user-agents")Free tool ### [AI bots in your logs](https://podrez.pl/en/ai-bots-in-logs/) Drop a server log and see which AI bots visit, which come from confirmed IP addresses and which ones your server turns away. Your file stays on your computer. - ![What bot is this? List of AI crawlers, search bots and user-agents](https://podrez.pl/wp-content/uploads/2026/05/common-crawl-hero-768x512.jpg "What bot is this? List of AI crawlers, search bots and user-agents")Free tool ### [Is your site in Common Crawl?](https://podrez.pl/en/common-crawl-checker/) Enter a domain to see how many pages CCBot saved in the last 6 crawls and whether anything blocks it. - ![What bot is this? List of AI crawlers, search bots and user-agents](https://podrez.pl/wp-content/uploads/2026/08/ai-cutoff-768x403.jpg "What bot is this? List of AI crawlers, search bots and user-agents")Table ### [AI knowledge cutoff dates](https://podrez.pl/en/ai-knowledge-cutoff-dates/) How current is the data in ChatGPT, Claude, Gemini and Grok? Cutoff dates checked against provider documentation. Bots don’t come to your site? Maybe they can’t! Check it in [High Five, AI!](https://high5ai.app/) by Maciej Chmurkowski. Questions Frequently asked questions -------------------------- ### How do I check which bot visited my site? The bot name sits in the user-agent field of your server’s access log, usually in `access.log` or `domain.com.log` in the `logs` directory. Copy the name and search for it in the table above. If you want to be sure the bot is genuine, compare its IP address with the ranges published by the operator. The quickest way is my [AI bots in logs](https://podrez.pl/en/ai-bots-in-logs/) tool: you load the log in your browser, the file never leaves your computer, and the tool recognises the bots and checks their IP addresses. ### Does blocking GPTBot remove me from ChatGPT results? No. GPTBot is responsible for collecting content for model training, while presence in ChatGPT search results is handled by OAI-SearchBot. These are two independent tokens and you can set them differently: block training, leave search open. Anthropic and Mistral apply an analogous three-way split. Amazon also divides its traffic across three agents, but its Amazonbot is a multi-purpose crawler that only may be used for training. Perplexity, by contrast, separates just the search index from fetches on user request and has no documented training crawler. ### Why don’t I see Google-Extended in my logs? Because `Google-Extended` is not a bot. It is a control token in `robots.txt` with which you decide about the use of your content for training Gemini models and for grounding. The fetching itself is done by Google’s ordinary robots. If you do see this name in your log, you are looking at an impersonator. ### How do I opt out of AI Overviews? Through `robots.txt` you can’t. AI Overviews and AI Mode use Google’s ordinary index, built by Googlebot, and the `Google-Extended` token covers only the training of Gemini models and grounding. The only available tool is the preview directives: `nosnippet`, `data-nosnippet`, `max-snippet` and `noindex`. Each of them also limits your ordinary search result, so leaving AI Overviews means partly leaving Google. ### Will a robots.txt block stop every AI bot? No. Fetchers triggered by a user, such as Gemini Notebook or ChatGPT-User, may skip `robots.txt`, because the request is initiated by a person, not a machine. To stop them you need a rule at the server level, in your WAF configuration or in the `.htaccess` file. ### What changed with NotebookLM? On 16 July 2026 Google updated its documentation and changed the user-agent name from `Google-NotebookLM` to `Google-GeminiNotebook`, after the product itself was renamed Gemini Notebook. It announced support for the old name until August 2026. In my logs, up to 6 October all requests from this fetcher still arrived under the old name, while the new one did not appear even once, so blocking rules should cover both. **Last updated: 24 October 2026** I check bot tokens and purposes in the operators’ documentation (listed in the sources above; last full review on 7–8 October 2026, including Cloudflare). Entries marked as “no clear documentation” are described on the basis of observation. Log data comes from the podrez.pl access logs for 2 Feb – 6 Oct 2026, over 2.6 million requests, and I count each request in a single row. I verify IP addresses against operator lists downloaded on 7 October 2026. Page history and timeline of changes at the operators **History of this page** 08.10.2026 – log data extended to 6 October 2026, new entries (including ExaSearchBot, Gemini-Deep-Research, OWLer and the fake Jetpack), verified-request counts for some bots and an expanded section on bot impersonation. 06.08.2026 – first version. **Timeline of changes at the operators** 17.09.2026 – Google extended the `Mediapartners-Google` documentation to cover various advertising products, including AdSense and Google Ad Manager. 15.09.2026 – the date Cloudflare announced on 1 July for new default settings: on ad-supported pages of newly onboarded domains the Training and Agent categories are blocked, and crawlers that combine search with training – including Googlebot, bingbot and Applebot – fall under a Training block. August 2026 – the end of support for the old `Google-NotebookLM` name announced by Google; in my logs it still works (as of 6 October). 16.07.2026 – Google changed the NotebookLM user-agent name to `Google-GeminiNotebook`; the old name supported for a transition period. 01.07.2026 – Cloudflare introduced a split of bots into the Search, Agent and Training categories. 20.03.2026 – Google documented the `Google-Agent` user-agent. Next steps Not sure which bots your site lets in? -------------------------------------- The name in the log is only the beginning. Far more interesting is *what* that bot got in response – and whether it got anything at all. I described how to check that, and what I found on my own site, in the article on [server log analysis](https://podrez.pl/en/server-log-analysis-ai-visibility/). In my AI search visibility projects I go through server logs whenever they are available, because robots.txt alone won’t tell you what a bot actually received. Log analysis covering AI bots, search engine crawlers, SEO tools and suspicious traffic is part of my audit. If you would rather start with a conversation, book a short call. [Book a consultation](https://podrez.pl/en/contact/)