Skip to content
Contact
Paper collage: a bot directory with crawler icons, a spider, a magnifying glass and servers

Ewelina Podrez-Siama · bot profiles

ClaudeBot Anthropic’s training crawler: user-agent, IP addresses and robots.txt

ClaudeBot is Anthropic’s web crawler. It collects public pages that may be used to train Claude models, and it honors robots.txt.

Below you’ll find the user-agent strings that reach my server, Anthropic’s IP list, ready-made robots.txt rules and what ClaudeBot actually does on podrez.pl. I checked the facts against Anthropic’s documentation and the sources below on November 12, 2026.

1,551

requests claiming to be ClaudeBot
Oct 7 – Nov 11, 2026

96%

came from
Anthropic’s addresses

350

robots.txt checks,
about 10 a day

At a glance

ClaudeBot at a glance

QuestionAnswer
OperatorAnthropic
PurposeCollects web content that may be used to train Anthropic’s generative AI models. Blocking it tells Anthropic to leave your future content out of its training datasets.
robots.txt tokenClaudeBot
Without a ClaudeBot groupThe User-agent: * group applies. Since October 30, 2026, Anthropic has also recognized the shared training signal __usage_ai-training, but only as a training preference. A rule that names ClaudeBot always takes precedence.
Honors robots.txtYes, according to Anthropic. It also supports Crawl-delay.
IP addressesclaude.com/crawling/bots.json: 14 prefixes in the file dated October 20, 2026. The list covers Anthropic’s three crawlers and fetchers. Claude-Agent, the Anthropic-hosted browser, has its own list at claude.com/.well-known/sail.
Reverse DNSNot documented. Use the IP list.
Runs JavaScriptNot documented. A 2024 measurement found that it fetches scripts but doesn’t execute them.
Other Anthropic botsClaude-SearchBot (search index) and Claude-User (user-triggered: fetches a page when someone asks Claude). Each has its own token.
In my listAI: training (see the entry in “What bot is this?”)

User-agent

The ClaudeBot user-agent

Anthropic’s help page names the bot and its token but doesn’t give the full user-agent string. These two versions reached my site from Anthropic’s addresses between October 7 and November 11, 2026:

RequestsUser-agent string
1,397Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
99Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; ClaudeBot/1.0; +claudebot@anthropic.com)

Requests from Anthropic’s published addresses to podrez.pl, Oct 7 – Nov 11, 2026.

The second string poses as desktop Chrome. In my logs it fetched mostly images and scripts (73 of its 99 requests), so it looks like the part of the crawler that loads page resources. That’s my reading of the logs; Anthropic doesn’t document it.

To spot ClaudeBot in a log or a WAF rule, match ClaudeBot/ – the name followed by a slash and the version – ignoring case. A plain search for claudebot isn’t enough: Anthropic’s other bots, and the impersonators copying them, include claudebot@anthropic.com in their strings.

robots.txt

How to block or allow ClaudeBot in robots.txt

ClaudeBot reads robots.txt and follows the most specific group that names it (a group is a block of rules that starts with User-agent). Anthropic asks you to repeat the rule for every subdomain you want to cover.

Variant 1 · No training

ClaudeBot stays out

I keep Anthropic’s training crawler away from the whole site.

User-agent: ClaudeBot
Disallow: /
Claude-SearchBot and Claude-User have tokens of their own, so Claude can still find and quote your pages.

Variant 2 · Slower and selective

ClaudeBot stays in

I let ClaudeBot in, but ask it to slow down and skip one directory.

User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /private/
Allow: /
A ClaudeBot group replaces your User-agent: * rules for this bot, so copy over every Disallow line you still need.

Blocking ClaudeBot’s IP addresses is no substitute. Anthropic says an IP block may not work as a lasting opt-out, because it also stops the bot from reading your robots.txt.

Since October 30, 2026, Anthropic has also recognized the shared training signal __usage_ai-training, rolling it out gradually. If your robots.txt names ClaudeBot, that rule wins over the token. I explain the token in a separate article.

Check your file

Not sure what your robots.txt tells ClaudeBot? Enter your domain or paste the file into Can AI bots read your site?, my free robots.txt checker. It shows whether ClaudeBot may crawl your pages and which line decides it.

IP check

Is it really ClaudeBot? How to check the IP address

Anthropic publishes its bots’ addresses at claude.com/crawling/bots.json. On November 12, 2026, the file, dated October 20, 2026, listed 14 prefixes: one /22 block and 13 single addresses, all of them in Google Cloud. Anthropic documents no reverse DNS name for its bots, so the list is the only check.

Quick glossary: the user-agent is the name a bot gives itself, and anyone can type any name. A prefix is a block of IP addresses. Reverse DNS turns an address back into a host name. A WAF (web application firewall) is the filter in front of your site, such as Cloudflare, your host’s firewall or a security plugin.

  1. Take the IP address from the log line.
  2. Check whether it falls within one of the prefixes in bots.json. A match confirms Anthropic, but not which of its three bots sent the request: the user-agent tells you that.
  3. No match means an impersonator. If it’s causing trouble, block it on the server or in your WAF, not in robots.txt.

Between October 7 and November 11, 2026, 55 of the 1,551 requests claiming to be ClaudeBot (3.5%) came from outside Anthropic’s list, 44 of them from Google Cloud. All confirmed ClaudeBot traffic came from the 216.73.216.0/22 block. Google Cloud on its own proves nothing, since some of Anthropic’s listed addresses are in Google Cloud too: only an exact match with the list counts.

To check a whole log file at once, load it into Check AI bots in your logs, my free log checker. The file never leaves your browser, and the tool checks every address against Anthropic’s list.

JavaScript

Does ClaudeBot run JavaScript?

Probably not. Anthropic doesn’t say.

A widely cited measurement by Vercel and MERJ (published December 17, 2024) found that ClaudeBot fetches JavaScript files – 23.84% of its requests in their sample – but doesn’t execute them.

A log can show that a script was downloaded, not that it ran. Between October 7 and November 11, 2026, confirmed ClaudeBot requests included 26 JavaScript files and 6 stylesheets out of 1,496, mostly from the Chrome-like variant.

The practical rule: if you want ClaudeBot to see a piece of text – or not to see it – make sure it’s in the HTML your server sends. Text that only appears once JavaScript runs is probably invisible to it.

Data from my logs

What ClaudeBot does on my site

ClaudeBot visited podrez.pl on all 36 days of the period, 42 times a day on average, peaking at 209 requests on October 27, 2026. Earlier in 2026 it came less often: 26 confirmed requests a day between February 2 and October 6, 2026.

It’s a tidy crawler. Nearly half of its requests went to robots.txt (350) and sitemaps (348), so it checks the rules almost 10 times a day. Pages got 418 requests, the home page 166 and images 181. Only 28% of its successful requests landed on content.

Since October 16, 2026, my pages have also had Markdown versions, linked from the page head. ClaudeBot fetched 118 Markdown versions of my pages, more than any other bot here. Confirmed ClaudeBot requests never got a 403 from my server, either: every 403 sent to a “ClaudeBot” in this period went to an impersonator.

podrez.pl logsFeb 2 – Oct 6, 2026Oct 7 – Nov 11, 2026
Requests claiming to be ClaudeBot6,7891,551
Confirmed requests6,518 (96%)1,496 (96%)
Confirmed requests per day2642

“Confirmed” means the IP address is on Anthropic’s published list. “Content” means any page other than the home page, robots.txt, sitemaps, WordPress system paths (/wp-content/, /wp-includes/, /wp-json/) and .php files. The February⁠–⁠October figures are the ones in my list What bot is this?; October⁠–⁠November is a fresh count against the list as it stood on November 12, 2026. Identical log lines are counted once, and each request is assigned to a single bot: the one whose name appears first in the user-agent.

Questions

Frequently asked questions about ClaudeBot

Is ClaudeBot the same as Clawdbot?

No. Clawdbot was the original name of an open-source personal AI assistant. After Anthropic objected to the name, it became Moltbot and then, at the end of January 2026, OpenClaw. It runs on the user’s own computer and isn’t a web crawler. ClaudeBot is Anthropic’s crawler; the crawler and the assistant are separate things.

Does blocking ClaudeBot remove my site from Claude’s answers?

No. ClaudeBot only collects content for training. Claude-SearchBot indexes pages for search, and Claude-User fetches a page when someone asks Claude about it. Each has its own token, so you can block training and still let Claude find and quote you.

Should I block ClaudeBot?

Only if you don’t want your future content used to train Anthropic’s models. Blocking it doesn’t take you out of Claude’s answers – that’s Claude-SearchBot and Claude-User – but your pages stop feeding Anthropic’s training data, and with it what Claude knows about you without searching. On podrez.pl I let all AI bots in, ClaudeBot included.

Does a block remove content ClaudeBot has already collected?

No. According to Anthropic, restricting ClaudeBot tells it to leave your future content out of its training datasets. A model that has already been trained won’t forget what it learned.

Can I block ClaudeBot by IP address instead?

You can, but Anthropic warns that an IP block may not work as a lasting opt-out, because the bot can no longer read your robots.txt. Use robots.txt to opt out, and keep IP rules for impersonators.

How do I slow ClaudeBot down?

Add Crawl-delay to its group, for example Crawl-delay: 1. Anthropic supports this non-standard directive. On my site ClaudeBot averages 42 requests a day, so I don’t need it.

Sources

Sources and method

I read all the sources below on November 12, 2026. Figures about my site come from the podrez.pl server logs for February 2 – October 6 and October 7 – November 11, 2026.

  1. Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? – ClaudeBot, Claude-SearchBot, Claude-User, Crawl-delay, the training signal (page updated November 4, 2026).
  2. Anthropic: bot IP address list (bots.json) – 14 prefixes, file dated October 20, 2026.
  3. Anthropic Help Center: How do Anthropic-hosted browsers identify themselves when they browse the web on a user’s behalf? – Claude-Agent and its own address list at claude.com/.well-known/sail.
  4. IETF: Machine-Readable Training Preference Signal (individual Internet-Draft -00) – the __usage_ai-training token, published October 30, 2026, expires May 3, 2027.
  5. Vercel: The rise of the AI crawler – measurement with MERJ, published December 17, 2024: which AI crawlers render JavaScript.
  6. TechCrunch: OpenClaw’s AI assistants are now building their own social network – Clawdbot renamed Moltbot and then OpenClaw, January 30, 2026.

Compiled by Ewelina Podrez-Siama, working in SEO since 2009. Facts last checked: November 12, 2026.

Other bot profiles: GPTBot · OAI-SearchBot · Bytespider · Applebot · Amazonbot · Meta-ExternalAgent · CCBot · PerplexityBot. Or go back to What bot is this?, my searchable list of bots and user-agents.

Check your site

What does your site tell ClaudeBot?

Two free tools: one reads your robots.txt the way ClaudeBot would, the other shows whether ClaudeBot really visits you.

Send a project brief