Skip to content
Contact
Paper collage: a bot directory with crawler icons, a spider, a magnifying glass and servers

Ewelina Podrez-Siama · bot profiles

CCBot Common Crawl’s crawler: user-agent, IP addresses and robots.txt

CCBot is the crawler of Common Crawl, a non-profit that keeps an open repository of web crawl data. That archive is a common source of training data for language models.

Below you’ll find CCBot’s user-agent, its IP ranges and reverse DNS check, robots.txt rules and what CCBot – and the many fake CCBots – did on podrez.pl. I checked the facts against Common Crawl’s documentation and the sources below on November 12, 2026.

942

requests claiming to be
CCBot, Oct 7 – Nov 11, 2026

12%

came from
Common Crawl

86%

of its successful requests
landed on content

At a glance

CCBot at a glance

QuestionAnswer
OperatorCommon Crawl, a non-profit foundation
PurposeBuilds an open repository of web crawl data that anyone can access and analyze. It’s often used as a training dataset for language models.
robots.txt tokenCCBot
Without a CCBot groupThe User-agent: * group applies.
Honors robots.txtYes. CCBot checks robots.txt before fetching a page, obeys Crawl-delay and keeps checking the file for updates.
IP addressesindex.commoncrawl.org/ccbot.json and the FAQ: 4 IPv4 ranges and 1 IPv6 range, as of August 11, 2026.
Reverse DNSYes, over IPv4: names in crawl.commoncrawl.org. Not available over IPv6.
Runs JavaScriptNo. Common Crawl says JavaScript is not executed and cookies are not used.
Check your siteMy free tool Is your site in Common Crawl? shows how many of your pages CCBot saved.
In my listAI: training (see the entry in “What bot is this?”)

User-agent

The CCBot user-agent

Common Crawl documents a single current string, and confirmed CCBot requests on my site used exactly that:

RequestsUser-agent string
117CCBot/2.0 (https://commoncrawl.org/faq/)

Number: requests from Common Crawl’s published addresses to podrez.pl, Oct 7 – Nov 11, 2026.

That’s the problem: the string is short and easy to copy. Between October 7 and November 11, 2026, 825 requests with exactly the same string came from other networks: a Polish hosting company, eTOP (559), BlueVPS in Estonia (149), Google Cloud (70) and others (47). Common Crawl itself warns that some crawlers falsely identify themselves as CCBot.

Common Crawl says it may increment the version number in the future, so match CCBot, not the whole string.

robots.txt

How to block or allow CCBot in robots.txt

CCBot checks robots.txt before it fetches a page and follows the group that names it (a group is a block of rules that starts with User-agent).

Variant 1 · Out of Common Crawl

CCBot stays out

I keep my site out of Common Crawl’s future crawls.

User-agent: CCBot
Disallow: /
Common Crawl says it will keep checking whether your robots.txt has been updated.

Variant 2 · Slower

CCBot stays in

I let CCBot in but ask for at most one request every 2 seconds.

User-agent: CCBot
Crawl-delay: 2
Common Crawl obeys Crawl-delay. A CCBot group replaces your User-agent: * rules for this bot, so copy over every Disallow line you still need.

CCBot also slows down on its own when your server answers with 429 or 5xx, reads the sitemaps listed in robots.txt and honors nofollow on links.

To see whether CCBot has already saved your pages, try my free tool Is your site in Common Crawl?.

Check your file

Not sure what your robots.txt tells CCBot? Enter your domain or paste the file into Can AI bots read your site?, my free robots.txt checker. It shows whether CCBot may crawl your pages and which line decides it.

IP check

Is it really CCBot? How to check the IP address

Common Crawl runs CCBot on dedicated IP ranges with reverse DNS. The ranges are published at index.commoncrawl.org/ccbot.json and in the FAQ: on November 12, 2026, the FAQ listed 4 IPv4 ranges and 1 IPv6 range, as of August 11, 2026. The JSON file didn’t respond when I checked on November 12, 2026, so I used the FAQ copy, which matches my August download.

Quick glossary: the user-agent is the name a bot gives itself, and anyone can type any name. A prefix is a block of IP addresses. Reverse DNS turns an address back into a host name. A WAF (web application firewall) is the filter in front of your site, such as Cloudflare, your host’s firewall or a security plugin.

  1. Take the IP address from the log line.
  2. For IPv4, run host on it (in Terminal on a Mac or Linux): the name should end in crawl.commoncrawl.org, and running host on that name should return the same address. For IPv6, check the published range.
  3. If the check fails, it’s an impersonator. Block it on the server or in your WAF, not in robots.txt.

This check matters more for CCBot than for any other bot I profile. Between October 7 and November 11, 2026, only 117 of the 942 requests claiming to be CCBot (12%) were confirmed. Earlier in the year it was the other way around: 647 of 718 (90%) between February 2 and October 6, 2026.

To check a whole log file at once, load it into Check AI bots in your logs, my free log checker. The file never leaves your browser, and the tool checks every address against Common Crawl’s list.

JavaScript

Does CCBot run JavaScript?

No.

Common Crawl says JavaScript is not executed and cookies are not used. A widely cited measurement by Vercel and MERJ (published December 17, 2024) also found that CCBot doesn’t render pages.

My logs agree: confirmed CCBot requests included no scripts or stylesheets between October 7 and November 11, 2026. Whatever isn’t in your server’s HTML won’t get into Common Crawl.

Data from my logs

What CCBot does on my site

The real CCBot doesn’t visit often. Between October 7 and November 11, 2026, it came on 3 days, all in November, making 117 confirmed requests – 77 of them on November 11, 2026. Between February 2 and October 6, 2026, it made 647 confirmed requests.

When it comes, it reads: 102 of its 117 confirmed requests were for pages, plus 7 robots.txt checks, 5 sitemap requests and no images or scripts. 86% of its successful requests landed on content, the highest share among the bots I profile.

The impersonators were busier than the real thing: 825 requests in the same period, mostly for blog posts and articles. My server served 400 of them with a 200.

podrez.pl logsFeb 2 – Oct 6, 2026Oct 7 – Nov 11, 2026
Requests claiming to be CCBot718942
Confirmed requests647 (90%)117 (12%)
Confirmed requests per day33

“Confirmed” means the IP address is in Common Crawl’s published ranges. “Content” means any page other than the home page, robots.txt, sitemaps, WordPress system paths (/wp-content/, /wp-includes/, /wp-json/) and .php files. The February⁠–⁠October figures are the ones in my list What bot is this?; October⁠–⁠November is a fresh count against the list as it stood on November 12, 2026. Identical log lines are counted once, and each request is assigned to a single bot: the one whose name appears first in the user-agent.

Questions

Frequently asked questions about CCBot

Is CCBot used to train AI?

CCBot itself doesn’t train anything; it builds an open archive. That archive is often used as training data for language models, so blocking CCBot keeps your pages out of future crawls that others may train on.

Should I block CCBot?

Blocking it keeps your future pages out of Common Crawl’s archive, and so out of the models and research projects that use it. The cost: your site disappears from an open dataset that researchers, tools and AI companies rely on. On podrez.pl I let all AI bots in, CCBot included.

How do I check whether my site is already in Common Crawl?

Enter your domain in my free tool Is your site in Common Crawl?. It shows how many pages CCBot saved in recent crawls and whether anything blocks it.

Does CCBot run JavaScript?

No. Common Crawl says JavaScript is not executed and cookies are not used.

How do I slow CCBot down?

Add Crawl-delay to its group, for example Crawl-delay: 2. Common Crawl obeys it.

Why do I see so many CCBot requests?

Possibly because most of them aren’t CCBot. On my site, 825 of 942 requests in five weeks came from other networks. Check the reverse DNS.

Sources

Sources and method

I read all the sources below on November 12, 2026. Figures about my site come from the podrez.pl server logs for February 2 – October 6 and October 7 – November 11, 2026.

  1. Common Crawl: CCBot – user-agent, robots.txt block, a warning about fake CCBots, IP ranges with reverse DNS.
  2. Common Crawl: FAQ – robots.txt, Crawl-delay, JavaScript, sitemaps, nofollow, IP ranges as of August 11, 2026.
  3. Vercel: The rise of the AI crawler – measurement with MERJ, published December 17, 2024: which AI crawlers render JavaScript.

Compiled by Ewelina Podrez-Siama, working in SEO since 2009. Facts last checked: November 12, 2026.

Other bot profiles: ClaudeBot · GPTBot · OAI-SearchBot · Bytespider · Applebot · Amazonbot · Meta-ExternalAgent · PerplexityBot. Or go back to What bot is this?, my searchable list of bots and user-agents.

Check your site

What does your site tell CCBot?

Two free tools: one reads your robots.txt the way CCBot would, the other shows whether CCBot really visits you.

Send a project brief