Ewelina Podrez-Siama · bot profiles
CCBot Common Crawl’s crawler: user-agent, IP addresses and robots.txt
CCBot is the crawler of Common Crawl, a non-profit that keeps an open repository of web crawl data. That archive is a common source of training data for language models.
Below you’ll find CCBot’s user-agent, its IP ranges and reverse DNS check, robots.txt rules and what CCBot – and the many fake CCBots – did on podrez.pl. I checked the facts against Common Crawl’s documentation and the sources below on November 12, 2026.
requests claiming to be
CCBot, Oct 7 – Nov 11, 2026
came from
Common Crawl
of its successful requests
landed on content
At a glance
CCBot at a glance
| Question | Answer |
|---|---|
| Operator | Common Crawl, a non-profit foundation |
| Purpose | Builds an open repository of web crawl data that anyone can access and analyze. It’s often used as a training dataset for language models. |
| robots.txt token | CCBot |
| Without a CCBot group | The User-agent: * group applies. |
| Honors robots.txt | Yes. CCBot checks robots.txt before fetching a page, obeys Crawl-delay and keeps checking the file for updates. |
| IP addresses | index.commoncrawl.org/ccbot.json and the FAQ: 4 IPv4 ranges and 1 IPv6 range, as of August 11, 2026. |
| Reverse DNS | Yes, over IPv4: names in crawl.commoncrawl.org. Not available over IPv6. |
| Runs JavaScript | No. Common Crawl says JavaScript is not executed and cookies are not used. |
| Check your site | My free tool Is your site in Common Crawl? shows how many of your pages CCBot saved. |
| In my list | AI: training (see the entry in “What bot is this?”) |
User-agent
The CCBot user-agent
Common Crawl documents a single current string, and confirmed CCBot requests on my site used exactly that:
| Requests | User-agent string |
|---|---|
| 117 | CCBot/2.0 (https://commoncrawl.org/faq/) |
Number: requests from Common Crawl’s published addresses to podrez.pl, Oct 7 – Nov 11, 2026.
That’s the problem: the string is short and easy to copy. Between October 7 and November 11, 2026, 825 requests with exactly the same string came from other networks: a Polish hosting company, eTOP (559), BlueVPS in Estonia (149), Google Cloud (70) and others (47). Common Crawl itself warns that some crawlers falsely identify themselves as CCBot.
Common Crawl says it may increment the version number in the future, so match CCBot, not the whole string.
robots.txt
How to block or allow CCBot in robots.txt
CCBot checks robots.txt before it fetches a page and follows the group that names it (a group is a block of rules that starts with User-agent).
Variant 1 · Out of Common Crawl
CCBot stays outI keep my site out of Common Crawl’s future crawls.
User-agent: CCBot
Disallow: /
Variant 2 · Slower
CCBot stays inI let CCBot in but ask for at most one request every 2 seconds.
User-agent: CCBot
Crawl-delay: 2
User-agent: * rules for this bot, so copy over every Disallow line you still need.CCBot also slows down on its own when your server answers with 429 or 5xx, reads the sitemaps listed in robots.txt and honors nofollow on links.
To see whether CCBot has already saved your pages, try my free tool Is your site in Common Crawl?.
Check your file
Not sure what your robots.txt tells CCBot? Enter your domain or paste the file into Can AI bots read your site?, my free robots.txt checker. It shows whether CCBot may crawl your pages and which line decides it.
IP check
Is it really CCBot? How to check the IP address
Common Crawl runs CCBot on dedicated IP ranges with reverse DNS. The ranges are published at index.commoncrawl.org/ccbot.json and in the FAQ: on November 12, 2026, the FAQ listed 4 IPv4 ranges and 1 IPv6 range, as of August 11, 2026. The JSON file didn’t respond when I checked on November 12, 2026, so I used the FAQ copy, which matches my August download.
Quick glossary: the user-agent is the name a bot gives itself, and anyone can type any name. A prefix is a block of IP addresses. Reverse DNS turns an address back into a host name. A WAF (web application firewall) is the filter in front of your site, such as Cloudflare, your host’s firewall or a security plugin.
- Take the IP address from the log line.
- For IPv4, run
hoston it (in Terminal on a Mac or Linux): the name should end incrawl.commoncrawl.org, and runninghoston that name should return the same address. For IPv6, check the published range. - If the check fails, it’s an impersonator. Block it on the server or in your WAF, not in robots.txt.
This check matters more for CCBot than for any other bot I profile. Between October 7 and November 11, 2026, only 117 of the 942 requests claiming to be CCBot (12%) were confirmed. Earlier in the year it was the other way around: 647 of 718 (90%) between February 2 and October 6, 2026.
To check a whole log file at once, load it into Check AI bots in your logs, my free log checker. The file never leaves your browser, and the tool checks every address against Common Crawl’s list.
JavaScript
Does CCBot run JavaScript?
No.
Common Crawl says JavaScript is not executed and cookies are not used. A widely cited measurement by Vercel and MERJ (published December 17, 2024) also found that CCBot doesn’t render pages.
My logs agree: confirmed CCBot requests included no scripts or stylesheets between October 7 and November 11, 2026. Whatever isn’t in your server’s HTML won’t get into Common Crawl.
Data from my logs
What CCBot does on my site
The real CCBot doesn’t visit often. Between October 7 and November 11, 2026, it came on 3 days, all in November, making 117 confirmed requests – 77 of them on November 11, 2026. Between February 2 and October 6, 2026, it made 647 confirmed requests.
When it comes, it reads: 102 of its 117 confirmed requests were for pages, plus 7 robots.txt checks, 5 sitemap requests and no images or scripts. 86% of its successful requests landed on content, the highest share among the bots I profile.
The impersonators were busier than the real thing: 825 requests in the same period, mostly for blog posts and articles. My server served 400 of them with a 200.
| podrez.pl logs | Feb 2 – Oct 6, 2026 | Oct 7 – Nov 11, 2026 |
|---|---|---|
| Requests claiming to be CCBot | 718 | 942 |
| Confirmed requests | 647 (90%) | 117 (12%) |
| Confirmed requests per day | 3 | 3 |
“Confirmed” means the IP address is in Common Crawl’s published ranges. “Content” means any page other than the home page, robots.txt, sitemaps, WordPress system paths (/wp-content/, /wp-includes/, /wp-json/) and .php files. The February–October figures are the ones in my list What bot is this?; October–November is a fresh count against the list as it stood on November 12, 2026. Identical log lines are counted once, and each request is assigned to a single bot: the one whose name appears first in the user-agent.
Questions
Frequently asked questions about CCBot
Is CCBot used to train AI?
CCBot itself doesn’t train anything; it builds an open archive. That archive is often used as training data for language models, so blocking CCBot keeps your pages out of future crawls that others may train on.
Should I block CCBot?
Blocking it keeps your future pages out of Common Crawl’s archive, and so out of the models and research projects that use it. The cost: your site disappears from an open dataset that researchers, tools and AI companies rely on. On podrez.pl I let all AI bots in, CCBot included.
How do I check whether my site is already in Common Crawl?
Enter your domain in my free tool Is your site in Common Crawl?. It shows how many pages CCBot saved in recent crawls and whether anything blocks it.
Does CCBot run JavaScript?
No. Common Crawl says JavaScript is not executed and cookies are not used.
How do I slow CCBot down?
Add Crawl-delay to its group, for example Crawl-delay: 2. Common Crawl obeys it.
Why do I see so many CCBot requests?
Possibly because most of them aren’t CCBot. On my site, 825 of 942 requests in five weeks came from other networks. Check the reverse DNS.
Sources
Sources and method
I read all the sources below on November 12, 2026. Figures about my site come from the podrez.pl server logs for February 2 – October 6 and October 7 – November 11, 2026.
- Common Crawl: CCBot – user-agent, robots.txt block, a warning about fake CCBots, IP ranges with reverse DNS.
- Common Crawl: FAQ – robots.txt, Crawl-delay, JavaScript, sitemaps, nofollow, IP ranges as of August 11, 2026.
- Vercel: The rise of the AI crawler – measurement with MERJ, published December 17, 2024: which AI crawlers render JavaScript.
Compiled by Ewelina Podrez-Siama, working in SEO since 2009. Facts last checked: November 12, 2026.
Other bot profiles: ClaudeBot · GPTBot · OAI-SearchBot · Bytespider · Applebot · Amazonbot · Meta-ExternalAgent · PerplexityBot. Or go back to What bot is this?, my searchable list of bots and user-agents.
Check your site
What does your site tell CCBot?
Two free tools: one reads your robots.txt the way CCBot would, the other shows whether CCBot really visits you.