Tool · Common Crawl

Is your site in Common Crawl?

What matters is not only whether CCBot visits you, but what it actually saves from your site.

How to check if your site is in Common Crawl? Enter a domain to see the last six months of crawls: how many pages made it into the archive, whether anything blocks CCBot, and what your home page looks like without JavaScript.

The tool at a glance

6 crawls

roughly six months, checked in one go. Older crawls are one click away.

3 sources

the Common Crawl index, robots.txt from the archive and the home page as CCBot saved it.

0 middlemen

between your browser and Common Crawl. Nothing passes through my server.

5 tries

per request when the Common Crawl server does not answer. An outage never passes for a miss.

How it works

What the tool reads from the Common Crawl index

Every few weeks Common Crawl publishes a new crawl with an index: one line for every URL CCBot fetched. The tool asks the index about your domain in the last six crawls and takes five things from each line.

{"urlkey": "pl,podrez)/", "timestamp": "202610051205271", "url": "https://podrez.pl/2", "mime": "text/html", "status": "2003", "languages": "pol4", "filename": "crawl-data/CC-MAIN-2026-43/segments/…/warc/…-00599.warc.gz5", "offset": "411259486", "length": "88348"}

Date and time – when CCBot fetched the page: year, month, day, hour.
URL – the tool counts unique URLs and looks for your home page among them.
Server response – 200 means the page was saved, 301 is a redirect, and 403, 406 or 429 is a refusal.
Language – how Common Crawl identified the page language. Training datasets split pages by language.
Archive file – the path shows the subset: warc holds saved pages, crawldiagnostics redirects and errors, robotstxt the robots.txt files.

Result for this line: Home page 200 · saved Polish

Then it fetches two records from the archive: the robots.txt CCBot saw and your home page. It reads robots.txt the way robots do, under the RFC 9309 standard. From the home page it takes the title, description, H1, language and word count.

If you prefer to check by hand, I describe the steps in the post about Common Crawl.

How sure is the result

Not in a crawl is not the same as no answer

The Common Crawl index server is often overloaded. In my tests about every second request got a 502 or 504 error on the first try, and an answer could take more than ten seconds.

When you check by hand, this is easy to misread: an empty page or an error looks like “I’m not in Common Crawl”. That is why the tool retries each request up to five times, waiting longer each time. It reports a domain as missing only when Common Crawl explicitly says there is nothing. A crawl that did not answer gets its own row and a “Retry” button.

The same goes for robots.txt. If a crawl holds only robots.txt and no pages, the tool will not say you are in Common Crawl. It will say that CCBot visits but captures no pages.

Privacy

The address you enter does not reach me

The tool runs in your browser. From podrez.pl it loads only its own code. Requests about your domain go straight to Common Crawl, to index.commoncrawl.org and data.commoncrawl.org, just as when you search their index by hand.

What I measure. Google Analytics only learns that someone used the tool and the type of result, e.g. “in every crawl” or “not found”. Never the address you entered, and only within your cookie settings.

Result

What the result shows

Verdict, crawl by crawl – whether the domain is in the last six crawls, how many pages CCBot saved in each, and how many redirects and errors it got.
Blocks – CCBot rules in robots.txt and server responses such as 403, 406 or 429, which robots.txt never produces. On podrez.pl the hosting turned CCBot away with 406 for a year and a half while robots.txt was open.
The home page from the archive – title, description, H1, language and word count in the HTML CCBot saved, without JavaScript. If a script loads your content, you will see that the archive does not have it.
What exactly was saved – the list of saved URLs, redirects and errors, page languages and a CSV file for your own analysis.

Before you fix anything

What to do with the result

Missing from some crawls: normal for smaller sites. Each crawl is a different sample of the web.
Blocked in robots.txt: your call. If you want to be in the datasets models learn from, remove the CCBot rule. I cover the pros and cons in the post about Common Crawl.
403, 406 or 429: check your firewall, CDN and hosting settings. robots.txt will not help here.
Few words on the home page: JavaScript loads your content. Make sure the key text is in the HTML from the start.
Not in any crawl: check your server logs to see whether CCBot comes at all. The AI bots in logs tool will help.

More tools

Free, no sign-up. Each one answers one specific question.

Tool
AI bots in logs
Drop in a server log and see which AI bots actually visit, CCBot included, and whether your server turns any away.
Tool
List of bots and user agents
A searchable list of bots: who runs them, what they do and how to handle them in robots.txt.
Tool
AI knowledge cutoff dates
How far the knowledge of ChatGPT, Claude, Gemini and Grok reaches. Cutoff dates in one place.

Posts on Common Crawl, bots and AI visibility

Post
Common Crawl: does AI have anything to remember you by?
What Common Crawl is, how it feeds model memory and what CCBot took from podrez.pl over nine years.
Post
Server log analysis
Before you work on AI visibility, check whether AI bots actually reach your content.
Post
Grounding queries in Bing Webmaster Tools
Free AI visibility data most SEOs ignore: how often, and for which queries, your content gets cited.

What next

Did the result surprise you?

A CCBot block, server refusals or an empty home page in the archive are signals worth a closer look. I check Common Crawl as part of my AI search visibility work, together with server logs and what models already know about you.

Want me to take a look?

Send a short brief: your site and what the tool showed. I’ll tell you what is worth checking first.

Send a short brief AI search visibility

Frequently asked questions

How do I check if my website is in Common Crawl?

The quickest way is this tool: enter your domain in the box at the top of the page. It checks the last six crawls at once and shows how many pages CCBot saved, what it saw in robots.txt and whether your server turns it away. By hand, you can search index.commoncrawl.org one crawl at a time; I describe how in my post about Common Crawl.

How is this different from the search on index.commoncrawl.org?

The Common Crawl search shows one crawl at a time and raw index lines. This tool checks six crawls at once, tells a missing domain apart from a server that did not answer, and reads robots.txt and the home page from the archive. The manual way is in my post about Common Crawl.

Why can a check take up to two minutes?

The Common Crawl index server is often overloaded, so the tool retries. Results appear crawl by crawl, so you don’t have to wait for all of them.

Common Crawl has only some of my pages. Is that a problem?

No. Common Crawl never fetches whole sites, only a sample. What matters more is what ends up in the archive: whether it includes the pages that show your expertise and your offer.

I’m in Common Crawl. Do AI models know me?

Not necessarily. Common Crawl is a raw archive, and model builders filter it in their own way. More on this in the post about Common Crawl.

What does a 403 or 406 in the result mean?

Your server or firewall turns CCBot away even when robots.txt is open. Check your firewall rules and ask your host. To see exactly who visits your site, use AI bots in logs.

Do you store the addresses I check?

No. The address goes from your browser straight to Common Crawl. Google Analytics only gets the type of result, never the address.

Prepared by Ewelina Podrez-Siama, in SEO since 2009. Latest version of the tool: 1.0.8, 1 November 2026.

Data: the Common Crawl Foundation crawl index and WARC archive. This tool is not affiliated with Common Crawl.