Common Crawl: does AI have anything to remember you by?

What is Common Crawl?

Common Crawl is a non-profit organization founded in 2007 by Gil Elbaz, which since 2008 has done one thing: regularly crawl the internet and make the collected data available for free.

Every month CCBot visits billions of pages, downloads their content, and stores it in WARC files (Web ARChive – a page-archiving format that holds the raw HTML together with HTTP headers and metadata) on Amazon S3 servers. As of today that’s hundreds of billions of pages and over 10 petabytes of data, publicly available, with no licensing fees.

How much is 10 PB of data in Common Crawl? Roughly three times more than Netflix’s entire catalog of films and series, and over twenty times more than the ~100 million tracks available on Spotify (at standard streaming quality).

That’s a comparison of volume, not information density (text is incomparably denser than video or audio), but it gives you the scale: a lot.

Why does this matter? Because this data is one of the most important public sources on which many large language models were trained – especially in their filtered versions.

A report by Stefan Baack of the Mozilla Foundation (2024), presented at the ACM FAccT conference, showed that 64% of the 47 models analyzed used at least one filtered version of Common Crawl.

In the case of GPT-3, over 80% of the training tokens came from filtered Common Crawl. Meta’s LLaMA, Google Research’s C4 dataset, Dolma, and RedPajama – all are built on Common Crawl.

Wikidata, which I’ve written about before, tells the systems “who you are.” Common Crawl gives them the material from which they learn how to talk about you.

How to check whether your domain is in Common Crawl?

Common Crawl provides an index of every crawl (the so-called cc-index) through a simple interface. You go to index.commoncrawl.org, pick a crawl from the list (the newest is at the top), and enter your domain in the URL field. You can search a whole domain (e.g. podrez.pl/*) or a specific URL. You’ll get the same result with this query:

https://index.commoncrawl.org/CC-MAIN-2026-17-index?url=yourdomain.com/*&output=json

The crawl numbers (here CC-MAIN-2026-17) change roughly every month, and you’ll find the full list at index.commoncrawl.org/collinfo.json.

If your domain is in the index, you’ll get a response in JSON format. The most important fields:

• timestamp – exactly when CCBot downloaded your page (format: year, month, day, hour).
• status – the HTTP response code. 200 means the page was available and downloaded correctly.
• languages – the language CC categorized your page under. This matters because multilingual models may treat content in different languages with different priority.
• filename – the path to the WARC file where your page’s raw HTML was stored. Technical, but useful if you want to see exactly what CCBot “saw.”

If the query returns no response, then your domain isn’t in that crawl. Remember too that Common Crawl doesn’t archive entire sites – it downloads a selected subset of pages. The absence of a specific URL in a given crawl doesn’t mean the page is “invisible to AI.” It may mean CC only downloaded a slice of the domain in that pass.

It’s not WHETHER you’re in Common Crawl that counts, but WHAT Common Crawl pulls from your site [podrez.pl case]

The podrez.pl domain (the one you’re visiting right now) has been registered since 2008 – roughly since the start of my SEO career. At first it held a “placeholder,” then a redirect to my agency’s site, and around 2017 it began functioning as a one-pager, eventually becoming a full-fledged website.

When I started checking my presence in Common Crawl, I assumed there wasn’t much of one. I checked a few crawls at random – many had no results for podrez.pl at all. It looked as though the domain had only appeared in CC recently. But Mateusz Godzic, after reading a draft of this article, suggested searching the cc-index more broadly (thank you!), and it turned out the story is far more interesting than “zero in Common Crawl.”

Podrez.pl has been in Common Crawl since January 2017. Uninterrupted. Across dozens of crawls. Except that what CCBot pulled from it changed a great deal over the years.

Nine years in Common Crawl in a nutshell

• 2017–2018: podrez.pl is a one-pager – the homepage and robots.txt. CCBot pulls 2 URLs. There’s nothing more to crawl.
• 2019: CCBot catches the portfolio subpages – conference talks from SEO events, interviews, Fox Strategy agency case studies. The site grows, but it’s still the calling card of a specialist with a track record, not a content hub.
• 2020–2021: something at the hosting level starts returning a 406 (Not Acceptable) response to CCBot. Not just podrez.pl – the same code was returned to ms-fox.pl, my culinary blog, on the same server. Robots.txt was open, CCBot wasn’t blocked. But the server rejected it. For a year and a half, both domains were effectively invisible to Common Crawl.
• 2022: the first genuinely deeper crawl – ~36 URLs. But what does CCBot pull? The portfolio. Talks from semKRK, Festiwal SEO, workshops. Content that says “I was at a conference,” not “I’m an SEO expert.”
• 2023: 58 URLs. The site had grown with video sections, podcasts, educational materials, books. CCBot went deep. But when I reviewed exactly what it was pulling, I saw a problem:

See the difference? In 2023, not a single article that would build the picture of an SEO strategist. A model trained on that data would see someone who appears a lot on podcasts and at conferences, but not an expert who writes about entities, E-E-A-T, or generative search.

• Only from late 2025 did the crawl start to reflect my actual expertise. In the April 2026 crawl, CCBot pulled articles on brand entity, E-E-A-T, GEO vs SEO, personal branding in LLMs, brand mentions, and AI and SEO.
  
  For the first time in its history, Common Crawl had material from podrez.pl from which a model could build the picture of “an SEO strategist who also specializes in AI visibility.”

To see the scale of the problem, just compare podrez.pl with my other domains:

Other domains, other stories

• Ms-fox.pl – my culinary blog – has been in Common Crawl since 2019, with a deep crawl from day one. From the very first pass, CCBot pulled dozens of recipes, cuisine categories, lifestyle articles.
  In 2022, CC had ~90 recipe URLs from ms-fox.pl. It got a 406 too in 2020–2021 on the same server – but once unblocked, CC went straight back to deep crawling.
• Bornholm-online.pl – a guide to Bornholm – launched in February 2024. It appeared in CC two months later, and from summer 2025 CC pulled ~20 URLs per crawl: towns, attractions, trails, accommodation, cuisine.
• Madera-online.pl – a guide to Madeira – launched in February 2026. It appeared in CC after just a month. In the April 2026 crawl, 31 URLs – almost the entire, very young site: towns, attractions, trails, cuisine, history.

One thing connects these domains: from the start, I built them as a single ecosystem. Bornholm-online.pl and madera-online.pl were linked from podrez.pl and ms-fox.pl – domains CC had been crawling for years. They shared structured data (schema.org connecting Person with each domain through sameAs) and a shared author entity.

I don’t know whether that sped up their appearance in CC, but I do know that CCBot discovers new URLs through links. If you link to a new domain from a page CC already knows, you’re giving it a path.

What does this mean for AI models?

Models trained on data from 2023 had podcasts, video, and conference talks from podrez.pl. From ms-fox.pl – dozens of keto recipes. A model asked about me could describe me as a cookbook author, often skipping over more than 15 years of work in SEO.

This isn’t a hallucination in the classic sense. The model tells the truth, but… an incomplete one – that’s all it had in the training data. And that picture stays frozen until the next round of training.

This is exactly what I “fought” with in the context of my own entity, and that work became research material for my book (available in Polish)

The takeaway for you? If your site right now is a calling card with three subpages, or a portfolio of conference recordings, that portrait will stay with you for years – even if CCBot crawls it.

A few unknowns

I’d like to write that I know exactly why CCBot pulled video and podcasts from podrez.pl for years instead of articles. I don’t – but in fairness, it has to be said that there were few expert articles on the site; I only started writing more of them from 2024. Before that, the site offered CCBot what it had: a conference portfolio, recordings, and educational materials.

What I do know for certain is that:

• In 2020–2021 the hosting returned a 406 response to CCBot – on both podrez.pl and ms-fox.pl. I don’t know the reason, and it’s hard to establish now. Robots.txt wasn’t blocking CCBot, but the server rejected it.
  
  This confirms that an open robots.txt alone doesn’t guarantee a page is accessible to crawlers.
• The newer domains in my ecosystem (bornholm-online.pl, madera-online.pl) made it into CC within 1–2 months of launch – with a deep crawl right away.
  
  CC doesn’t need years to find you if you have content and linking.
• The only meaningful change I can point to on my side is the appearance of expert articles about SEO and AI. CC had been crawling podrez.pl deeply for years, but only now did it have something real to pull.

Common Crawl publishes web graphs and rankings based, among other things, on Harmonic Centrality, and its technical materials mention that this kind of data can be used to steer subsequent crawls. Does the same mechanism affect crawl depth? I’m not sure. But I’m observing something else: once articles forming a cluster around a single topic – personal brand × SEO × AI – appeared on the site, the crawl started to reflect that specialization.

I’m not claiming, by the same token, that CCBot measures a page’s topical depth. But I do know that what you publish shapes what CCBot pulls from your site.

Correlation? Strong. Causation? Impossible to prove without access to Common Crawl’s internal mechanisms.

But as a practitioner, I don’t treat this case as proof – only as a signal:

Being in Common Crawl is only the beginning. What counts is what CCBot pulls from your site – that’s the material from which training datasets can be built.

Common Crawl and the memory of LLMs

As a reminder, language models have two sources of knowledge about you:

• Parametric memory – everything the model “absorbed” during training. If your content wasn’t in Common Crawl at the moment the model was trained, the model has a far smaller chance of knowing you. Even if Google sees you. Even if you have a Knowledge Panel. Even if your Wikidata is filled in flawlessly.
• RAG (Retrieval-Augmented Generation) – systems like Perplexity, ChatGPT with web search, or Google AI Overviews reach for current sources at the moment they generate an answer. Here what counts is the page’s indexability and visibility, not its presence in the training data.

And these two channels work independently.

What can you do with knowledge about Common Crawl?

Common Crawl isn’t Wikidata – you don’t create an entry there. You can, however, influence whether CCBot finds you and whether your content has a better chance of appearing in the public index. Remember, though, that this isn’t a guide to “how to get into an LLM,” but to how not to fall out at the first stage – crawlability, accessibility, and the quality of the source HTML.

• Check robots.txt. Don’t block CCBot. If you have User-agent: CCBot with Disallow: /, that’s a deliberate decision to be absent from Common Crawl – one of the most important public data sources used in model training datasets.
  
  Many companies do this (the New York Times, Reddit). In most cases, if you’re building a personal brand and care about presence in public web datasets, blocking CCBot works against that goal.
  
• Check your firewall. Your hosting or Cloudflare may block CCBot at the firewall level, even if robots.txt is open. Aggressive anti-bot settings (e.g. Bot Fight Mode in Cloudflare) can return a 403 error without your knowledge.
  
  In my case, the hosting returned a 406 response to CCBot for a year and a half – on both of my domains at once. I didn’t know about it until I checked the cc-index. A 406 code often results from Mod_Security rules on the server that reject requests without certain headers – if you see it in the cc-index, contact your hosting support.

• Publish regularly. My case suggests that a long time can pass between CCBot’s first visit and the domain appearing in the public index. Unless the site develops dynamically and is linked from pages already present in CC.
• Take care of internal linking. CCBot discovers subpages through links, so if your most important articles are buried three clicks from the homepage, the bot may not reach them.
  
  A “Latest posts” section or a well-organized menu isn’t only UX – it’s the path the bot takes to your content.
  
• Take care of clean HTML. Common Crawl stores your page’s raw HTML – together with everything on it: navigation, footer, sidebars, cookie bars, and all the “wrapping” that isn’t the actual content (i.e. boilerplate).
  
  But note: AI models don’t train directly on that raw dump. Before your page reaches the training data, it passes through cleaning scripts like trafilatura (and given my ties to the culinary world, I have to stress that I mean text extraction here, not the pasta-shaping process 😉) or resiliparse, which try to separate the actual content from the ornaments – cut out the menu, footer, ads, and leave only what’s the article.
  
  These filters aren’t perfect, though. If your most important information appears in a sidebar or widget, the parser may treat it as a navigation element and filter it out.
  
  Even though it was in the crawl, it won’t make it into the training data. And one more thing: CCBot doesn’t render JavaScript – it pulls raw HTML. If your content loads dynamically through JS or sits behind a paywall, CCBot won’t see it.
• Expose your key content. Keep it in the main article block. Use HTML tags that clearly say “this is content” (article, section, headings) – in other words, take care of semantic HTML.
  
  And use BLUF – expose the most important conclusion at the start of the text, not at the end. The easier it is for the parser to find the actual content, the lower the risk that key information gets treated as boilerplate and filtered out.
• Be linked from domains CC crawls regularly. Common Crawl discovers new URLs through links, like a typical crawler.
  
  If no one links to you from domains present in Common Crawl, CCBot may simply never find you. Links from industry media, guest posts, citations with a link to your site are the paths by which CCBot can reach you.
• Monitor your presence. Check the cc-index after every new crawl (updated monthly). If your domain has started to appear, track which URLs are being crawled. If it isn’t appearing, work on the volume and quality of your content. CCBot doesn’t execute JavaScript, so you won’t see it in Google Analytics – the only reliable sources are your server access logs or the cc-index.

What Common Crawl doesn’t give you

Common Crawl is raw data – text, HTML, metadata. CCBot doesn’t know you’re an SEO expert. It doesn’t know you wrote a book. It doesn’t know your company has existed for ten years. It just… collects text.

That’s why Common Crawl and Wikidata are two complementary channels:

• Common Crawl gives models raw fuel – the text from which they learn language and context.
• Wikidata gives algorithms structured facts – a label on that fuel that says who you are and what confirms your expertise.
• Your site with structured data connects both worlds – it provides both content to crawl and structure to recognize.

Visibility in 2026 isn’t one place. It’s an ecosystem: Common Crawl, Wikidata, structured data, content on your site, mentions across the web. Each element strengthens the others.

Common Crawl FAQ

Sources and further reading

• Common Crawl Foundation – the project’s site, commoncrawl.org.
• Baack S., A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl, ACM FAccT, 2024.
• Baack S., Mozilla Insights, Training Data for the Price of a Sandwich: Common Crawl’s Impact on Generative AI, Mozilla Foundation, 2024.
• Brown T.B. et al., Language Models are Few-Shot Learners (GPT-3), NeurIPS 2020.
• Touvron H. et al., LLaMA: Open and Efficient Foundation Language Models, Meta AI, arXiv 2023.
• Raffel C. et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5/C4), JMLR 2020.
• Penedo G. et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, arXiv, 2024.
• CCBot – Common Crawl crawler – information about the bot, robots.txt, and crawl prioritization.
• Common Crawl Index Server – the API for searching the crawl index.
• Stephen Burns
• Wikidata step by step – how to get into the database Google trains its language models on, podrez.pl.