CCBot and Common Crawl: The Dataset Behind Most Open Models

Common Crawl is the most consequential crawler most marketing teams have never discussed. It is not a search engine, it will never send you a visitor, and its archives have fed a large share of the language models released over the past decade.

That combination makes it awkward to reason about. There is no traffic to lose and no citation to win, so the usual cost-benefit framing does not apply. What you are actually deciding is whether your content appears in a public dataset that anybody can download and train on, including organisations that do not exist yet.

And unlike a per-vendor opt-out, this decision has a hard deadline built into it. Whatever is already in the archive is already distributed.

What Common Crawl Is

Common Crawl is a non-profit foundation that publishes an open repository of web crawl data. Its stated purpose, in its own CCBot documentation, is to democratize access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.

That mission is genuinely worth something. Academic researchers, non-profits, and small teams who could never fund a web-scale crawl of their own build on it, and a meaningful body of published research depends on it existing.

It is also the reason the archive is impossible to control after the fact. Universally accessible means downloadable by anyone, and downloadable by anyone means copies exist outside the foundation's hands the moment a snapshot ships.

The crawler identifies itself in its user agent as CCBot/2.0 followed by a link to the foundation's FAQ. Snapshots are released on a regular cadence, historically several per year, and each one is published as a set of archive files running to petabytes that anybody can pull without an account.

Why It Sits Upstream of Everything Else

The reason a CCBot decision matters more than its traffic value suggests is positional. Common Crawl is not a consumer of your content, it is a supplier to consumers of your content.

Every other crawler control you write is a bilateral arrangement. Disallow GPTBot and you have told OpenAI something. Disallow Google-Extended and you have told Google something. The relationship is one to one, the vendor is identifiable, and if their policy changes you can change your directive.

Common Crawl breaks that model in 3 ways.

The consumers are unbounded. Anybody can download a snapshot and train on it, which means a single inclusion decision propagates to an unknown and growing set of models. You cannot enumerate the counterparties.

The consumers are often unreachable. A research lab that trained on a 2024 snapshot has no obligation to you, no opt-out form, and frequently no commercial relationship to negotiate. There is nobody to write to.

The lag is long. A snapshot taken this year may be used to train a model released in two years by an organisation that does not exist today. The consequences of the decision arrive well after the decision.

This is why the upstream position deserves an explicit choice rather than a default. Our post on the difference between AI training and AI retrieval covers why training decisions behave differently from retrieval decisions, and Common Crawl is the purest training case there is. It never retrieves anything on a user's behalf, so there is no citation upside to weigh.

Why a Block Is Never Retroactive

Disallowing CCBot prevents your pages appearing in future snapshots. It does nothing about the ones already published, and the distinction is absolute rather than a matter of degree.

Think about what a released snapshot actually is. It is a set of files sitting in public storage, already downloaded by an unknown number of parties, already used as training input by some of them, and already redistributed in derived datasets by others. There is no recall mechanism, because there was never a single copy to recall.

Three practical consequences follow, and the first one is the one that surprises people.

Blocking today protects tomorrow only. If your content has been publicly reachable for years, assume it is in the archive and plan on that basis rather than hoping otherwise.

Blocking does not remove you from models already trained. Those weights are shipped. No robots.txt directive reaches them.

Blocking is still worth doing if your position is forward-looking. New content, newly published research, and material behind a business model you are protecting all benefit from the rule existing now.

The honest framing for a stakeholder conversation is this: a CCBot disallow is a policy statement about future publishing, not a remediation of past exposure. Teams that expect the second are disappointed, and teams that wanted the first are well served.

If your actual concern is content that was never meant to be public in the first place, the crawler rule is the wrong layer entirely. Our post on LLM data leakage covers the pattern where the real problem was an exposed staging site or an unlisted documentation subdomain rather than a crawler policy.

How to Block It

Common Crawl documents the directive directly, and it is two lines.

User-agent: CCBot
Disallow: /

Partial scope works the same way it does for any other agent, and it is usually the more sensible position.

User-agent: CCBot
Allow: /blog/
Disallow: /research/
Disallow: /

Four notes on getting it right.

• The rule must sit in the root robots.txt of each host that serves independently. A directive on your primary domain does not govern a documentation subdomain.
• CCBot honours robots.txt only. There is no meta tag equivalent, so exclusion is path-level.
• Expect a lag between publishing the rule and seeing it take effect, because it applies from the crawler's next pass rather than immediately.
• Keep the rule in place. Removing it during a site migration quietly re-enrols you, and this is exactly the kind of directive that gets dropped when robots.txt is regenerated by a new platform.

Verifying the Real CCBot

Common Crawl states plainly that it is aware of crawlers falsely identifying themselves as CCBot, and recommends verifying user agent strings to ensure authenticity. This matters more than it sounds.

A spoofed user agent is trivial to set. Any scraper can send CCBot/2.0 in its headers, which means log entries claiming to be CCBot are unreliable evidence on their own. Two failure modes follow from trusting them.

You attribute load to Common Crawl that belongs to somebody else, then write a rule that has no effect on the actual traffic because the impostor was never honouring robots.txt in the first place.

You conclude your CCBot block is being ignored, and escalate to a blanket firewall rule that catches legitimate crawlers you wanted.

The foundation documents the fix. CCBot now runs on dedicated IP address ranges with reverse DNS, except over IPv6 where reverse DNS is not yet supported, which allows webmasters to verify whether a logged request came from the real crawler. The verification pattern is the same forward-confirmed reverse DNS check used for Googlebot, and it is 2 lookups: resolve the requesting IP to a hostname, then resolve that hostname back to an IP and confirm the 2 match.

host 18.97.14.84

Anything claiming to be CCBot from an IP that fails that check is an impostor, and impostors are a firewall problem rather than a robots.txt problem. Our guide to AI crawler log analysis covers how to build that verification into a repeatable review rather than doing it once during an incident.

How to Decide

The decision splits on whether your content is a marketing asset or a commercial asset, and the split is cleaner here than for any other crawler.

Allow CCBot if your content exists to be found and quoted, and you take a broadly permissive view of open research. There is no citation to lose either way, so the argument for allowing is mission-based rather than commercial: the archive supports work you may believe should exist, and your marketing content is published free anyway.

Disallow CCBot if your content is the thing you sell. Publishers, research firms, data providers, and course businesses are contributing a sellable asset to a public dataset with no compensation, no attribution, and no ability to identify who benefits. That is a straightforward business case for exclusion, and the absence of any retrieval upside makes it an easier call than the equivalent Google-Extended decision.

Disallow selectively if you are somewhere in the middle, which most companies are. Marketing pages allowed, proprietary research and original datasets excluded by path. This is the position we would recommend by default for a B2B company with genuine primary research.

One more consideration that argues for deciding now rather than later. Because the archive is not retroactive, delay has a real cost. Every month of deliberation is another snapshot. If the answer is likely to be exclusion for some paths, writing the rule early is worth more than writing the perfect rule late.

Common Mistakes

• Expecting a block to remove you from the archive. Published snapshots are already distributed. The directive governs future crawls only.
• Trusting the user agent in your logs. Common Crawl warns that impostors exist. Verify by reverse DNS before drawing conclusions.
• Treating it like a vendor opt-out. There is no single counterparty, no negotiation, and no citation upside to weigh against the training use.
• Dropping the rule in a migration. Regenerated robots.txt files lose custom directives, and nobody notices because CCBot sends no traffic to miss.
• Blocking CCBot while leaving a staging subdomain open. The archive picks up whatever is reachable, and an unlisted host is still reachable.
• Blocking it for crawl budget. If load is the problem, rate limit at the edge. A robots.txt rule is a policy tool, not a throttle.

Frequently Asked Questions

Does blocking CCBot remove my content from AI models?

No. It prevents inclusion in future Common Crawl snapshots. Archives already published are already distributed, and models already trained on them are unaffected by any directive you add now.

Is CCBot the same as GPTBot or ClaudeBot?

No. Those are vendor crawlers collecting for one company each. CCBot collects for a public archive that many organisations download and train on, which puts it upstream of the vendor-specific crawlers rather than alongside them.

How do I tell a real CCBot request from a fake one?

Common Crawl runs the crawler from dedicated IP ranges with reverse DNS. Resolve the requesting IP to a hostname and resolve that hostname back to an IP, then confirm the two match. Requests that fail the check are impostors and belong at the firewall.

Will blocking CCBot affect my Google rankings?

No. Common Crawl is unrelated to Google Search. The archive is an independent non-profit dataset, and excluding yourself from it has no bearing on how Googlebot crawls or how Search ranks your pages.

Should a normal marketing site block CCBot?

Usually not. With no citation upside and no traffic either way, the decision rests on whether your content is a commercial asset. For a standard marketing site the pragmatic answer is to allow it and exclude only genuine proprietary research by path.