Google-Extended Explained: Controlling Gemini Training Without Losing Search

Google-Extended is the one AI opt-out you can take without paying for it in rankings, and it is still widely misread as a ranking risk. Teams postpone the decision for quarters because somebody in the room believes that disallowing anything with Google in the name will cost organic traffic.

It will not. Google says so in its own crawler documentation, in one sentence, without hedging.

That makes this a licensing decision rather than a traffic decision, and licensing decisions belong to different people than robots.txt usually does. This post covers exactly what the token governs, what it leaves untouched, how to implement it at full or partial scope, and how the choice compares to the equivalent controls from OpenAI, Apple, and Common Crawl.

What Is Google-Extended?

Google-Extended is a product token, not a crawler. That distinction is the source of most of the confusion around it.

Google's crawler documentation states that Google-Extended does not have a separate HTTP request user agent string. Crawling is performed with existing Google user agent strings, and the robots.txt token exists purely in what Google calls a control capacity.

Three consequences follow, and each one trips somebody up.

You will never see Google-Extended in your server logs. Grep 12 months of access logs for the string and you get 0 rows, because no fetch was ever attributed to it. Teams that go looking for its crawl volume before deciding find nothing and conclude the token is inactive. It is not inactive, it simply is not a fetching agent.

There is no crawl budget argument for or against it. Disallowing Google-Extended does not reduce requests to your origin by one, because no request was ever attributed to it. If your reason for blocking a bot is server load, this is not the token to reach for.

It cannot be verified by reverse DNS the way a real crawler can. Verification techniques that work for Googlebot are irrelevant here, because there is no Google-Extended fetch to verify.

What It Actually Controls

Google-Extended governs two downstream uses of content Google has already crawled, and both sit outside Search.

The first is model training. Content Google crawls may be used to train future generations of Gemini models, the ones powering Gemini Apps and the Vertex AI API for Gemini. Disallowing the token removes your content from that training corpus going forward.

The second is grounding. Google defines grounding as providing content from the Google Search index to the model at prompt time to improve factuality and relevancy, and it applies in Gemini Apps and in Grounding with Google Search on Vertex AI. This is the part most teams miss, and it is the part with a visibility cost.

Grounding is what produces a citation. When Gemini answers a question by pulling current information from the Search index and attributing it, that is grounding at work. Disallow Google-Extended and you are opting out of being the grounded source, which means opting out of the citation that comes with it.

So the token bundles a training decision and a visibility decision into a single directive. You cannot currently take one and leave the other. That is the real cost of the control, and it is worth naming plainly before anybody signs off on it.

Does Blocking It Hurt Rankings?

No. Google states that Google-Extended does not impact a site's inclusion in Google Search, nor is it used as a ranking signal in Google Search.

That sentence is unusually direct by the standards of Google's documentation, which normally prefers guidance to guarantees. It is worth reading twice, because it settles the question that stalls the decision in most organisations.

Two clarifications keep the reassurance honest.

The statement covers inclusion and ranking. It does not promise that every surface Google operates behaves identically, and Google has historically shipped new AI products under new tokens rather than retrofitting old ones. A decision made today governs the products documented today.

The statement is about Search, and Gemini is not Search. If your organic traffic is the only metric you are protecting, disallowing costs you nothing measurable. If you are also measuring citation share in Gemini, disallowing costs you real ground, and you should expect to see it.

Our guide to measuring GEO performance covers how to instrument that second number before you make a change that moves it.

Google-Extended Does Not Control AI Overviews

This is the single most common misconception, and getting it wrong produces a decision that fails to do the thing it was made to do.

AI Overviews are a Google Search feature. They are assembled from the Search index, which means they are governed by Googlebot and by the standard snippet controls, not by Google-Extended. Disallowing Google-Extended does not remove you from AI Overviews.

If your objective is to stay out of AI Overviews, the levers are different and blunter.

• The nosnippet meta tag removes the text snippet Google can display, which removes the material an Overview would draw on.
• max-snippet with a low character value limits how much text can be shown.
• data-nosnippet on specific HTML elements excludes just those passages.
• Disallowing Googlebot removes you from Search entirely, which is almost never the intent.

Every one of those has a genuine traffic cost, unlike Google-Extended. That asymmetry is the point: the training and grounding opt-out is cheap, and the Search-surface opt-out is expensive. Teams that conflate the two either pay a cost they did not intend or fail to achieve the exclusion they wanted.

How to Implement It

Implementation is a robots.txt group, and the useful part is that it supports partial scope.

A full opt-out is 2 lines and takes about 30 seconds.

User-agent: Google-Extended
Disallow: /

A partial opt-out is more interesting and usually the better answer. Google's own documentation shows an example that allows one path while disallowing its parent, and Google's robots.txt specification resolves conflicts by the most specific matching rule rather than by order.

User-agent: Google-Extended
Allow: /blog/
Disallow: /

That pattern reflects how most publishers actually feel. Marketing content exists to be found and quoted, so let it ground. Proprietary research, customer data, gated material, and original datasets are the assets you may not want inside a training corpus, so exclude those paths.

The 3 tokens people confuse with this one are Googlebot, which governs Search itself, Google-CloudVertexBot, which covers site-owner-requested Vertex AI Agent crawls, and Googlebot-News. None of them is interchangeable with Google-Extended.

Four implementation notes worth checking before you ship.

• Put the group in the root robots.txt at your canonical host. Subdomains need their own file, and a rule on www does not govern a bare-apex variant that resolves separately.
• Google-Extended honours only robots.txt. There is no meta tag equivalent, so page-level control is not available.
• The directive is forward-looking. It governs future training and current grounding, and it does not retract content already used in a shipped model.
• Verify the parsed result in Search Console's robots.txt report rather than by eye, since a stray rule earlier in the file changes what applies.

How It Compares to Other Opt-Outs

Google-Extended is one control in a set, and the set is inconsistent by design. Each vendor drew the training and retrieval line in a different place.

• Google-Extended bundles training and grounding into one token, with no separation available, and explicitly does not affect Search.
• GPTBot governs OpenAI training. It is a real crawler with its own user agent, and it is separate from OAI-SearchBot, which handles retrieval for ChatGPT search. OpenAI split what Google bundled, so you can block training and keep citations. Our post on GPTBot versus OAI-SearchBot covers why blocking one does not block the other.
• Applebot-Extended governs Apple foundation model training while plain Applebot continues to power Spotlight, Siri, and Safari. Apple also split the decision.
• CCBot governs Common Crawl, which is an upstream dataset feeding many models you will never negotiate with directly.

The practical consequence is that there is no single AI opt-out and no coherent middle position across vendors. A policy has to be written per token, and the pattern most publishers land on is to allow retrieval everywhere, restrict training selectively, and accept that Google currently forces those two together. Our guide to robots.txt for AI crawlers covers the full directive set.

The deeper distinction underneath all of this is the difference between AI training and AI retrieval, which is the framework worth having before writing any of these rules.

How to Decide

The decision comes down to what your content is for, and it splits cleanly along business model.

Allow Google-Extended if your content exists to generate demand. Marketing sites, SaaS documentation, agency blogs, and most B2B content gain more from being the grounded, cited source in Gemini than they lose by contributing to a training corpus. For these sites the grounding citation is the whole point, and blocking it forfeits visibility on a growing surface to protect content you publish for free anyway.

Disallow it, at least partly, if your content is the product. Publishers selling subscriptions, research firms selling reports, course businesses, and data providers are in a different position. Training use is uncompensated substitution, and there is a reasonable argument for excluding the paths that carry the sellable asset while allowing the marketing pages that sell it.

Disallow it broadly if you have a licensing negotiation or a legal position to protect. Some organisations need the ability to say their content was withheld. That is a legitimate reason and it does not need a traffic justification, particularly given the token costs nothing in Search.

Whatever you choose, record the reasoning and the date somewhere durable. These tokens change, vendors add new ones, and a decision with no written rationale gets silently reversed at the next site migration.

Common Mistakes

• Expecting to see it in server logs. There is no Google-Extended user agent, so its absence from your logs tells you nothing about whether the directive is working.
• Blocking it to stay out of AI Overviews. Overviews come from the Search index and are governed by Googlebot and snippet controls. This token has no effect on them.
• Treating it as a crawl budget lever. It never issued a request, so disallowing it saves zero bandwidth.
• Blocking it and expecting Gemini citations to continue. Grounding is bundled into the same token, so a full disallow removes you from the grounded source pool.
• Writing the rule on one host only. Subdomains and apex variants that serve independently need their own robots.txt.
• Assuming a block is retroactive. The directive governs future training, not models already shipped.

Frequently Asked Questions

Does blocking Google-Extended affect my Google rankings?

No. Google's crawler documentation states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It governs Gemini training and grounding only, which sit outside Search entirely.

Will blocking Google-Extended remove me from AI Overviews?

No. AI Overviews are built from the Google Search index and are controlled by Googlebot access plus the snippet directives such as nosnippet and max-snippet. Google-Extended has no effect on that surface.

Can I block training but keep Gemini citations?

Not with Google-Extended as currently documented, because the token bundles training and grounding together. OpenAI separates the two with GPTBot for training and OAI-SearchBot for retrieval, so the split is available there but not here.

Is there a meta tag version of Google-Extended?

No. The token is honoured only in robots.txt, so control is path-level rather than page-level. If you need finer granularity, structure the URLs so the content you want excluded sits under its own path prefix.

Does disallowing Google-Extended remove content already used in training?

No. The directive is forward-looking. It prevents future training use and stops current grounding, and it does not retract content already incorporated into a released model.