Applebot and Applebot-Extended: Two Bots, Two Very Different Decisions

Apple is the only major AI vendor that gives publishers three separate controls over three separate uses of the same crawl, and almost nobody uses all three deliberately. The usual outcome is a robots.txt rule that blocks the wrong thing and quietly removes a site from Siri, Spotlight, and Safari while leaving the training use it was written to prevent fully intact.

The names are the problem. Applebot and Applebot-Extended read like a crawler and a more aggressive version of the same crawler. They are not. One is a crawler, one is a policy token, and they answer different questions.

This post separates the three controls, shows what each one costs, and gives you the robots.txt and meta tag configuration for the position most publishers actually want.

What Applebot Does

Applebot is the crawler that makes your site visible inside Apple's ecosystem, and it is the one you almost certainly want to allow.

Apple's Applebot documentation states that the data Applebot crawls powers the search technology integrated across Apple's ecosystem, naming Spotlight, Siri, and Safari specifically. It also says, in plain terms, that enabling Applebot in robots.txt is what allows your content to appear in search results for Apple users in those products.

That single crawler feeds 3 distinct surfaces, and a 1-line disallow removes you from all 3 at once.

• Spotlight search on every Mac, iPhone, and iPad, which is how a large share of Apple users start a query without opening a browser.
• Siri results, including the suggestions surfaced before a query is fully typed.
• Safari Suggestions in the address bar, which intercepts navigational intent at the moment of typing.

None of that traffic shows up in your analytics as an Apple search referral in the way Google organic does, which is exactly why it gets undervalued. The visibility is real and the measurement is poor, and teams cut what they cannot measure.

Blocking plain Applebot is the expensive mistake in this whole area. It is a one-line rule that removes a site from three surfaces on a billion devices, and it is frequently written by somebody whose actual goal was to decline AI training.

What Applebot-Extended Does

Applebot-Extended is a training opt-out, and it does nothing else.

Apple documents that content crawled by Applebot may also be used to help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools. Publishers who do not want that can opt out by disallowing Applebot-Extended in robots.txt.

The critical detail is what the opt-out leaves alone. Apple states that even if you disallow Applebot-Extended and tag content with nosnippet, your site instructions may still allow Applebot to crawl your webpages. The training decision and the crawling decision are independent.

That independence is the whole value of the control. You can decline to contribute to Apple's foundation models and keep every unit of Spotlight, Siri, and Safari visibility. Very few AI opt-outs are that cheap, and Apple's is one of them.

Like Google-Extended, Applebot-Extended is forward-looking. It governs future training use rather than retracting content from a model already trained and shipped.

The Third Control Most Teams Miss

Apple describes a third, distinct use of crawled content, and it is governed by a meta tag rather than robots.txt.

Apple states that crawled data may be used to provide additional context and up-to-date content when AI models generate output in Apple products, giving the example of answering broad world knowledge questions in Siri and Search, including links to the sources used to help generate the answer. Publishers can opt out of that specific use by applying the nosnippet meta tag to the content in question.

Read the mechanism carefully, because it cuts both ways.

Those answers include links to sources. That is a citation, and citations on an Apple surface are worth having. Opting out with nosnippet removes you from a surface that sends attributed traffic.

The control is page-level, not path-level. Because it runs through a meta tag rather than robots.txt, you can exclude one sensitive page without touching the rest of the site, which is finer granularity than either robots.txt token offers.

nosnippet is not Apple-specific. It is a standard directive that other search engines honour, so applying it to remove yourself from Apple's world knowledge answers also suppresses your snippet in Google. That side effect is usually the reason not to use it.

So the practical hierarchy is clear. Applebot-Extended is a cheap, Apple-specific training opt-out. nosnippet is an expensive, cross-engine snippet suppression that happens to also cover Apple's generated answers.

How to Implement All Three

Here is the configuration most publishers want: keep all Apple search visibility, decline foundation model training, and stay in world knowledge answers.

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

Two rules, 4 lines, and the outcome is full Spotlight, Siri, and Safari visibility across all 3 surfaces with no training contribution.

If you want to allow training on marketing content while protecting a proprietary section, scope it by path.

User-agent: Applebot-Extended
Allow: /blog/
Disallow: /research/
Disallow: /

If a specific page must stay out of Apple's generated answers, add the meta tag to that page only, accepting that it suppresses your Google snippet too.

<meta name="robots" content="nosnippet">

Three checks before shipping.

• Confirm the rules resolve at the canonical host, and add them to any subdomain that serves independently.
• Do not write a blanket User-agent: * disallow and expect the Applebot groups to save you, since the most specific matching group wins and a site-wide block written elsewhere can still catch other agents you meant to allow.
• Watch your logs for the Applebot user agent after the change to confirm crawling continues. Its presence is how you verify you blocked the token and not the crawler.

What You Lose by Blocking the Wrong One

The 2 failure modes have very different costs, and only 1 of them is recoverable quickly.

Blocking Applebot when you meant Applebot-Extended removes you from Apple search entirely. Spotlight stops returning your pages, Siri stops surfacing them, Safari Suggestions drops them. Training continues to be unaffected because you never wrote the token that governs it. You paid the maximum price and achieved nothing you intended. Recovery requires a robots.txt fix plus a recrawl cycle.

Blocking Applebot-Extended when you actually wanted out of everything leaves crawling and world knowledge answers intact. That is a much smaller error, and for most publishers it is the position they should have chosen anyway.

The asymmetry matters when you are deciding how carefully to review the change. A mistake in one direction is a rounding error and a mistake in the other is a visibility outage on a billion devices.

Our post on Apple Intelligence search covers the optimisation side of the Apple surfaces this decision protects.

How Apple Compares to Google and OpenAI

Apple, OpenAI, and Google drew the training and retrieval line in three different places, which is why a single AI policy statement cannot be applied across all of them.

• Apple separates crawling, training, and generated answers into three controls, giving publishers the finest granularity of the three vendors.
• OpenAI separates training from retrieval with GPTBot and OAI-SearchBot as distinct crawlers, which our post on GPTBot versus OAI-SearchBot covers in detail.
• Google bundles training and grounding into the single Google-Extended token, so declining training also declines Gemini grounding citations.

Apple's split is the most publisher-friendly of the three. You can keep every search surface and every citation surface while declining the training use, which is precisely the position most content businesses would choose if the vendor allowed it. Google currently does not.

Write your directives per vendor and stop looking for a coherent cross-vendor policy, because none exists. Our guide to robots.txt for AI crawlers covers the full set, and the training versus retrieval distinction is the framework to have before writing any of them.

How to Decide

For the large majority of sites, the answer is allow Applebot, disallow Applebot-Extended, skip nosnippet.

That position keeps all three Apple search surfaces, keeps you eligible for cited world knowledge answers, and declines the one use that returns nothing to you. It costs nothing measurable and it is two lines of robots.txt.

Reconsider only in two situations.

If your content is the product you sell, and Apple Intelligence summarising it substitutes for a subscription, then a broader stance is defensible. Even then, scope it by path so your marketing pages keep working.

If you have a licensing conversation open with Apple or a legal position to preserve, the documented opt-out is the artifact that demonstrates content was withheld. That reason needs no traffic justification.

Whatever you land on, write the decision and the date into the same place you keep your robots.txt rationale. Apple has added tokens before and will again, and an undocumented rule is one migration away from being deleted by somebody who does not know why it existed.

Frequently Asked Questions

Does blocking Applebot-Extended remove me from Siri and Spotlight?

No. Apple documents that disallowing Applebot-Extended stops foundation model training while Applebot continues to crawl, and Applebot is the agent that feeds Spotlight, Siri, and Safari. Blocking plain Applebot is what removes you from those surfaces.

Is Applebot-Extended a separate crawler I will see in my logs?

No. It functions as a robots.txt policy token governing how already-crawled content may be used. The crawling itself is attributed to Applebot, which is what appears in your server logs.

How do I stay out of Apple's AI answers but keep Apple search?

Apply the nosnippet meta tag to the pages in question. Be aware it is a cross-engine directive, so it also suppresses your Google snippet, which is usually a bigger cost than the Apple exclusion is worth.

Does the opt-out apply retroactively to models already trained?

No. Applebot-Extended governs future training use. Content already incorporated into a released Apple foundation model is not retracted by adding the directive now.

Should I block Applebot to save crawl budget?

Almost never. Applebot is generally well behaved, and the cost of blocking it is losing visibility across Spotlight, Siri, and Safari. If crawl load is the genuine problem, rate limiting at the edge is the better tool than a robots.txt disallow.