Always-on monitoring of Anthropic's three-bot fleet. ClaudeBot trains models. Claude-User retrieves pages mid-conversation. Claude-SearchBot indexes for Claude's search tool. All three honor robots.txt. Sentry catches per-bot mismatches and stale legacy rules. Cortex handles the fix.
Continuous audits of Anthropic's three-bot fleet against the 8 things that decide whether you're trained into, fetched live by, and indexed for Claude. Per Anthropic: all three honor robots.txt independently. A single rule does not cover all three. Sentry catches the gaps. Cortex fixes it.
User-agent: ClaudeBot (the training crawler) is not Disallowed. Per Anthropic: ClaudeBot 'helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.'
User-agent: Claude-User is addressed in robots.txt. Per Anthropic: Claude-User 'may access websites using a Claude-User agent when individuals ask questions to Claude.' All three Anthropic bots honor robots.txt; no implicit consent.
User-agent: Claude-SearchBot is not Disallowed. Independent token from ClaudeBot and Claude-User. Per Anthropic: 'analyzes online content specifically to enhance the relevance and accuracy of search responses.'
Cloudflare's 'Block AI Scrapers' managed rule has ClaudeBot in its allowlist. Anthropic bots are caught by default on most Cloudflare plans, blocking the fleet at the edge regardless of robots.txt.
/robots.txt returns with a text/plain Content-Type rather than an HTML 404 page. Sites that route /robots.txt through their SPA shell return text/html, which RFC 9309 parsers reject - making the file effectively absent to ClaudeBot.
Critical content appears in the initial server-rendered HTML. Anthropic does not publish a JS-render commitment for any of the three bots; SSR is the safe path.
Deprecated `anthropic-ai` and `Claude-Web` directives removed from robots.txt. Their presence is a no-op block (Anthropic no longer references these UAs) and signals stale config to maintainers.
robots.txt includes a Sitemap: line pointing to an XML sitemap. Claude-SearchBot indexes URLs for Claude's search tool; an explicit sitemap gives it a canonical inventory instead of relying on link discovery.
Paste your homepage URL. Sentry verifies robots.txt rules for all three Anthropic bots, Cloudflare AI Audit posture, IP-range fidelity, and legacy-rule cleanliness, then ships a per-rule report. No signup, instant results, always free.
Sentry fetches your site, runs every CLAUDEBOT rule, and renders the full result page before your next sip of coffee.
Each failed rule ships with a prescription paragraph. Hand it to engineering and the gap is closed before lunch.
Add your site to the daily Sentry sweep with one click. New regressions get caught the next morning.
8 rules in the CLAUDEBOT Sentry. Daily 3:30 AM ET sweep.
One brain. Thirty-six pairs of eyes. Sentries monitor every visibility signal that decides whether search engines, AI engines, and ad platforms show you. Cortex reads what they see, weighs it against a unified corpus of platform documentation, and acts. Every move follows a defined decision protocol: action stated, reason given, impact named.