Always-on monitoring of ByteDance's crawler. Bytespider feeds TikTok recommendation training and Doubao's foundation model. ByteDance publishes no official docs and no IP ranges - third-party UA fingerprinting is the only verification path. Sentry catches accidental blocks and accidental consent. Cortex handles the fix.
Continuous audits of Bytespider's access to your site against the 6 things that matter for ByteDance's AI surfaces. Bytespider is the only crawler in our coverage without first-party documentation or a published IP list. Third-party fingerprinting is the only verification path. Sentry catches both visible and implicit decisions. Cortex fixes it.
robots.txt contains an explicit Allow or Disallow for User-agent: Bytespider. Default-wildcard absence is treated as consent by ByteDance per third-party crawler-log analysis. Make the decision intentional.
Cloudflare or Akamai bot management is not silently rejecting Bytespider's Singapore-origin requests when the publisher intends to allow it, nor accepting them when the publisher intends to block. The robots.txt directive and the WAF rule agree.
Bytespider's documented bursting pattern is not triggering blanket WAF rate-limits that catch other legitimate crawlers (Googlebot, Bingbot) in the cross-fire. WAF rules match on UA + IP, not raw connection count.
Site is reachable from Bytespider's Singapore-routed crawler infrastructure. No country-block or geo-WAF rule accidentally hiding the site from ByteDance.
The Bytespider directive matches the publisher's stated stance on ByteDance training (Doubao LLM + TikTok recommendation). Most publishers have never consciously made this decision; the rule should be intentional.
Since ByteDance publishes no IP range list, the team documents the heuristic used to identify Bytespider (UA token + spider-feedback@bytedance.com email signature + Singapore ASN observation). Future audits can re-apply consistently.
Paste your homepage URL. Sentry verifies the Bytespider directive, WAF posture, burst-pattern footprint, and geographic reachability from ByteDance's Singapore crawl infrastructure, then ships a per-rule report. No signup, instant results, always free.
Sentry fetches your site, runs every BYTESPIDER rule, and renders the full result page before your next sip of coffee.
Each failed rule ships with a prescription paragraph. Hand it to engineering and the gap is closed before lunch.
Add your site to the daily Sentry sweep with one click. New regressions get caught the next morning.
6 rules in the BYTESPIDER Sentry. Daily 3:30 AM ET sweep.
One brain. Thirty-six pairs of eyes. Sentries monitor every visibility signal that decides whether search engines, AI engines, and ad platforms show you. Cortex reads what they see, weighs it against a unified corpus of platform documentation, and acts. Every move follows a defined decision protocol: action stated, reason given, impact named.