Always-on monitoring of ByteDance's crawler. Bytespider feeds TikTok recommendation training and Doubao's foundation model. ByteDance publishes no official docs and no IP ranges - third-party UA fingerprinting is the only verification path. Sentry catches accidental blocks and accidental consent. Cortex handles the fix.
Continuous audits of Bytespider's access to your site against the 6 things that matter for ByteDance's AI surfaces. Bytespider is the only crawler in our coverage without first-party documentation or a published IP list. Third-party fingerprinting is the only verification path. Sentry catches both visible and implicit decisions. Cortex fixes it.
robots.txt contains an explicit Allow or Disallow for User-agent: Bytespider. Default-wildcard absence is treated as consent by ByteDance per third-party crawler-log analysis. Make the decision intentional.
Cloudflare or Akamai bot management is not silently rejecting Bytespider's Singapore-origin requests when the publisher intends to allow it, nor accepting them when the publisher intends to block. The robots.txt directive and the WAF rule agree.
Bytespider's documented bursting pattern is not triggering blanket WAF rate-limits that catch other legitimate crawlers (Googlebot, Bingbot) in the cross-fire. WAF rules match on UA + IP, not raw connection count.
Site is reachable from Bytespider's Singapore-routed crawler infrastructure. No country-block or geo-WAF rule accidentally hiding the site from ByteDance.
The Bytespider directive matches the publisher's stated stance on ByteDance training (Doubao LLM + TikTok recommendation). Most publishers have never consciously made this decision; the rule should be intentional.
Since ByteDance publishes no IP range list, the team documents the heuristic used to identify Bytespider (UA token + spider-feedback@bytedance.com email signature + Singapore ASN observation). Future audits can re-apply consistently.
Paste your homepage URL. Sentry verifies the Bytespider directive, WAF posture, burst-pattern footprint, and geographic reachability from ByteDance's Singapore crawl infrastructure, then ships a per-rule report. No signup, instant results, always free.
Sentry fetches your site, runs every BYTESPIDER rule, and renders the full result page before your next sip of coffee.
Each failed rule ships with a prescription paragraph. Hand it to engineering and the gap is closed before lunch.
Add your site to the daily Sentry sweep with one click. New regressions get caught the next morning.
6 rules in the BYTESPIDER Sentry. Daily 3:30 AM ET sweep.
Stop Guessing. Start Seeing. Get Cortex.
Cortex is your marketing decision engine. It knows every best practice ever published across search and AI platforms - and every visibility signal these platforms use to rank your business. Cortex monitors your site against those signals in real time and automatically optimizes your site for them. The result - more conversions, lower cost.