Observed arrival · 2026-10-13
BotCanary checks the crawl path, not just robots.txt
A free checker tests whether search and AI crawlers can reach a site, including through its CDN, firewall, and robots.txt rules.
○Open
⊠Login
$Paid
†Ads
✦Pretty
●Pro
◎Niche
◉Human
⚑Risk
ƒJS
- For
- Site owners and SEO teams checking crawler access
- Worth noticing
- The page says its checks use crawler user-agents from its own servers, not real Googlebot traffic, and flags cases that need confirmation elsewhere.
Field notes
The checker compares crawler-specific robots.txt rules with responses from CDNs, WAFs, and security plugins, and can test from several countries. Its crawler directory and guides link to focused material on topics such as Cloudflare Bot Fight Mode and distinguishing verified Googlebot from lookalikes. The page distinguishes its own user-agent checks from real Googlebot traffic and says some findings require confirmation from the CDN, Search Console, or logs.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
○OpenPublic substance visible
$PaidCommerce or pricing visible
✦PrettyNotable craft visible
●ProPolished or operationally mature
◎NicheUnusually specific use
ƒJavaScriptBrowser-side code central
One card from the complete issue