Observed arrival · 2026-09-07
CrawlState audits what websites tell AI crawlers
CrawlState checks a domain’s robots.txt and live responses to show whether its stated AI-crawler policy matches what the site actually serves.
○Open
⊠Login
$Paid
†Ads
✦Pretty
●Pro
◎Niche
◉Human
⚑Risk
ƒJS
Field notes
CrawlState compares robots.txt and related policy declarations with responses gathered by three named AI crawlers. Its dated record covers 1,163,843 domains, including 899,525 that answered and 264,318 that did not. The site reports that 76.3% of 63,146 domains declaring an AI-crawler restriction still served all three crawlers, and it exposes the record through a searchable lookup and an API-oriented developer offering.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
○OpenPublic substance visible
$PaidCommerce or pricing visible
✦PrettyNotable craft visible
●ProPolished or operationally mature
◎NicheUnusually specific use
ƒJavaScriptBrowser-side code central
One card from the complete issue