Observed arrival · 2026-11-10

The Crawler Index

agentreadycrawlers.com Visit website

Editorial interest 83/100 Selection signal · not a rating of the site

An index of 70 AI crawlers and agents, with their user-agent tokens, purposes, verification methods, and measured robots.txt block rates.

Landing page captured for the 2026-11-10 issue.

For: Site operators tracking AI crawler access
Worth noticing: It distinguishes explicit token blocks from wildcard robots.txt refusals, rather than treating every refusal as deliberate.

Field notes

Each crawler entry pairs its exact user-agent token with a stated purpose, robots.txt behavior, and verification method. The report says 5,938 of the top 10,000 sites had a parsed robots.txt; it also separates sites naming a token from those caught by a wildcard rule. The page dates this crawl to November 4, 2026, and labels its own crawler token as a control.

Observed signals

Read the marks

Editorial observations of this landing page, not a rating.

○OpenPublic substance visible

✦PrettyNotable craft visible

●ProPolished or operationally mature

◎NicheUnusually specific use

One card from the complete issue

Detergent Lessons in Sheep Suits

328,969 arrived 1,000 judged 984 catalogued Enter the complete issue