Observed arrival · 2026-11-20
AnswerIndex’s map of AI crawler policy
AnswerIndex publishes measurements of robots.txt rules for AI crawlers and adoption of llms.txt files across the Tranco top 10,000 websites.
○Open
⊠Login
$Paid
†Ads
✦Pretty
●Pro
◎Niche
◉Human
⚑Risk
ƒJS
- For
- Researchers tracking AI crawler policies
- Worth noticing
- The November sample covers 10,000 Tranco domains and 14 AI agents; its per-domain dataset is offered free with attribution.
Field notes
The homepage describes separate checks for robots.txt policies and the presence of llms.txt or llms-full.txt files, then links readers to the report and dataset. Its November comparison says llms.txt appeared on 1,123 sampled sites, against 556 in June 2026; those figures are reported by the project, not independently verified in the supplied extract.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
○OpenPublic substance visible
✦PrettyNotable craft visible
◎NicheUnusually specific use
One card from the complete issue