Skip to the card

2026-11-20 issue

Observed arrival · 2026-11-20

AnswerIndex’s map of AI crawler policy

answerindex.org Visit website
Editorial interest 77/100 Selection signal · not a rating of the site

AnswerIndex publishes measurements of robots.txt rules for AI crawlers and adoption of llms.txt files across the Tranco top 10,000 websites.

Landing page captured for the 2026-11-20 issue.
For
Researchers tracking AI crawler policies
Worth noticing
The November sample covers 10,000 Tranco domains and 14 AI agents; its per-domain dataset is offered free with attribution.

Field notes

The homepage describes separate checks for robots.txt policies and the presence of llms.txt or llms-full.txt files, then links readers to the report and dataset. Its November comparison says llms.txt appeared on 1,123 sampled sites, against 556 in June 2026; those figures are reported by the project, not independently verified in the supplied extract.

Observed signals

Read the marks

Editorial observations of this landing page, not a rating.

○OpenPublic substance visible
✦PrettyNotable craft visible
◎NicheUnusually specific use

One card from the complete issue

The Woven Index

386,049 arrived 1,000 judged 972 catalogued Enter the complete issue
answerindex.org

Landing page observed 2026-11-20. The live site may have changed.