Observed arrival · 2026-09-10

The robots.txt file cabinet for AI crawlers

bot-policy.com Observed source

Editorial interest 78/100 Selection signal · not a rating of the site

Bot Policy looks up and compares what websites tell GPTBot, ClaudeBot, GrokBot, and other AI crawlers.

Landing page captured for the 2026-09-10 issue.

Field notes

The service limits collection to three policy-file paths on a target host: robots.txt, llms.txt, and llms-full.txt. Its dictionary covers named crawlers such as GPTBot, ClaudeBot, GrokBot, PerplexityBot, Google-Extended, Applebot-Extended, and CCBot, while also retaining the robots.txt wildcard. Public API endpoints are documented under /v1/public/; history and comparison require an API key.

Observed signals

Read the marks

Editorial observations of this landing page, not a rating.

○OpenPublic substance visible

✦PrettyNotable craft visible

●ProPolished or operationally mature

◎NicheUnusually specific use

One card from the complete issue

All Approaches to Piccadilly

405,063 arrived 1,000 judged 997 catalogued Enter the complete issue