Observed arrival · 2026-09-30
rastrolog: who crawls, and who sends visitors
A browser-based checker for AI crawlers allowed by a site’s robots.txt, paired with an analyzer for AI referrals in server logs.
○Open
⊠Login
$Paid
†Ads
✦Pretty
●Pro
◎Niche
◉Human
⚑Risk
ƒJS
- For
- Site owners tracking AI crawlers and referrals
- Worth noticing
- Its crawler list exposes 28 tokens; the page says 27 come from vendor documentation and links the source data as signals.json.
Field notes
Visitors can check both robots.txt and llms.txt, then inspect request logs in nginx, Apache, CloudFront, or AWS ALB formats; the page says parsing happens in the browser and files are not uploaded. The project also documents a Python 3.10+ command-line parser with FastAPI and Django middleware, plus a sub-2-KB script for sending an ai_referral event to several analytics platforms.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
○OpenPublic substance visible
✦PrettyNotable craft visible
●ProPolished or operationally mature
◎NicheUnusually specific use
One card from the complete issue