Observed arrival · 2026-10-24
The AI Crawler Cache Study
An open research project measuring whether AI search and research crawlers revalidate pages before downloading them again.
○Open
⊠Login
$Paid
†Ads
✦Pretty
●Pro
◎Niche
◉Human
⚑Risk
ƒJS
- For
- AI crawler operators and website owners
- Worth noticing
- It excludes repeat visits unless the crawler previously received a page fingerprint, addressing a source of inflated failure counts.
Field notes
The project hosts stable pages on its own logging server and records requests for the stated research purpose. Its classification rule limits analysis to return visits where a crawler had already received a page fingerprint; the overview says it reports three measures per crawler and links separate method, metrics, and limitations documents. The study says it describes behavior by crawler family rather than individual request.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
○OpenPublic substance visible
✦PrettyNotable craft visible
◎NicheUnusually specific use
One card from the complete issue