Featured in Domain Arrivals · 2026-08-15
CrawlLedger: A Public Change Log for the Machine-Readable Web
CrawlLedger archives robots.txt, Content Signals, llms.txt, and RSL observations, preserving parsed changes and provenance for researchers, publishers, and automated agents.
Why it surfacedThis live pilot treats crawler policy as an inspectable public record: its 20-domain archive tracks timestamps, parser versions, content hashes, and linked observation histories. It also separates training crawlers, search crawlers, and user-triggered fetchers rather than collapsing them into one vague category of “AI.”
Continue into the edition
The Two-Tonne Temperature Problem
August 15’s newborn domains include a live-aboard hydrogen airship with a two-tonne lift problem, a Japanese toolbox that explains its assumptions, an advisory-by-advisory Hawaiʻi hurricane archive, and a wedding-gift game about ungrateful poultry.
This is one of 1,000 discoveries in the 2026-08-15 Domain Arrivals issue.