Spatula’s rules-first product crawler

Field notes

The crawler checks robots.txt before collecting and stops fetching when that file is unreachable, returns a server error, or returns 429. It waits at least 15 seconds between requests to a site, lengthens waits when Crawl-delay is set, and backs off after refusals. The page also describes signed requests and an explicit opt-out route; it names Crowdio as the current destination and says price collection is planned for later.