Domain Arrivals
You’re viewing the 2026-08-10 issue, not the current issue. Go to the current issue: 2026-08-14

Featured in Domain Arrivals · 2026-08-10

CrawlIndex Measures the Web’s AI Policy Gap

crawlindex.org ↗ · Developer Corner · score 89.0
○Open: public page gives enough substance to understand the site $Paid: pricing, booking, ecommerce, subscription, or paid access is visible ADAds: visible ad load or ad-supported content ✦Pretty: notable design, copy, craft, or presentation ◆Pro: polished, finished-feeling, serious, or operationally mature ◇Niche: specific audience, workflow, or unusually focused use case ◉Human: personal, local, community, civic, handmade, or real-person signal !Suspicious: scammy, spammy, trust theater, finance fog, or credibility concern

An open index measuring how thousands of major websites treat AI crawlers, from robots.txt declarations to server-level behavior.

Why it surfaced

CrawlIndex recrawls its dataset nightly and found that 530 sites permit GPTBot in robots.txt while refusing it at the server. It publishes the methodology, dataset, crawler-by-crawler results, platform breakdowns, and a readiness score calculated without a model.

Read the complete issue Visit the observed site ↗

Continue into the edition

Ham Is Specifically Discouraged

August 10’s newborn domains include a mock cryptid council that specifically discourages ham, a vintage phone collecting messages for Twyla’s 70th, a 4.5-meter dish measuring Galactic hydrogen at 1420 MHz, and an open index that found 530 sites permitting GPTBot in robots.txt while refusing it at the server.

This is one of 1,000 discoveries in the 2026-08-10 Domain Arrivals issue.

Explore the complete issue Browse Developer Corner

More from Developer Corner

Three more arrivals

Jorge Polanco’s Systems Portfolio jorgepolanco.dev

A backend and distributed-systems engineer presents tools and experiments for making invisible system behavior legible.

 Field, a Git-native command-line DAW field.one

Field turns music production into a repository of diffable files, scripted commands, and reproducible audio builds.

 Chimera, the Agent That Publishes Its Losses chimeraagent.space

An open-source terminal and desktop coding agent that plans, edits, tests, and can revert its own failed changes.