The Crawler Index
An index of 70 AI crawlers and agents, with their user-agent tokens, purposes, verification methods, and measured robots.txt block rates.
Issue 2026-09-20Search the complete card catalogue
Across every issue
Titles, domains, summaries, categories, signals, dates, and editorial-interest scores across the entire published catalogue.
Results
An index of 70 AI crawlers and agents, with their user-agent tokens, purposes, verification methods, and measured robots.txt block rates.
An Android app designed to call businesses for phone-only reservations, using the owner’s phone number and instructions they approve first.
A free browser game where players use clues and a logic grid to solve cases alongside Inspector Grane.
A dating application invites people to apply for a first interview with Amelia Larson, with approved applicants promised access to photos.
ArtTraceX reports on stolen, recovered, and returned artworks, as well as forgery and court disputes, alongside country clocks and an art-theft map.
A Windows app for DJs and VJs to map projector visuals onto club walls, booths, and other surfaces, with layers that can respond to the music.
A Mam-language digital library with a searchable dictionary, word games, grammar, cultural materials, and classroom guides.
A free beta for a third-person MOBA built around bird heroes, with Windows and macOS downloads.
A downloadable desktop companion that takes voice or typed requests and uses Claude Code to act on your Mac, alongside notch-based timers, music controls, and meeting information.
A free beta simulation lets you make choices in fictional lives built from historical and demographic data across 180 countries, from 1900 to today.
An interactive archive of bag charms and the memories, meanings, and everyday uses their owners attach to them.
CheckNoise estimates aircraft noise at a U.S. address, including changes tied to runway direction, and offers on-site measurement studies.
Enter an aircraft identifier, dates, time zone, and ground locations to find nearby flight passes and download their altitudes as a CSV.
A 1996-dated homepage presents Chopin’s Shadow as a gothic-synth band, with album notes, tour dates, and a guestbook.
Comunistas presents a sourced, visual account of Augusto Pinochet’s 1998 arrest in London and the extradition proceedings that followed.
A browsable reference for Kubernetes APIs and custom resources, organized by field, project, vendor, and release.
A free, open-source Windows fork of WSJT-CB for using FT8 on CB radio, with support for longer and event callsigns.
A free, phone-friendly set of tools for daily Buddhist practice, including a meditation timer, counters, prayers, a diary, and a practice-day calendar.
A searchable tracker of Discord client builds, experiments, server rollouts, and API changes.
A referenced knowledge platform on EEF1A2-associated neurological disorders, with a focused investigation of the D91N variant.
A catalog of traditional board games, grouped by family and linked to the places and histories associated with them.
Seth is a free, open-source Mac app for viewing medical DICOM images, including CT, MRI, X-ray, ultrasound, and mammography.
An illustrated reference to Hindu motifs, temples, colors, textiles and dance, alongside a library of patterns and other materials presented as free to use.
Hydris presents a glasses-and-phone system for treatment-plant alerts, maps, guided rounds, and inspection records.