Observed arrival · 2026-11-01
Tallyglass-bench: three tries, one blind recount
A visual-counting benchmark tests frontier models on precise counting questions drawn from real photos.
○Open
⊠Login
$Paid
†Ads
✦Pretty
●Pro
◎Niche
◉Human
⚑Risk
ƒJS
- For
- Vision-model builders and evaluators
- Worth noticing
- Each question gets three model runs, then a second person recounts without seeing the first number.
Field notes
The task format uses questions with precise counting conditions, and the page says each is explained with what counted and what did not. Colored areas indicate the original object mask; run markings distinguish a model that missed most attempts from one that matched the answer. The displayed leaderboard is a preview, with placeholder scores pending full runs.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
○OpenPublic substance visible
✦PrettyNotable craft visible
◎NicheUnusually specific use
One card from the complete issue