Observed arrival · 2026-10-31
Open Benchmark Review: an asterisk beside the leaderboard
An independent project reviews AI benchmarks and publishes evidence about what their scores measure and where their grading may fail.
○Open
⊠Login
$Paid
†Ads
✦Pretty
●Pro
◎Niche
◉Human
⚑Risk
ƒJS
- For
- AI benchmark readers, researchers, and benchmark developers
- Worth noticing
- Reviews apply Epoch AI’s published rubric and say each counted error points to a file and line, reproduction, or log.
Field notes
The review process described on the homepage covers benchmark tasks, grading code and, when released, agents’ runs. The project says it applies Epoch AI’s published rubric, attaches evidence to counted errors and ships scripts for reproducing its counts. It also describes a right of reply for benchmark developers and says a repaired benchmark can receive another review. The homepage reports six reviews to date and characterizes five as flawed.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
○OpenPublic substance visible
✦PrettyNotable craft visible
●ProPolished or operationally mature
One card from the complete issue