LocoMusa: A Benchmark for Thinking With Local LLMs

Field notes

LocoMusa evaluates local models through blind pairwise comparisons rather than a single intelligence score. Its listed task areas cover ideation, reframing, steelmanning, and multi-turn dialogue, while the published ratings use a Bradley–Terry model normalised against frontier anchors averaging 100. The linked repository contains the tasks, evaluation harness, judge, and methodology; the homepage also points to a JSON leaderboard.