Observed arrival · 2026-11-11
LongTail-SWE benchmarks coding agents in 13 less-represented languages
A public benchmark of 202 real-world software engineering tasks drawn from GitHub issues and pull requests across 13 programming languages.
○Open
⊠Login
$Paid
†Ads
✦Pretty
●Pro
◎Niche
◉Human
⚑Risk
ƒJS
- For
- Researchers evaluating coding agents on less-benchmarked languages
- Worth noticing
- Each task has its own Docker image; the page reports results for nine model configurations and three attempts per task.
Field notes
The visible workflow begins with merged pull requests linked to a single issue, then checks whether projects install and tests run cleanly. A solver and three judges contribute to review before a repair step addresses task descriptions and tests. The leaderboard reports Pass@1, Pass@3, and All-3, alongside average tokens and steps; language-level counts range from 12 to 27 tasks.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
○OpenPublic substance visible
✦PrettyNotable craft visible
●ProPolished or operationally mature
◎NicheUnusually specific use
One card from the complete issue