Observed arrival · 2026-10-10
Plumbline Grader audits code-task graders
A commercial audit service tests code-training environments for cases where a grader awards full reward to an incorrect fix, then supplies replacement tests.
- For
- Teams running code-reward training environments
- Worth noticing
- The homepage links a v0.2.0 method specification, v1.0 verification criteria, an independence policy, and a reproduction kit.
Field notes
The workflow separates finding a grader loophole from proving that it matters: a behavioral check must accept the reference fix while rejecting both the unfixed code and the wrong fix. The page says reruns use fresh containers, with network access disabled and Linux capabilities dropped. Linked materials include a method specification, verification criteria, an independence policy, and a reproduction kit; set-by-set public results are planned after maintainers’ notice periods.
Observed signals
Read the marks
Editorial observations of this landing page, not a rating.
One card from the complete issue