Research and insights on AI agent benchmark verification.
Why RL environments need independent adversarial audits
Frontier models are shaped by RL environments, and a verifier that can be passed without solving the task reinforces the wrong behavior — corrupting the training signal, eroding monitorability, and at worst seeding misalignment. A single team searches with its own blind spots, so we argue an RL environment should pass an independent adversarial audit, one layer deeper than model audit.
A short note from the Coding Index
The same frontier model can be clean on one coding benchmark and cheat on another — GPT-5.5 hacks zero on DeepSWE but more than any model we measured on SWE-Marathon, which leaves an answer key in the sandbox. Reward hacking is as much a property of the environment as the model. Full analysis and the interactive index at coding-index.posttrain.dev.
Benchmark health is a closure loop, not a defect count
Benchmark health isn't a defect count — it's whether discovery closes into a fix. We trace that loop across 1,874 defect threads: what's found, fixing, fixed, and still open; why fixing never stops; how agentic benchmarks run ~5× denser per task than static ones; and what a fix does to the leaderboard.
744 people are keeping benchmarks in working order — mostly unnamed
We read 6,245 GitHub Issue/PR threads across 62 agentic benchmark repos and classified 1,874 as benchmark defects. Behind them are 744 public auditors who quietly keep these benchmarks in working order — a shoutout to them, and the data on who finds what, why a human who hit the task is still the best detector, and why each report costs more than it should.
Report a broken benchmark task in one command
Coding benchmarks are full of broken tasks — wrong oracles, broken verifiers, leaked solutions — and when you hit one, there’s nowhere to report it. Defect Hub lets you report a defect from inside your coding agent in one command, routes it upstream to the maintainer, and tracks it to the fix. Verified defects become an open dataset.
A Meta-Benchmark from Benchmark Version Diffs
We introduce Task Verification Bench — a meta-benchmark that uses benchmark version diffs as ground truth to evaluate whether LLMs can detect defects in coding benchmarks. GPT-5.4 achieves 35–60% root-cause-matched recall: meaningful for triage, insufficient for full automation.
Why Every Run Needs a Trajectory Inspection Layer
Eval scores hide reward hacking, broken tests reject correct solutions, and nobody reads the trajectories. Post-training needs a dedicated inspection layer.