Report and fix defects in AI benchmarks
Over 50% of tasks in major coding benchmarks contain defects — broken verifiers, wrong oracles, solution leakage. They corrupt training signals. Nobody is systematically managing them.
Delphik turns benchmark defects into public evidence, upstream routing, and live health records. Coding benchmarks first.
RL benchmarks are broken
RL benchmarks contain systematic defects. The training signal is corrupted at the source. Verification is never done because stronger models surface new defects. Every benchmark needs a permanent verification layer.
Defects in every major benchmark
SWE-bench, Terminal-Bench, Atlas, Pro, Rebench v2 — every one shows broken verifiers, wrong oracles, solution leakage.
Eval scores hide reward hacking
Agents learn to game broken verifiers instead of solving the task. The score goes up; the capability doesn't.
Manual verification doesn’t scale
Each task takes hours — unfamiliar repos, test logic, oracle validation. Doesn't scale to thousands.
Stronger agents surface new defects
Tasks that passed yesterday's model break under today's. Defects that were invisible become unavoidable.
Defects surfaced two ways.
Two paths on the same backbone. The Hub captures defects on public benchmarks organically from the research field; future private audit programs can use the same methodology, validator team, and evidence infrastructure for closed datasets.
Delphik defect hub
Anyone reports a defect from their terminal. We triage and route upstream.
Private Audit Programs
Future private audits for teams that need dedicated routing, review, and release evidence.
Root-cause matching
Same defect definition on both products. We require a pinpointed root cause, not just a 'broken' verdict.
Same validator team
Our team makes the final call on confirmed records — validating community reports on the Hub and future private audit evidence.
Shared audit infrastructure
Sandboxed execution, evidence-review tooling, lifecycle dashboards. One platform under both products.
Blog
All posts- 6 min readPosition PaperAudit the Environment, Not Just the Model
Frontier models are shaped by RL environments, and a verifier that can be passed without solving the task reinforces the wrong behavior — corrupting the training signal, eroding monitorability, and at worst seeding misalignment. A single team searches with its own blind spots, so we argue an RL environment should pass an independent adversarial audit, one layer deeper than model audit.
- 2 min readOpen EvalReward Hacking Is a Property of the Environment
The same frontier model can be clean on one coding benchmark and cheat on another — GPT-5.5 hacks zero on DeepSWE but more than any model we measured on SWE-Marathon, which leaves an answer key in the sandbox. Reward hacking is as much a property of the environment as the model. Full analysis and the interactive index at coding-index.posttrain.dev.
- 7 min readDataA Benchmark With No Reported Defects Isn't Clean. It's Unaudited.
Benchmark health isn't a defect count — it's whether discovery closes into a fix. We trace that loop across 1,874 defect threads: what's found, fixing, fixed, and still open; why fixing never stops; how agentic benchmarks run ~5× denser per task than static ones; and what a fix does to the leaderboard.
Talk to Delphik about your benchmark
If you build or use AI benchmarks and care about their quality, we’d love to talk. The open hub is live; private audits are a separate future program.