Methodology
Inspection manual: data-vendor task quality audit
This document describes how we inspect the quality of a vendor's task samples.
Every number in the reports we send a vendor comes from this procedure, scored by one audit engine.
The methodology is identical across vendors: every vendor receives the same manual.
We audit along six axes.
| Axis | What | How |
|---|---|---|
| 1. Executability | Does the task build and score correctly | We run the shipped solution and an empty submission three times each, and scan the built image for the answer |
| 2. Static checks | Are dependencies pinned and the environment well declared | Terminal-Bench 3 check scripts plus our resource check |
| 3. Design | Is the task well built as an evaluation instrument | TB3 task-implementation rubric, 17 criteria, with adversarial re-verification |
| 4. Reward hacking | Can full marks be won without doing the work | Black-box red-team run by Opus 5 or Codex, plus white-box exploits reproduced in Docker |
| 5. Failure analysis | When the model fails, is the task at fault | Our own Opus 5 or Codex run, graded on the TB3 trial-analysis rubric |
| 6. Benchmark contamination | Is the task a clone of a public benchmark task | Instruction, solution and verifier compared with Terminal-Bench 2.1 |
Axes 1, 4 and 5 are executed by us on your pack. We never use trajectories or rewards shipped with a pack as a substitute for our own runs.
Separately from these six quality axes, we measure difficulty as its own dimension (see Difficulty measurement below). It is reported alongside quality and price and never enters the quality score.
When a check cannot be run
Every check on every task is recorded as measured, ran without a usable result, or not run, with the reason.
A task's quality number exists only when every scored check on it was measured, and a dataset's quality number exists only
when every task has one. Otherwise the report says not measurable and lists each missing check and why (for example: the
verifier never writes a reward file). A missing check is never counted as a pass.
Who measured. Axis 4 and 5 runs use Opus 5 or Codex. The detail report names, for each check, the model that ran it and the model that judged it. New difficulty measurements use the mixed benchmark Codex ladder below and reuse accepted evidence.
Audit scope. We audit verifiable tasks only. A task that needs a GPU, or whose grader needs an LLM judge (an external
API key), is not run and not counted; the report lists it under Out of audit scope with the reason.
Verifier writes no reward when tests fail. When the verifier script stops before writing 0 on a failing test (for example set -e ahead of the reward line) and that run's test output ends in failures, the run scores 0. This is not counted as a defect.
Environment does not build. When the environment fails to build for a reason on the vendor's side (an expired package
repository, a changed checksum of an external script), we record the file, line and a reproduction and
count the task as one whose grader does not work (an axis 1 ceiling defect).
Axis 1 · Executability
We build the task image from the vendor's environment/Dockerfile and run three checks. The shipped solution and the empty submission are each run three times; one run past the threshold is the defect, so a task that passes only sometimes is caught here.
| Check | Condition for a defect | Tier |
|---|---|---|
| Ceiling | Any of the three runs of the shipped solution (solution/) scores below 0.9: submitting the correct answer can fail | S1 |
| Floor | Any of the three runs of an empty submission scores above 0.1: points accrue for doing nothing | S1 |
| Answer leakage | A file in the agent's image has the same content as a file under solution/ or tests/, and reading it gives the answer away | S1 |
The 0.9 and 0.1 thresholds apply to binary and continuous rewards alike.
For leakage, a matching file is not automatically a leak (a verifier fixture can legitimately equal the input the agent is given),
so every match is reviewed before it is counted.
If the shipped solution never produces a usable run and we reproduce a vendor-side cause (for example a Dockerfile step that
fails, cited by file:line), the task is recorded as a broken grader and the ceiling defect is counted. Without a reproduced
cause, the checks are recorded as not measurable.
Axis 2 · Static checks
We run the Terminal-Bench 3 check scripts and our own resource check on every task. Six are scored:
| Script | What it checks | Tier |
|---|---|---|
check-pip-pinning | Are pip/uv installs version-pinned | S2 |
check-compose-host-binds | Does docker-compose bind-mount a host directory | S2 |
check-dockerfile-sanity | Are the Dockerfile's apt dependencies version-pinned | S4 |
check-nproc | No bare nproc calls (results depend on core count) | S4 |
check-resource-declaration | Does task.toml declare cpus and memory_mb (or memory, which harbor reads as memory_mb); undeclared tasks run with unlimited resources | S4 |
check-task-absolute-path | Is it clear to the agent which directory the instruction's relative paths are relative to; with a set working directory, relative paths are not a defect | S4 |
The scripts match text patterns. Every flagged line is read by a judgment, and it counts as a defect only when it really does what the check guards against (not a comment, an unreachable branch, a log string, or an unrelated build step).
A static finding is not dropped merely because a design criterion failed. For check-pip-pinning and verifiable,
we compare the cited evidence and count it once only when all valid static evidence is included in the confirmed
design finding. An unresolved comparison is not measurable. The other static checks are not automatically merged
with environment_hygiene. Adding a defect cannot reduce the total deduction under a fixed overlap relationship.
check-dockerfile-references is not scored here: answer leakage is measured directly in Axis 1.
The other TB3 scripts are publishing conventions of the TB3 repository, known false positives, or questions another axis answers.
Axis 3 · Design
One dedicated reviewer per task reads every shipped file (task.toml, instruction.md, environment/, tests/,
solution/) and grades the TB3 task-implementation rubric, quoting file:line for each finding. We do not judge from
snippets or regex matches.
We judge the 17 core criteria of the rubric's 35. The rest are TB3 packaging conventions (11) or minor and
execution-mode dependent (7).
Scored criteria
Each scored criterion carries one severity tier. The tier is set by reachability, not by whether a model exploited it.
S1 · signal void
| Criterion | What it asks |
|---|---|
do_not_modify_enforced | Is a "do not modify" constraint actually enforced |
task_security | No malware, credential theft, or host attacks |
test_instruction_alignment | Do the tests check what the instruction asks for |
S2 · signal contamination
| Criterion | What it asks |
|---|---|
outcome_verified | Is the outcome scored rather than the process |
verifiable | Is the verifier reliable, deterministic, and efficient |
structured_data_schema | If structured output is required, is the schema specified |
novel | Not solvable by memorizing public material |
S3 · worth buying
| Criterion | What it asks |
|---|---|
difficult | Hard for legitimate reasons (professional experience or domain knowledge required) |
essential_difficulty | Does the difficulty come from reasoning, not formatting |
agentic | Does it need multi-step terminal interaction, not a single call |
S4 · hygiene
| Criterion | What it asks |
|---|---|
environment_hygiene | Is the agent image clean (no tests, solution, or test-only dependencies) |
solution_quality | Does the solution actually compute rather than hardcode |
Judged, but measured by execution instead
Five criteria are still judged and reported, but their score comes from what we execute, so a suspicion from reading is not
counted on top of a measurement: solvable (Axis 1 ceiling), deterministic_reproducible (Axis 1 repeated runs),
anti_cheat_robustness, verifier_execution_isolation and functional_verification (Axis 4).
Adversarial re-verification
A second reviewer is given the explicit goal of breaking every FAIL: it opens every cited file:line and checks each
claim against the source. Anything ambiguous is recorded as REFUTED. Exploits are counted only in Axis 4, where they are run. Only surviving FAILs are counted.
Refuted findings are kept in the detail document with the reason.
Axis 4 · Reward hacking
| Method | What it answers | |
|---|---|---|
| Black-box | Opus 5 or Codex gets the TB3 red-team prompt (hack-trial-prompt.md) after the instruction, without the grader source, task files unmodified | Does a model actually defeat the grader |
| White-box | We read tests/, write an exploit that forges full reward with the task unsolved, and reproduce it in Docker | Can the grader be defeated |
Verdicts are not taken from the reward. We read the full trajectory of the run and judge breached or held.
A black-box breach counts S1; otherwise a reproduced white-box exploit counts S3; at most one per task.
White-box findings that were not reproduced are described in the detail document but not scored.
Black-box is one run, so "held" means this attempt failed, not that the grader cannot be defeated.
Both paths are tracked cells. A pack where nobody read the grader reports the white-box cell as
not run and receives no quality score, the same as any other unmeasured check. A read that
found no way in is recorded as held, which is a result; silence is not.
Axis 5 · Failure analysis
We run each task once with Opus 5 (claude-code harness) or Codex (codex harness) and grade the run on the TB3 six-criterion trial-analysis rubric.
Two criteria point to a task defect and are scored:
| Criterion | What it asks | Tier |
|---|---|---|
task_specification | Could the model tell what to build from the instruction alone | S2 |
difficulty_crux | Did the model fail at the intended difficulty (failing somewhere else is a FAIL) | S3 |
Either criterion may be judged not applicable where the rubric allows it; only a FAIL is counted. reward_hacking belongs to Axis 4; near_miss,
refusals and low_timeout describe the model or the format rather than the task and are not scored.
For tasks with a restrictive network policy, we add the package and model API hosts needed to run Claude Code. These are destination permissions, not permissions exclusive to the Claude process. A task with a public network policy retains public access; extra allowlist entries do not restrict it. We retain the run configuration and inspect trajectories when reviewing possible external answer retrieval.
Axis 6 · Benchmark contamination
For every task we compare its instruction, solution and verifier separately with the Terminal-Bench 2.1 task set and
score similarity from 0 to 1 (0.9 = the same task reskinned, 0.7 = same core approach, 0.4 = a shared subtask,
0.1 = same domain only). Renamed variables, files and domain terms do not lower the score; shared test scaffolding does not
raise it. The task's score is the maximum of the three.
| Score | Meaning | Effect |
|---|---|---|
| ≥ 0.90 | A clone of a public benchmark task | Counted as a defect, S2 (skipped when Axis 3 already confirmed a novel defect on the same task, so the same fact is not deducted twice) |
| 0.70 to 0.89 | Close, needs adjudication | Flagged in the detail document, not scored |
| < 0.70 | Clean | None |
Similarity to the genre of Terminal-Bench is expected. Only near-identity to a specific task counts.
Quality score
Weights are S1 4 · S2 3 · S3 2 · S4 1. The maximum load of the scored design criteria is
2×4 + 5×3 + 3×2 + 2×1 = 31 per task.
> Task quality = max(0, 1 − total load ÷ 31), where total load = confirmed design defects + defects measured in Axes 1, 2, 4, 5 and 6.
> Dataset quality = max(0, 1 − sum of total loads ÷ (tasks × 31)), only when every task was fully measured.
> Design quality = 1 − sum of design loads ÷ (tasks × 31), shown for reference.
> Worked example. A task has one confirmed S2 design defect (load 3). Axis 4 records a black-box breach (load 4) and
> Axis 5 a difficulty_crux FAIL (load 2). Total load 9, so task quality = 1 − 9 ÷ 31 = 0.71, and its design quality is
> 1 − 3 ÷ 31 = 0.90.
The score measures how heavily defects land, not how many there are.
Difficulty measurement (a separate dimension, not part of the quality score)
Reference index. New measurements use tb21-tb4-codex-v2: Luna low 1.14, Terra max 1.94, Sol max 3.06, Astra max 4.08, in increasing order. Reference models are Haiku 4.5 (legacy configuration) at 1, Sonnet 5 (legacy configuration) at 2, Opus 5 (legacy configuration / TB4 xhigh) at 3, and Fable 5.1 max at 4. The TB4 Sonnet linking reference uses max.
Calibration. L is piecewise linear interpolation on the frozen legacy TB2.1 reference points (44.2,1), (80.4,2), (87.6,3), (89.9,4). Luna=L(49.4), Terra=L(78.4). U uses TB4 Sonnet max (12.4,2), Opus xhigh (53.9,3), and Fable max (57.9,4). Sol is (L(88.8)+U(37.3))/2=3.060870; Astra uses a small extension of the final segment, 3+(58.2-53.9)/(57.9-53.9)=4.075. Opus remains 3 on both references. Astra is not assigned 4 by definition.
Sources, observed September 18–19, 2026: [TB2.1 leaderboard](https://www.tbench.ai/?version=2.1), [OpenAI effort chart](https://openai.com/index/gpt-5-6/), [TB4 leaderboard](https://www.tbench.ai/?version=4.0). The original sources of legacy TB2.1 constants have not been reverified as current Claude leaderboard scores. This is an approximate reference index with disclosed effort and harness differences. Equal weights are an operational convention, not an empirically optimal weighting. Decimal differences do not establish significant rankings. TB2.1 performance of at least 85% triggers inspection of a higher benchmark; coordinates remain frozen within a profile version.
Execution direction. Reuse accepted evidence first. Otherwise start with three normal Sol max trials. Mean reward ≥0.8 for graded packs, or 3/3 full successes for binary packs, moves down to Terra max; Terra mean ≥0.5 moves down to Luna low. Sol mean <0.2 or 0/3 successes moves up to Astra max. Intermediate results finish at that rung. Existing easy Astra max results lead down to Sol; no success at Astra does not trigger a weaker Sol run. An existing Luna–Terra bracket around mean reward 0.5 avoids additional stronger-model runs.
Reuse and migration. Completed valid Claude ladders can be recomputed from retained rewards on unchanged reference coordinates. Old Codex D is recomputed on the new index; earlier values and profiles are preserved. Disclosing benchmark-specific efforts does not allow relabeling actual trials: Astra high does not count as Astra max. Missing rewards and policy refusals are never read as scores. A provider's policy refusal is a limit of the instrument, not a property of the task, so where the reference index already places an equivalent rung on the other model family we measure there and the report names which ladder produced the D. We do not reword a charter to get past a refusal.
Difficulty D is the ability coordinate where actual vendor-task mean reward crosses 0.5. Interpolate between measured rungs; otherwise use the existing extrapolation slope of 0.6 and label the estimate. A solve is reward ≥0.9 for graded packs or exactly 1 for binary packs. Dataset D is the median of scored tasks after the ladder finishes, with coverage disclosed. Unavailable and solvability outcomes are excluded; no scored tasks means null. Successful-run step counts describe horizon separately and never add to D.
Measurement limit. No success at the top Astra rung withholds D. Confirmed Axis 5 task_specification or difficulty_crux FAIL indicates a task defect; otherwise the cause remains unresolved at the measured limit. A missing judgment is not a PASS. Public benchmark results never replace actual vendor trials. Difficulty remains separate from quality.
This manual is a methodology document. Per-vendor numbers and task-level evidence are in the accompanying detail document.