Inspection manual: data-vendor task quality audit

This document describes how we inspect the quality of a vendor's task samples.

Every number in the reports we send a vendor comes from this procedure, scored by one audit engine.

The methodology is identical across vendors: every vendor receives the same manual.

We audit along six axes.

Axes 1, 4 and 5 are executed by us on your pack. We never use trajectories or rewards shipped with a pack as a substitute for our own runs.

Separately from these six quality axes, we measure difficulty as its own dimension (see Difficulty measurement below). It is reported alongside quality and price and never enters the quality score.

When a check cannot be run

Every check on every task is recorded as measured, ran without a usable result, or not run, with the reason.

A task's quality number exists only when every scored check on it was measured, and a dataset's quality number exists only

when every task has one. Otherwise the report says not measurable and lists each missing check and why (for example: the

verifier never writes a reward file). A missing check is never counted as a pass.

Who measured. Axis 4 and 5 runs use Opus 5 or Codex. The detail report names, for each check, the model that ran it and the model that judged it. New difficulty measurements use the mixed benchmark Codex ladder below and reuse accepted evidence.

Audit scope. We audit verifiable tasks only. A task that needs a GPU, or whose grader needs an LLM judge (an external

API key), is not run and not counted; the report lists it under Out of audit scope with the reason.

Verifier writes no reward when tests fail. When the verifier script stops before writing 0 on a failing test (for example set -e ahead of the reward line) and that run's test output ends in failures, the run scores 0. This is not counted as a defect.

Cause not established. When a check fails in a way that could come from our own harness rather than the task (for

example a process that crashes on every run, the reference solution included), we do not charge it: the check is recorded

as ran without a usable result, with the reason, until the cause is found.

Environment does not build. When the environment fails to build for a reason on the vendor's side (an expired package

repository, a changed checksum of an external script), we record the file, line and a reproduction and

count the task as one whose grader does not work (an axis 1 ceiling defect).

Axis 1 · Executability

We build the task image from the vendor's environment/Dockerfile and run three checks. The shipped solution and the empty submission are each run three times; one run past the threshold is the defect, so a task that passes only sometimes is caught here.

The 0.9 and 0.1 thresholds apply to binary and continuous rewards alike.

For leakage we look in three places: files in the built image with the same content as a file under solution/ or tests/;

files left where working files land, outside the tree the task hands the agent; and the vendor's own environment/ build

context compared with the answer from source (whole copied files, copied code, hidden constants, grader logs).

A candidate is not automatically a leak (a verifier fixture can legitimately equal the input the agent is given), so every

candidate is read before it is counted, and finding no candidate does not count as clean until the image has been read.

If the task does not run now and we reproduce a vendor-side cause (for example a Dockerfile step that fails, cited by

file:line), the task is recorded as a broken grader and the ceiling defect is counted. "Does not run now" means the shipped

solution never produced three usable runs, or our most recent run of the shipped solution or the empty submission produced

none; runs that passed before the cause appeared (for example before a pinned package was withdrawn) do not keep the task open. Without a reproduced

cause, the checks are recorded as not measurable.

Axis 2 · Static checks

We run the Terminal-Bench 3 check scripts and our own resource check on every task. Six are scored:

The scripts match text patterns. Every flagged line is read by a judgment, and it counts as a defect only when it really does what the check guards against (not a comment, an unreachable branch, a log string, or an unrelated build step).

A static finding is not dropped merely because a design criterion failed. For check-pip-pinning and verifiable,

we compare the cited evidence and count it once only when all valid static evidence is included in the confirmed

design finding. An unresolved comparison is not measurable. The other static checks are not automatically merged

with environment_hygiene. Adding a defect cannot reduce the total deduction under a fixed overlap relationship.

check-dockerfile-references is not scored here: answer leakage is measured directly in Axis 1.

The other TB3 scripts are publishing conventions of the TB3 repository, known false positives, or questions another axis answers.

Axis 3 · Design

One dedicated reviewer per task reads every shipped file (task.toml, instruction.md, environment/, tests/,

solution/) and grades the TB3 task-implementation rubric, quoting file:line for each finding. We do not judge from

snippets or regex matches. A passing criterion carries its written reason too; a task whose verdicts lack a reason is recorded

as not measurable rather than as clean.

We judge the 17 core criteria of the rubric's 35. The rest are TB3 packaging conventions (11) or minor and

execution-mode dependent (7).

Scored criteria

Each scored criterion carries one severity tier. The tier is set by reachability, not by whether a model exploited it.

S1 · signal void

S2 · signal contamination

S3 · worth buying

S4 · hygiene

Judged, but measured by execution instead

Five criteria are still judged and reported, but their score comes from what we execute, so a suspicion from reading is not

counted on top of a measurement: solvable (Axis 1 ceiling), deterministic_reproducible (Axis 1 repeated runs),

anti_cheat_robustness, verifier_execution_isolation and functional_verification (Axis 4).

Adversarial re-verification

A second reviewer is given the explicit goal of breaking every FAIL: it opens every cited file:line and checks each

claim against the source. Anything ambiguous is recorded as REFUTED. Exploits are counted only in Axis 4, where they are run. Only surviving FAILs are counted.

Refuted findings are kept in the detail document with the reason.

Axis 4 · Reward hacking

Verdicts are not taken from the reward. We read the full trajectory of the run and judge breached or held.

If the plain, non-adversarial run (the Axis 5 run, given no red-team prompt) already earns full reward without doing the

work, the grader is rewarding undone work with no attacker required: that is the most severe case, counted S0. Otherwise

a black-box breach counts S1; otherwise a reproduced white-box exploit counts S3; at most one per task.

White-box findings that were not reproduced are described in the detail document but not scored.

Black-box is one run, so "held" means this attempt failed, not that the grader cannot be defeated.

A run in which the model declines the red-team instruction (a provider safety refusal, or solving the task without trying

any bypass) did not test the grader: it is recorded as not run, never as "held".

A run that continued on another model after a refusal counts as a run of that model.

A breach by an earlier red-team run still counts after the red-team prompt changes; an earlier "held" does not.

A plain run that earns reward by reading an answer left in the image (for example a compiled reference module) is

reward hacking (S0), and that task has no difficulty value.

When Axis 3 do_not_modify_enforced fails on a task that Axis 4 already charges, we judge whether it is the same route;

the same route is charged once, in Axis 4.

Both paths are tracked cells. A pack where nobody read the grader reports the white-box cell as

not run and receives no quality score, the same as any other unmeasured check. A read that

found no way in is recorded as held, which is a result; silence is not.

Axis 5 · Failure analysis

We run each task once with Opus 5 (claude-code harness) or Codex (codex harness) and grade the run on the TB3 six-criterion trial-analysis rubric.

Two criteria point to a task defect and are scored:

Either criterion may be judged not applicable where the rubric allows it; only a FAIL is counted. A run that earned full

reward cannot fail task_specification (tests that check more than the instruction says are scored under Axis 3

test_instruction_alignment), and a run judged as reward hacking is charged once, in Axis 4, not again here. When task.toml has no author difficulty_explanation, we infer the intended difficulty from the instruction, tests and solution, state it in the finding, and judge against it; the missing explanation is reported as a recommendation and is not scored. reward_hacking belongs to Axis 4; near_miss,

refusals and low_timeout describe the model or the format rather than the task and are not scored.

For tasks with a restrictive network policy, we add the package and model API hosts needed to run Claude Code. These are destination permissions, not permissions exclusive to the Claude process. A task with a public network policy retains public access; extra allowlist entries do not restrict it. We retain the run configuration and inspect trajectories when reviewing possible external answer retrieval.

Axis 6 · Benchmark contamination

For every task we compare its instruction, solution and verifier separately with the Terminal-Bench 2.1 task set and

score similarity from 0 to 1 (0.9 = the same task reskinned, 0.7 = same core approach, 0.4 = a shared subtask,

0.1 = same domain only). Renamed variables, files and domain terms do not lower the score; shared test scaffolding does not

raise it. The task's score is the maximum of the three.

Similarity to the genre of Terminal-Bench is expected. Only near-identity to a specific task counts.

Quality score

Weights are S0 6 · S1 4 · S2 3 · S3 2 · S4 1. S0 applies only to Axis 4 normal-run reward hacking; it is not a design

criterion, so it does not enter the table below or the denominator. The maximum load of the scored design criteria is

3×4 + 4×3 + 3×2 + 2×1 = 32 per task.

> Task quality = max(0, 1 − total load ÷ 32), where total load = confirmed design defects + defects measured in Axes 1, 2, 4, 5 and 6.

> Dataset quality = max(0, 1 − sum of total loads ÷ (tasks × 32)), only when every task was fully measured.

> Design quality = 1 − sum of design loads ÷ (tasks × 32), shown for reference.

> Worked example. A task has one confirmed S2 design defect (load 3). Axis 4 records a black-box breach (load 4) and

> Axis 5 a difficulty_crux FAIL (load 2). Total load 9, so task quality = 1 − 9 ÷ 32 = 0.72, and its design quality is

> 1 − 3 ÷ 32 = 0.91.

The score measures how heavily defects land, not how many there are.

Difficulty measurement (a separate dimension, not part of the quality score)

Reference index. New measurements use tb21-tb4-codex-v2: Luna low 1.14, Terra max 1.94, Sol max 3.06, Astra max 4.08, in increasing order. Reference models are Haiku 4.5 (legacy configuration) at 1, Sonnet 5 (legacy configuration) at 2, Opus 5 (legacy configuration / TB4 xhigh) at 3, and Fable 5.1 max at 4. The TB4 Sonnet linking reference uses max.

Calibration. L is piecewise linear interpolation on the frozen legacy TB2.1 reference points (44.2,1), (80.4,2), (87.6,3), (89.9,4). Luna=L(49.4), Terra=L(78.4). U uses TB4 Sonnet max (12.4,2), Opus xhigh (53.9,3), and Fable max (57.9,4). Sol is (L(88.8)+U(37.3))/2=3.060870; Astra uses a small extension of the final segment, 3+(58.2-53.9)/(57.9-53.9)=4.075. Opus remains 3 on both references. Astra is not assigned 4 by definition.

Sources, observed September 18–19, 2026: TB2.1 leaderboard, OpenAI effort chart, TB4 leaderboard. The original sources of legacy TB2.1 constants have not been reverified as current Claude leaderboard scores. This is an approximate reference index with disclosed effort and harness differences. Equal weights are an operational convention, not an empirically optimal weighting. Decimal differences do not establish significant rankings. TB2.1 performance of at least 85% triggers inspection of a higher benchmark; coordinates remain frozen within a profile version.

Execution direction. Reuse accepted evidence first. Otherwise start with three normal Sol max trials. Mean reward ≥0.8 for graded packs, or 3/3 full successes for binary packs, moves down to Terra max; Terra mean ≥0.5 moves down to Luna low. Sol mean <0.2 or 0/3 successes moves up to Astra max. Intermediate results finish at that rung. Existing easy Astra max results lead down to Sol; no success at Astra does not trigger a weaker Sol run. If only Astra has a complete mid-band result, that evidence finishes the measurement without scheduling Sol. An existing Luna–Terra bracket around mean reward 0.5 avoids additional stronger-model runs.

Reuse and migration. Completed valid Claude ladders can be recomputed from retained rewards on unchanged reference coordinates. Old Codex D is recomputed on the new index; earlier values and profiles are preserved. Disclosing benchmark-specific efforts does not allow relabeling actual trials: Astra high does not count as Astra max. Missing rewards and policy refusals are not scores or grounds to bypass a refusal with another model.

Difficulty D is the ability coordinate where actual vendor-task mean reward crosses 0.5. Interpolate between measured rungs; otherwise use the existing extrapolation slope of 0.6 and label the estimate. A solve is reward ≥0.9 for graded packs or exactly 1 for binary packs. Dataset D is the median of scored tasks after the ladder finishes, with coverage disclosed. Unavailable and solvability outcomes are excluded; no scored tasks means null. Successful-run step counts describe horizon separately and never add to D.

Measurement limit. No success at the top Astra rung withholds D. Confirmed Axis 5 task_specification or difficulty_crux FAIL indicates a task defect; otherwise the cause remains unresolved at the measured limit. A missing judgment is not a PASS. Public benchmark results never replace actual vendor trials. Difficulty remains separate from quality.

This manual is a methodology document. Per-vendor numbers and task-level evidence are in the accompanying detail document.