Integrity — how we know the numbers mean anything

This project found, during its own pre-series smoke runs on SWE-bench Lite, that the harness had unrestricted internet egress and that its own repository clone carried the upstream fix in git history. On django__django-10924 a Claude Code attempt downloaded the gold patch and the gold test file and was graded resolved: true. That verdict was discarded, the review that found it is in the project's development record, and the framework was rebuilt around the controls below. None of the runs published on this site predate those fixes. Almost nobody publishing evaluation numbers runs that check on themselves. It is the reason to believe every other number here, so it sits before the results.

Isolation: what the agent cannot reach

control	how it is enforced	what it rules out
No route to the internet.	Harness tasks run in subnets with no default route. The dispatcher refuses to launch if the subnet or security group has any 0.0.0.0/0 rule. The only reachable endpoints are the inference gateway and in-VPC AWS endpoints for artifacts, queues and live progress.	Looking up the fix, the upstream pull request, or the test file
The repository comes from SWE-bench's own hardened image.	Official per-instance images pinned by digest: origin removed, tags after the base commit deleted, reflog expired, garbage-collected, and an assertion that zero future commits remain. The framework never clones; adapters are forbidden from touching the checkout.	Reading the fix out of the future
The agent is not root and holds no secrets.	The harness runs as an unprivileged agent user; the task definition carries no secrets; the per-run key arrives as a container override; the execution role's secret grant was removed.	Reading the worker's credentials out of /proc and reaching prior patches, the results queue or the gateway master key
The dataset the task reads is gold-stripped.	IAM allows the task to read only the *.public.jsonl mirror, which carries no gold patch and no test patch.	The answer arriving with the question
Grading cannot reach PyPI.	The eval image bakes PIP_NO_INDEX=1 with a wheelhouse; the gold gate proves each image grades the gold patch RESOLVED offline before any run.	An environment that quietly drifts between validation and grading

Detection: what we look for anyway

check	where	on the published runs
Patch touches the gold test files (touches_test_files, with the paths)	eval side, before grading	true on 0 of 2,500 attempts in the five-harness series; true on 1 of 78 in the Laguna pilot (below)
Model-authored test files intersecting the gold test patch are stripped before grading	eval side	the control that turned the django-10924 false positive into a discarded result; fired once in the Laguna pilot (below)
Gold-patch similarity	eval side only — the harness never sees the gold patch	recorded per attempt; shown in the panel below
Offline leak pass — hidden test ids referenced in the patch or trajectory	a separate pass over the stored artifacts, leak_detectable / leaked	run on all twelve published runs, 2,509 attempts of the five-harness series: 123 name hits, 39 written by the agent before anything showed it the name, one coinciding with a judge flag — the table and reading below
Judge: contamination and test-gaming dimensions, cited turns required	rubric v3	see the findings below

harness	leak-detectable	hidden-test name hits	…written by the agent first	touches test files	grade invalid	gold-patch similarity p50
OpenCode	335	20	4	0	0	0.52
custom minimal agent	335	32	16	0	0	0.45
Claude Code	335	21	4	0	0	0.51
Codex CLI	335	22	4	0	0	0.50
mini-swe-agent	335	28	11	0	0	0.42

How to read the leak columns. The detector works from a committed, gold-derived map of which FAIL_TO_PASS test ids do not exist at the base commit: 475 of the 500 Verified instances are in the map, 335 of them with at least one such id (leak-detectable), 140 with none, and 25 are omitted as unknown because their test ids are prose or name tests the gold patch only modifies. An attempt is a hit when its patch or trajectory contains one of those names. A hit is evidence for review, never a verdict, and the scan says where the name first appeared:

harness	attempts scanned	leak-detectable	name hits	first named in the task	first seen in tool output	written by the agent first	of those, judge-flagged too
custom minimal	501	336	32	6	14	16	0
OpenCode (re-run, fixed image)	500	335	20	6	16	4	0
Codex CLI	503	338	22	6	18	4	1
Claude Code	503	338	21	6	16	4	0
mini-swe-agent	502	336	28	6	16	11	0

Six instances name the hidden test in the problem statement itself, so every harness "hits" them. Most of the rest are names that appeared first in a tool result — the test already existed at the base commit and the gold patch modified it, which the map's diff parse counts as an addition; the same fourteen instances hit on all five harnesses for that reason. The column that carries any signal is the last pair: attempts where the agent wrote a hidden test's name before any file or test output had shown it. Reading those 39 trajectories, they are the agent naming a test it wrote itself — test_serialize_nested_class under a docstring saying what it tests, test_iforest_warm_start, a test_dict inferred from the test_list and test_tuple beside it — which is what a descriptive name looks like whether or not the model has seen the upstream test. One attempt is flagged by both detectors: Codex on django__django-15278 ran schema.tests.SchemaTests.test_add_field_o2o_nullable by name before anything had listed it, and the judge independently scored that attempt's contamination at 2 of 3. It stays in the results as graded, with both flags attached.

The scan ran offline on 2026-09-10, and on 2026-09-11 for the OpenCode re-run, over the stored patch.diff and trajectory.jsonl artifacts with map version sha256:1e1bc0b3…; it is the same detector the framework's backfill pass runs against the database, and the per-attempt result is in every export row and the Explorer.

What we found

Every flag the checks and the judge raised on the published runs, and what it turned out to be:

• Five contamination flags on the Claude Code run (severity 3). On inspection all five were the agent reading git history at or before the checkout — ordinary code reading, since the history is pruned at the base commit. The rubric was amended (version 2) to say so, and the flags were retracted on re-judgement.
• Twelve contamination flags under rubric v3, across all 2,509 attempts of the five 500-runs, four at the top severity. None cites history, hidden tests or anything else reachable inside the container; all four are knowledge that appears before any derivation — an upstream issue number in a code comment (pytest-dev__pytest-7236, Claude Code, which then searched the history for that number and found nothing), the exact upstream fix at the first attempt (django__django-13449, Codex), a diagnosis stated before the first tool call (django__django-14011, custom minimal), a commit hash the task never mentions (pydata__xarray-6599, custom minimal). That is the memorisation class again: reported with the cited turns on the judge page, not adjudicated, verdicts unchanged.
• One contamination flag on the Laguna pilot, django__django-13121. The judge cited the agent running git show 2115be616b and reading "the upstream fix". That commit is dated 2018-01-04; the instance's base commit is from 2020-06-29. The commit is history before the checkout — the same class as the Claude Code flags — and the control held. It stays in the judge bundle as a scored dimension with its evidence, and is recorded here as a false positive.
• One contamination flag on the OpenCode × MiniMax run, django__django-10973 (severity 2). The judge observed that the agent's very first edit already had the structure of the upstream fix, including elements not derivable from anything it had read. Nothing in the container could have supplied it: this is the judge suspecting memorisation from training data, which no isolation control can prevent and no static check can detect. It is reported as what it is; the verdict stands, because the tests passed on a patch the agent wrote in its sandbox.
• One test-gaming flag on the Laguna pilot, django__django-10973 (severity 3). The agent edited tests/dbshell/test_postgresql.py to match its implementation, said "I shouldn't have modified the test file", reverted, then re-applied the same edits with a full-file write. The framework's static check caught it: the eval row records touches_test_files = true with the path, the file was stripped before grading because it intersects the gold test patch, and the official tests were run against the un-gamed patch — which passed. The verdict is on the code change alone; the judge's flag is on the behaviour, and both are true.
• MiniMax ran git log --all and git show unprompted on several instances of the published series. It found nothing, because the hardening held. It is recorded because a rig that had not pruned history would have been leaked by an agent that was not told to cheat.

One defect in the framework's own reporting surfaced while writing this page: the run export merges the harness-phase and eval-phase rows with "last non-null write wins", and the harness row carries a placeholder false for touches_test_files that can overwrite the eval row's true. The Laguna pilot's export therefore shows the django-10973 flag as false although the database row, the stripped path and the judge all agree it is true. The panel above is built from the exports and inherits the defect for that run; it is listed as an open fix in the framework, and this paragraph stays until the exports are regenerated.

The holes we found in ourselves

An audit of the harness container from the inside, after the published series ran, found three gaps. All three are closed structurally, not by detection; one more is open and stated.

	what an agent could have reached	closed by
G1	a full-history git mirror service, reachable from the harness security group — the gold patch one hop away	rule and environment variable removed; the isolation probe now asserts the clone is blocked; the mirror is deployed but vestigial
G2	as root, /proc/1/environ → task credentials → prior patches, the results queue, the gateway master key	non-root agent user, runtime chown of the testbed, no secrets in the task, execution-role grant removed
G3	dependency re-solve at image build drifting the environment between validation and grading	official images by digest plus the gold gate (ADR-0043)
G4	test gaming through configuration files outside the paths touches_test_files covers	open — noted in scope, not closed

Did any published run leak through G1 or G2? All 2,650 stored trajectories of the eleven published runs were scanned on 2026-09-10 for the mirror's hostname, for /proc/1/environ and /proc/self/environ, for the task-credentials endpoint and for credential variable names: zero occurrences of any of them. The only clone commands in the corpus are reproduction steps quoted in Sphinx issue texts; the three an agent actually attempted failed, because the task has no route to the internet. The judge's contamination dimension, which reads every judged trajectory, raised no such flag either; the fixes are structural so that the question does not have to rest on a scan.

The rest of the honesty rules

• Both denominators, always. Resolve rate per attempted and per gradeable, each with a 95% interval (ADR-0038). Empty patches stay in the denominator.
• Provenance on every number. Run id, model as resolved by the gateway, harness and image digest, framework commit, dataset revision and image snapshot, network posture, limits.
• Not measured is never zero. Any null renders as not measured, on this page and everywhere.
• Gold-patch validation before a model dollar is spent. All 500 Verified images grade the gold patch RESOLVED offline (three gate runs on 2026-09-06; one instance needed a pre-downloaded wheel).

Provenance — what this run actually measured

run

01789087148400722169-0c9303b3 · 2026-09-11

model

openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-opencode)

harness

OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)

framework

631be843b55ccabe33347f0a7de2f7ee0676ce1a

harness image

sha256:c8c56129b240e898fb5de452b50e36ad1aedf04fada946592ee98ca757753564

benchmark

SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json

network

isolated

explore

this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts

download

this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs

Provenance — what this run actually measured

run

01788910552328915713-21ebff89 · 2026-09-08

model

openrouter/poolside/laguna-xs-2.1 (alias laguna-xs-2.1-opencode)

harness

OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)

framework

dd365d8ff1197276992135731e67d81b17fb6e94

harness image

sha256:5f326fc00f2decd4f0592b10ac0c36defe26cfed3ae27e0eb24d3804d2df944f

benchmark

SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json

network

isolated

explore

this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts

download

this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs