By harness
Finding. The harness decides what gets burnt, not what gets solved. Same model, same 500 instances, same limits, same isolated environments: the five resolve rates sit inside each other's intervals, while agent time per attempt, calls per attempt, context carried per call and the bill differ by up to a factor of two.
The scorecard
| harness | agent time per attempt | calls per attempt | waiting on the model | tokens | cost | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| median | p90 | median | p90 | share of agent time | median call latency | prompt per call | cached share | output per attempt (reasoning share) | per attempt | per resolved | |
| OpenCode 385 of 500 resolved · explore | 142 s | 468 s | 29 | 84 | 65% | 2.6 s | 25.0K | 97% | 8.7K (41%) | $0.053 | $0.068 |
| Codex CLI 383 of 500 resolved · explore | 283 s | 784 s | 34 | 83 | 89% | 10.1 s | 28.2K | 97% | 9.4K (36%) | $0.061 | $0.080 |
| mini-swe-agent 379 of 500 resolved · explore | 172 s | 410 s | 43 | 99 | 69% | 2.1 s † | 23.1K | 96% | 11.6K (34%) | $0.065 | $0.086 |
| custom minimal agent 384 of 500 resolved · explore | 161 s | 403 s | 43 | 95 | 75% | 2.3 s † | 25.4K | 96% | 11.5K (35%) | $0.068 | $0.088 |
| Claude Code 384 of 500 resolved · explore | 126 s | 408 s | 37 | 99 | 99% | 2.9 s | 30.4K | 95% | 12.1K (43%) | $0.079 | $0.103 |
Sorted by cost per resolved instance. Agent time is the harness phase of the attempt, from the export. Calls are ledger calls per attempt, including refusals and retries. "Waiting on the model" is the sum of per-call latency over agent time; a call's latency is measured at the shim. Prompt tokens per call add cache reads to the input figure on the Anthropic route (Claude Code), where the provider reports them separately; on the OpenAI-compatible route cache reads are already inside the input figure. † marks a harness that does not stream, so its call latency is the full response time. Cost is the metered ledger; the per-resolved figure uses the run total. Pacer wait is under one percent of inference time in every run.
Reading across. OpenCode makes the fewest calls, carries a mid-sized context and is the cheapest per resolved instance. Codex CLI takes twice as long per attempt as the others, and the time is spent waiting on the model: seventy percent of its calls sit at a flat ten seconds whatever their size, because Codex alone speaks the Responses API and the gateway routes those calls to the provider's separate Responses endpoint (the reading is on Same model, five harnesses). The two minimal agents, mini-swe-agent and the framework's own control, make the most calls per attempt and produce the most output. Claude Code finishes an attempt fastest, carries the largest context per call, reasons the most, and is the most expensive per resolved instance. Cache behaviour does not separate them: every harness sits between 95% and 97% cached, because every one of them re-sends a growing conversation on each call.
The OpenCode row is the re-run on the fixed image (01789087148400722169-0c9303b3,
2026-09-11). The first OpenCode 500-run used an image on which OpenCode's grep and glob tools did
not work and its permission system ended 23 sessions (see Integrity); it is kept
in the Explorer as the documented defect, not in this table.
The charts
Context carried per call
median prompt tokens at each call of an attempt · one line per harness · same model, same 500 instances
Median prompt tokens at each call index across the attempts that reached it. Where a line starts is the harness's fixed block (system prompt and tool definitions, re-sent on every call); its slope is the conversation growth per call. OpenCode's first two calls are housekeeping and carry almost nothing. The decomposition of Claude Code's bill against OpenCode's is on Same model, five harnesses.
Agent time per attempt
seconds in the harness phase · median and 90th percentile · lower is faster
Queue wait, image pull and grading are fleet properties, the same for every harness, and are not in this bar; they are on the Cost page.
Agent time per attempt, chat-completions route only
the four harnesses whose calls take the same route · seconds · median and 90th percentile
The like-for-like comparison. Codex CLI is omitted because its calls take the provider's Responses route, where 70% of them sit on a ten-second hold; the reading and the gateway setting that would change it are on Same model, five harnesses. On this route the median attempt runs 126 s to 172 s and the 90th percentiles are within 15% of each other.
Calls per attempt
ledger calls per attempt, including refusals and retries · median and 90th percentile
A call is one request through the shim. The harness decides how many it makes; the model decides what each one asks for.
Tokens per attempt
uncached input · cached input · output · mean per attempt · same model, same 500 instances
Mean per attempt. Cached input is drawn separately: it is priced at about a tenth of uncached input, so the cost of a bar is mostly its dark segments. Claude Code's cache reads are reported by the provider outside the input figure and are added here; the other four report them inside.
What the harness controls, and what it does not
| the harness controls | the harness does not control |
|---|---|
| how many calls an attempt makes, and whether the loop stops after its tests pass | the resolve rate on this model: five harnesses, one band, overlapping intervals |
| how much context each call carries: whole-file reads, re-reads, tool output kept in the conversation | the hard core: the instances no harness resolved, on the head-to-head page |
| the API path: streaming or not, OpenAI or Anthropic format, and with it the latency per call | the empty patches, which on inspection are model decisions (failure taxonomy) |
| the tool surface the model sees, and so what the model does with its turns (judge findings) | the per-instance caps, which are the run's and never bound in this series |
What a scaffold adds, then, shows up in the bill and in the judge's efficiency findings, not in the resolve rate. The one place a harness moved a verdict was the broken OpenCode image, a framework defect, not a scaffold property.
Each run in full
Verdicts, how attempts ended, and provenance, per harness
OpenCode
| verdict | instances |
|---|---|
| resolved | 385 |
| unresolved | 110 |
| EMPTY_PATCH | 5 |
Resolve rate 77.0% per gradeable · 77.0% per attempted · cost $26.29 · tokens 534.1M in / 4.4M out (515.8M cached).
| how attempts ended | count |
|---|---|
| completed | 500 |
- run
- 01789087148400722169-0c9303b3 · 2026-09-11
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- 631be843b55ccabe33347f0a7de2f7ee0676ce1a
- harness image
- sha256:c8c56129b240e898fb5de452b50e36ad1aedf04fada946592ee98ca757753564
- gateway config
- bb22fcd7bad15793e5814f4abbd4dfba760ea3c76a3d8a69d7ec935a3ad2df7e
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- limits
- max_tokens=not set · max_cost_usd_per_instance=2 · attempts_per_instance=1 · temperature=1
custom minimal agent
| verdict | instances |
|---|---|
| resolved | 384 |
| unresolved | 114 |
| EMPTY_PATCH | 2 |
Resolve rate 76.8% per gradeable · 76.8% per attempted · cost $33.98 · tokens 683.1M in / 5.8M out (658.9M cached).
| how attempts ended | count |
|---|---|
| completed | 500 |
- run
- 01788739268680819707-05bd05b8 · 2026-09-07
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-custom_minimal)
- harness
- custom minimal agent · CLI built into the framework (the agent is framework code; its version is the framework commit)
- framework
- d5478ecf3a3f39e4fef21839f18f1f51c53b5c57
- harness image
- sha256:b294634551e3cf677e76fa011fabc28272eca1ed13d8d4a930d7b89270229374
- gateway config
- bb22fcd7bad15793e5814f4abbd4dfba760ea3c76a3d8a69d7ec935a3ad2df7e
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- limits
- max_tokens=not set · max_cost_usd_per_instance=2 · attempts_per_instance=1 · temperature=1
Claude Code
| verdict | instances |
|---|---|
| resolved | 384 |
| unresolved | 113 |
| EMPTY_PATCH | 3 |
Resolve rate 76.8% per gradeable · 76.8% per attempted · cost $39.50 · tokens 36.7M in / 6.0M out (699.7M cached).
| how attempts ended | count |
|---|---|
| completed | 500 |
- run
- 01788799888585639836-1f722383 · 2026-09-07
- model
- anthropic/minimax/minimax-m2.5 (alias minimax-m2.5-claude_code)
- harness
- Claude Code · CLI 2.1.234 (pinned in the harness image build at this framework commit)
- framework
- 0976edbedc5ced395f24ca9251cd3c86a3db74d7
- harness image
- sha256:b294634551e3cf677e76fa011fabc28272eca1ed13d8d4a930d7b89270229374
- gateway config
- bb22fcd7bad15793e5814f4abbd4dfba760ea3c76a3d8a69d7ec935a3ad2df7e
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- limits
- max_tokens=not set · max_cost_usd_per_instance=2 · attempts_per_instance=1 · temperature=not set
Codex CLI
| verdict | instances |
|---|---|
| resolved | 383 |
| unresolved | 116 |
| EMPTY_PATCH | 1 |
Resolve rate 76.6% per gradeable · 76.6% per attempted · cost $30.63 · tokens 635.6M in / 4.7M out (614.4M cached).
| how attempts ended | count |
|---|---|
| completed | 500 |
- run
- 01788744084885192755-c10df654 · 2026-09-07
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-codex)
- harness
- Codex CLI · CLI 0.147.0 (pinned in the harness image build at this framework commit)
- framework
- d5478ecf3a3f39e4fef21839f18f1f51c53b5c57
- harness image
- sha256:b294634551e3cf677e76fa011fabc28272eca1ed13d8d4a930d7b89270229374
- gateway config
- bb22fcd7bad15793e5814f4abbd4dfba760ea3c76a3d8a69d7ec935a3ad2df7e
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- limits
- max_tokens=not set · max_cost_usd_per_instance=2 · attempts_per_instance=1 · temperature=1
mini-swe-agent
| verdict | instances |
|---|---|
| resolved | 379 |
| unresolved | 118 |
| EMPTY_PATCH | 3 |
Resolve rate 75.8% per gradeable · 75.8% per attempted · cost $32.72 · tokens 616.9M in / 5.8M out (590.4M cached).
| how attempts ended | count |
|---|---|
| completed | 500 |
- run
- 01788723779415700648-b4338f67 · 2026-09-06
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-mini)
- harness
- mini-swe-agent · CLI 2.4.6 (pinned in the harness image build at this framework commit)
- framework
- ff0c00d8838fa75b3164d8d21bf91cc3e188b2ee
- harness image
- sha256:b294634551e3cf677e76fa011fabc28272eca1ed13d8d4a930d7b89270229374
- gateway config
- bb22fcd7bad15793e5814f4abbd4dfba760ea3c76a3d8a69d7ec935a3ad2df7e
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- limits
- max_tokens=not set · max_cost_usd_per_instance=2 · attempts_per_instance=1 · temperature=1
The scorecard is built by scripts/build_harness_scorecard.py in the site repository from the
run exports and the per-call ledgers the Explorer replays; the rules for every column are in the
file it writes.