Same model, five harnesses

In five lines.

• Five agent harnesses, one model (MiniMax M2.5), the same 500 SWE-bench Verified instances, the same per-instance caps, the same isolated containers. The harness and its image are the only variables.
• The resolve rates are statistically the same: 379 to 385 of 500, every interval overlapping every other. On this model the scaffold does not move the outcome.
• Everything else moves. Calls per attempt run from 29 to 43, the bill from $26 to $40 for the same outcome, agent time from 126 s to 172 s at the median on a like-for-like API route, and 283 s for Codex, whose route adds a ten-second hold per call.
• The two-tool control agent the framework ships sits inside that band on every axis, which is what tells you the scaffold's contribution is efficiency, not capability.
• The judge sees the same thing from the trajectories: the harnesses differ in redundant calls, hallucinated tool output and test edits, not in whether they solve the task.

Cost per resolved instance

USD of inference per resolved SWE-bench Verified instance · minimax/minimax-m2.5 · lower is better

Inference spend only, from the gateway's own metering (compute cost is reported separately on the Cost page). Sorted cheapest first. A missing bar is an unmeasured run, not a free one.

What was held fixed

Model, alias parameters (temperature, top_p, reasoning effort high), the 500 instances, attempts per instance (one), the cost cap per instance, the turn cap, the context window, the network posture (no egress), the grading (the official harness in the official images). Each harness ran as close to its published defaults as the isolation allows; the changes are listed on the harness configuration page, and none is prompt engineering.

What the scaffold changed

Agent time per attempt

seconds in the harness phase · median and 90th percentile

• Time. Claude Code finishes an attempt in about two minutes at the median, Codex CLI in nearly five. Codex's extra time is not the agent thinking longer: it is the API path. See "Why Codex is slow" below.
• Calls. OpenCode makes 29 calls per attempt at the median; the two minimal agents make 43. A minimal loop asks the model more often and lets it decide less per turn.
• Context. Every harness re-sends a growing conversation on each call, so 95% to 97% of prompt tokens are cache reads on all five. Claude Code carries the largest context per call (30K tokens), mini-swe-agent the smallest (23K). Cost is context times calls, and the harness sets both.

Tokens per attempt

uncached input · cached input · output · mean per attempt · same model, same 500 instances

Mean per attempt. Cached input is priced at about a tenth of uncached input on this model, so the cost of a bar is mostly its two darker segments. Claude Code's cache reads are reported by the provider outside the input figure and are added here; the other four report them inside.

• Why Claude Code costs the most. It uses the most of all three token kinds per attempt: cached input 1.40M against 1.03M to 1.32M for the others, uncached input 72K against 37K to 53K, output 12.1K against 8.7K to 11.6K. Where each of those comes from is the next section.

Where Claude Code's tokens come from, against OpenCode

Same model, same instances, same caps. OpenCode is the cheapest harness and Claude Code the most expensive, $0.053 against $0.079 per attempt, so the pair is the cleanest place to see what a scaffold does to the bill. The bill for an attempt is the sum over its calls of the context each call carries, plus its output. Context has two parts: a fixed block the harness sends on every call (its system prompt and its tool definitions), and the conversation so far, which grows as tool output is appended. The chart shows both: where a line starts is the fixed block, its slope is the growth.

Context carried per call

median prompt tokens at each call of an attempt · one line per harness · same model, same 500 instances

Median prompt tokens at each call index across the attempts that reached it (at least 50). OpenCode's first two calls are housekeeping (a title, a summary) and carry almost nothing; its task context starts at call 3. Claude Code's first call already carries 19K tokens; the dip at its second call is a small side call of its own.

per attempt	Claude Code	OpenCode	ratio
fixed context on every call (system prompt and tools)	18.8K tokens	7.5K	2.5×
calls per attempt, median	37	29	1.3×
fixed context × calls	0.70M	0.22M	3.2×
conversation growth per call	0.41K	0.49K	0.8×
prompt tokens per attempt, all calls	1.47M	1.07M	1.4×
tool output appended to the conversation	92 KB	44 KB	2.1×
output tokens (share that is reasoning)	12.1K (43%)	8.7K (41%)	1.4×
attempts the judge flagged: looping / tool efficiency / verbose reasoning	191 / 258 / 166	72 / 82 / 65	2.7× / 3.1× / 2.6×
cost: cached + uncached + output	$0.042 + $0.022 + $0.015	$0.031 + $0.011 + $0.010	1.5×

Reading it:

• The largest single item is the fixed block. Claude Code's system prompt and tool schema are about 19K tokens and go out on every call; OpenCode's are about 7.5K. Over 37 calls that is 0.70M tokens of Claude Code's 1.47M, against 0.22M of OpenCode's 1.07M. Nearly all of it is cache reads, cheap per token, but it is half of Claude Code's prompt volume before the agent has read a single file. That is pure scaffold: the model did not choose it.
• The conversation itself grows more slowly on Claude Code. 0.41K per call against 0.49K. The bloat is not "Claude keeps more per turn"; the per-turn additions are ordinary.
• What it adds, it adds twice. Claude Code appends twice the tool output per attempt, and the judge, reading the trajectories with cited turns, flags it for looping 2.7× as often and for redundant tool use 3.1× as often as OpenCode: repeated reads of unchanged files, repeated probes that return the same answer. That doubling is what makes its uncached input, the new text per call, twice OpenCode's. Uncached input is the expensive kind, so this second item costs as much as the first despite being far fewer tokens.
• It writes more. 1.4× the output tokens, with a similar reasoning share, and 2.6× the verbose-reasoning flags. This is the smallest of the three items.

So of the $0.026 gap per attempt, about $0.011 is the fixed block re-sent on more calls (cached input), about $0.011 is the doubled new text per call from repeated and larger tool output (uncached input), and about $0.004 is output. Two of the three are scaffold behaviour a harness could change without touching the model: a smaller fixed block, and not re-issuing a call whose answer is already in the conversation.

The same decomposition across the model axis. On OpenCode, with the fixed block and the tool surface held constant, swapping MiniMax for Laguna multiplied calls per attempt by 2.6 and context per call by 2.9, for 7.5× the prompt tokens. The harness axis moved prompt tokens 1.4× between the leanest and heaviest scaffold; the model axis moved them 7.5× on one scaffold. Both are context times calls; the model sets the multiplier, the harness sets the floor. Same harness, two models.

Why Codex is slow

Codex's median agent time is 283 s against 126 s to 172 s for the others, and the scorecard shows 89% of that time is spent waiting on the model. The per-call ledger explains it. Seventy percent of Codex's 22,500 calls took between 10.08 and 10.15 seconds, regardless of size: a call that returned 50 output tokens took as long as one that returned 500, and the first call of nearly every attempt sat in that band. The remaining 30% took 1.5 to 5 s, like every other harness's calls. The response arrived whole, first byte and last byte 45 ms apart. The same plateau is in Codex's earlier 41-instance run on this model.

The reason is the wire path. Codex is the only harness that speaks the OpenAI Responses API, and the gateway (LiteLLM 1.99) routes Responses calls for this provider to OpenRouter's own Responses endpoint, not to the chat-completions endpoint the other four use (OpenAI format for OpenCode, mini-swe-agent and the custom agent; Anthropic messages format for Claude Code). Whatever OpenRouter does behind that endpoint for this model, the signature on our side is a ten-second hold on most calls and delivery of the finished response in one piece. Streaming is not the difference: mini-swe-agent and the custom agent do not stream either and their calls return in about two seconds.

Two consequences. Codex's time figure is a property of the provider's Responses endpoint, not of the Codex scaffold, and would change with a different route. And its cost is unaffected: tokens and prices are the same, which is why Codex is the second cheapest per resolved instance while being the slowest.

What we know about the fix, and why it was not re-run. Codex itself has no switch: it once accepted a chat-completions wire option, but that was removed in February 2026 and every provider it talks to must speak the Responses API. The gateway does have one. LiteLLM can be told, per alias, to serve a Responses request through its chat-completions bridge (use_chat_completions_api), which would put Codex's calls on the same endpoint OpenCode uses, with the reasoning effort carried across. OpenRouter documents no hold on its Responses endpoint, so the plateau is an observation from our ledger, not a documented behaviour. A re-run on the bridge would be a new arm rather than a replacement, because Codex replays its own reasoning between turns through Responses-only items that the bridge may not round-trip. We chose to document the finding and leave the published Codex row as it ran.

The time comparison without Codex. For the four harnesses on the chat-completions route the spread is what the scaffolds themselves produce:

Agent time per attempt, chat-completions route only

the four harnesses whose calls take the same route · seconds · median and 90th percentile

Codex CLI omitted: its calls take the provider's Responses route, where 70% of them sit on a ten-second hold. On this route the median attempt runs 126 s to 172 s, a 1.4× spread, and the 90th percentile 407 s to 469 s.

Read this way, Claude Code is the fastest scaffold and mini-swe-agent the slowest, 126 s against 172 s at the median, and the four 90th percentiles are within 15% of each other. Time separates the scaffolds far less than cost does.

• Reasoning. Between a third and 43% of every harness's output tokens are reasoning. Claude Code reasons the most; its effort is set through a thinking budget the gateway maps to high, the same level as the other four.
• Stopping. All five keep working after their own tests pass; the judge's redundant-call and duplicate-test flags are where that shows. Which harness stops soonest is the next experiment, and the prompt experiment already showed the model will not do it when merely asked.

The full table is on the By harness page.

What the scaffold did not change

• The resolve rate. 75.8% to 77.0%, intervals overlapping. The head-to-head page shows the sets overlap heavily too: the instances no harness resolved are the model's hard core, and picking the best harness per instance would add only a few points.
• The failure modes. Every one of the 2,500 attempts ended as completed; no cap bound, no harness crashed. The 14 empty patches are model decisions, not scaffold behaviour (failure taxonomy).
• Contamination. Twelve flags in 2,509 judged attempts, all memorisation-class, spread across harnesses (judge).

What the judge added

The rubric-v3 pass read every attempt. Per harness it found different habits: Claude Code carries the most hallucination flags, mostly asserting the content of lines its tool output had elided, and the most redundant-call flags; mini-swe-agent the most duplicate calls and test re-runs; the custom minimal agent the most test edits; Codex the fewest misreads and no test-gaming flag at all. None of those habits changed the verdicts. It also found the rig's own bug: 317 of 501 attempts on the first OpenCode image flagged for a broken search tool, fixed and re-run.

Where this leaves a harness choice

With the model fixed, the harness did not change how many problems got solved, so it cannot be chosen on resolve rate. It changed two things a reader can act on:

• Cost, by 1.5×. $26 for OpenCode to $40 for Claude Code for the same 500 problems, $0.068 to $0.103 per resolved instance. The gap is the fixed context block re-sent on every call, the number of calls, and how much repeated tool output gets appended, as the decomposition above shows. A scaffold pays for that, not the model.
• Time, by 1.4× on a like-for-like route, 2× including Codex. The four harnesses on the chat-completions route run 126 s to 172 s per attempt at the median. Codex's 283 s is the API route, not the agent: its calls sit on a ten-second hold on the provider's Responses endpoint, and the fix is a gateway setting documented above, not a scaffold change.

If a scaffold is going to earn its keep, it has to do so by enforcing what the model will not do on its own: search before reading, window the reads, do not re-issue a call whose answer is already in the conversation, and stop after green. The prompt experiment showed the first two respond to instruction and the last does not.

Runs: OpenCode 01789087148400722169-0c9303b3, custom minimal 01788739268680819707-05bd05b8, Claude Code 01788799888585639836-1f722383, Codex CLI 01788744084885192755-c10df654, mini-swe-agent 01788723779415700648-b4338f67. Provenance for each is on By harness; the raw exports are on Downloads.