Results
Twelve published runs: five harnesses on one model across the full 500-instance split, a two-model experiment on 78 problems, and a prompt experiment on the same 78. Resolve rates on the 500 sit in a 75.8% to 77.0% band with overlapping intervals; the bill spans 1.5× between scaffolds and 4.4× between models. Six findings come out of that. Each card links to the page with the run ids, intervals and provenance behind it; two are written up as longer reads, Same model, five harnesses and Same harness, two models, and the model finding has its own essay, Cheaper per token is not cheaper per task.
The reference run reproduces the official leaderboard
mini-swe-agent on MiniMax M2.5 resolves 379 of 500 here and 379 of 500 on swebench.com's Bash Only entry, with 88% per-instance agreement and cost within 11%. Same grading, same images, independently run.
Versus the official leaderboard →The harness decides what gets burnt, not what gets solved
Five harnesses on one model and the same 500 problems land between 75.8% and 77.0% with overlapping intervals. Agent time, calls per attempt and context per call differ by up to a factor of two; the bill differs 1.5×.
By harness →Cheaper per token is not cheaper per task
Laguna xs 2.1 and MiniMax M2.5 on the same harness, prompt, tools and limits resolve the same share of 78 problems; Laguna carries 7.5× the prompt tokens and costs 4.4× as much per resolved instance. Only the per-call ledger shows why.
Model vs model →Efficiency can be engineered, and it has to be enforced
Five working rules in the system prompt cut the bill by a quarter to a third on the same 78 problems with no measurable change in outcome. The rule both models ignored, stop once tests are green, is the one the harness now has to enforce.
What a prompt can buy →A two-tool loop keeps up with the commercial scaffolds
The framework's own control agent, one loop and two tools, resolves 384 of 500: inside the noise of the four CLIs. What a scaffold adds shows up in the bill and in the judge's efficiency findings, not in the resolve rate.
The custom minimal agent →The judge found the rig's bug first, and twelve memorisation flags in 2,509 attempts
Rubric v3 read every attempt of the five runs. Its first pass flagged 317 of 501 OpenCode attempts for a search tool broken in the isolated image, a framework defect fixed and re-run. Across all five it raised 12 contamination flags, none citing anything reachable inside the container, and one class of test gaming no static check sees.
LLM judge →Every run behind these cards is browsable in the Explorer: each instance with its trajectory and patch, every inference call on a timeline, the judge's verdicts. The five 500-runs: mini-swe-agent · custom minimal · Codex · Claude Code · OpenCode; the 78-instance arms are listed on What a prompt can buy and Model vs model. What the Explorer shows.
The pages
- Headline: cost per resolved instance and resolve rate with intervals, five harnesses.
- By harness: the scorecard, agent time, calls, context, tokens, and what the harness controls.
- Head-to-head: which instances each harness resolved that the others did not.
- What a prompt can buy: the 78-instance prompt experiment.
- Cheaper per token is not cheaper per task: Laguna against MiniMax on two harnesses.
- Model behaviour: how the two models spend their calls, from the trajectories.
- Failure taxonomy: how attempts ended, kept apart from the framework's own failures.
- LLM judge: what an evidence-required judge found.
- Versus the official leaderboard: the reference run against swebench.com.
Found a number that does not add up, or a run you read differently? Open an issue, start a discussion, or write to evals@preetdhaliwal.dev. Every figure above is pinned and re-gradable from a bundle, so a disagreement can be settled on the same data.