Skip to main content

Motivation

What this is. A framework that takes any coding-agent harness and any model, runs them on SWE-bench Verified under identical conditions, and reports resolve rate, cost, behaviour and integrity checks with enough provenance that someone else can reproduce the number.

Where it came from​

A friend's startup needed to evaluate agent behaviour — not just "did it pass" but how an agent works, what it costs, where it wastes — across models and harnesses they do not control. There was no rig for that. SWE-bench Verified was the starting point because it has published numbers to compare against, so the rig could be validated before anyone trusted it on anything else. The behaviour evaluation is where this goes next; the parts built for it — a shim that meters every call, a judge that has to cite, artifacts for every attempt — are what this site shows.

The two questions a leaderboard number does not answer​

Does the harness change the outcome? The same model, driven by five different agent scaffolds: does the resolve rate move, does the cost, does the shape of the trajectory? Nobody publishes this, because nobody runs one model through five harnesses under the same limits. We did — five harnesses × MiniMax M2.5 × 500 instances — and the answer is that resolve rate barely moves, cost and turns move a lot, and one harness's broken search tool cost a re-run. Results → Headline.

Which of two similar models is actually cheaper? Compare models of similar capability and context length on cost, tokens and time-to-solve, not on a leaderboard number. Two models with the same resolve rate differed 4.4× in cost per task; the difference lives in calls × context, which only a per-call ledger can show. Results → Cheaper per token is not cheaper per task.

The principles it was built to keep​

  • Cost is a first-class result, controlled during the run, not reconciled afterwards. The first question anyone with a budget asks is "what will this cost me", and the honest answer needs the caps, the pacer and the ledger — not a bill. Design → Admission control.
  • Receipts, or it didn't happen. Every number on this site links to a run id, a provenance block and the export it came from; the trajectories are readable; the judge has to cite the turn. That is what someone evaluating vendors needs before trusting a comparison.
  • Look for cheating, in the agent and in the rig. Isolation is structural, detection runs anyway, and the holes found in the rig itself are published with the fixes. Integrity.
  • Open and reproducible on a laptop budget. One person, four weeks, about $170 of inference for the five-harness series, about a dollar a day idle; the framework, the exports and the raw bundles are the record. How it was built.

What it took​

A control plane on AWS that runs 500 agents concurrently as isolated tasks; a local proxy in every task that meters and caps every call; a gateway with per-run keys and hard spend caps; admission control that shares a provider's rate limits fairly across hundreds of agents; official per-instance images validated against the gold patches before a model dollar is spent; an operator dashboard with pause, abort, restart and live limits; and the measurement layer this site publishes. The Design section is the map; the decision record is the why; the Capabilities page is the full list.