Harness configuration
Principle. Each harness runs as close to its published defaults as the isolation posture
allows. The framework changes only what is needed to point the harness at the gateway, keep it
inside /testbed, and read its output. Prompt engineering per harness was not done; the task
statement is the SWE-bench problem statement, framed the same way for every harness.
| harness | model route | changes from default | why |
|---|---|---|---|
| OpenCode | OpenAI-compatible, via the local shim | permission block allowing external_directory and the core tool kinds; static ripgrep on PATH | auto-rejected permissions ended sessions; grep/glob needed a binary it could not download |
| Claude Code | Anthropic /messages format | declared context window; native trajectory export | pre-flight token counting on the route |
| Codex CLI | OpenAI-compatible | none beyond routing | — |
| mini-swe-agent | OpenAI-compatible | one added sentence naming /testbed in the task framing | the reference agent otherwise searched the wrong directory |
| custom minimal | OpenAI-compatible | n/a — the control | — |
What is identical across harnesses
- The instance image, the repository state, the base commit and the pruned history.
- The gateway alias's model and provider routing, the per-instance limits, the run's spend cap.
- The output contract: a diff of
/testbed, graded by the official harness. - Sampling and reasoning parameters are set on the gateway alias, not left to each harness. For MiniMax M2.5 every OpenAI-compatible alias carries temperature 1.0, top_p 0.95, top_k 40 and reasoning effort high (the value the published leaderboard row used); the provider drops any parameter its endpoint does not accept, which for MiniMax's own endpoint is top_k. The alias value applies when the harness's request carries none of its own; the per-call ledger records what each request sent. The Claude Code alias goes through the Anthropic messages route, where the parameter is the thinking budget rather than an effort level: the harness is started with a thinking budget of 4,096 tokens, which the gateway translates to reasoning effort high upstream (1,024 maps to low, 2,048 to medium, 4,096 and above to high, measured live), so Claude Code reasons at the same effort as the other four. Without that setting Claude Code never asks an unknown backend for thinking at all. The alias parameters are recorded in each run's provenance.
Open question
Whether to publish each harness's exact configuration files verbatim (they contain no secrets) is the owner's call, and would make this page reproducible rather than descriptive.