What a prompt can buy: the 78-instance experiment
A problem the judge measured, an attempt to fix it with a prompt, the result, and what the fix itself cost. Four runs of OpenCode on the same 78 SWE-bench Verified instances under the same limits (350 turns, $2 per instance, a 262k context window), varying the model (Laguna xs 2.1 or MiniMax M2.5) and whether a block of working rules was appended to the harness's system prompt. One run per arm; every number is a paired comparison over the 78.
The problem: where the tokens went
The rubric-v3 judge pass over the Laguna pilot scored a token efficiency dimension on 84 attempts, attributing each attempt's avoidable tokens to a fixed list of causes, each requiring cited turns:
| cause | attempts flagged (of 84) | mean share of avoidable tokens |
|---|---|---|
| repeated reads of unchanged files | 57 | 16.5% |
| redundant test runs | 49 | 15.9% |
| probing without searching first | 39 | 11.7% |
| unbounded file reads | 27 | 6.1% |
| looping | 24 | 5.1% |
| verbose reasoning | 13 | 2.6% |
| unbounded command output | 9 | 1.7% |
Counters over the plain-prompt trajectories, computed independently of the judge, agree on the order. Per 78 attempts, Laguna opened 503 files whole, re-read an unchanged file 728 times, ran the same test with nothing changed 38 times, and spent $4.39 after its own tests were already green; MiniMax's figures were 97, 245, 3 and $1.45. Reads first, then tests, then probing. All of it is context re-sent on every later call, which is where the bill lives (model behaviour).
The attempt: five working rules
Five rules, written for OpenCode's tool names, appended by the framework under a "Working rules
(framework-provided)" heading: 1,540 characters, about 370 tokens on every call. In short: read
with grep and glob before opening files, and open files with an offset and a limit, never
whole; bound shell output with tail; run the repository's own test runner on the covering
tests; verify once after the fix and stop, one reproduction, one run, no edge-case sweep; and do
not repeat an action that already produced its answer. No task-specific hints, nothing about
SWE-bench. The exact text is versioned in the framework as preset
opencode-laguna-efficiency-v2, and the harness worker logs a line per attempt confirming the
block was applied: 78 of 78 in each rules arm, 0 of 78 in each plain arm.
The rules, verbatim
- Locate, then read windows. Find the code with grep or glob using a specific path or pattern, then read with offset and limit (at most 200 lines per read). Never read a whole file. Never re-read a file or range you have not edited since you last read it: what you read is still in your context.
- Bound your outputs. Grep and glob inside a specific directory, not the whole repo; do not list large directories; when a command prints more than a screen, pipe it through tail -40.
- Run tests the repo's way, once per change. Django: from /testbed run
python tests/runtests.py <app.module.Class.test>(labels without atests.prefix; no pytest, no scratch scripts that call settings.configure()). Other repos:pytest <path/to/test_file.py> -k <name> -x -q. Run a test once per code change; never re-run a test that already passed with no code change in between. - Verify once, then stop. After your edit, run the existing test module that covers the change once. If no existing test covers it, write ONE reproduction (preferably a test file run through the project's runner), run it once after the fix, then stop; do not rewrite or extend it more than once. If it passes, run at most one broader module. Then stop: do not re-run suites, do not re-read the file you edited, do not write further scripts to re-prove a fix that has passed, do not summarise your work.
- Do not loop. If the same action has produced the same result twice, change approach or stop. Keep reasoning short and proportional to the decision in front of you.
The result
Cost per resolved instance, four arms
OpenCode · same 78 instances · same limits · plain prompt vs the working rules · lower is better
Inference spend only, from the gateway's own metering (compute cost is reported separately on the Cost page). Sorted cheapest first. A missing bar is an unmeasured run, not a free one.
| 78 paired | A · Laguna plain | B · Laguna + rules | C · MiniMax plain | D · MiniMax + rules |
|---|---|---|---|---|
| cost | $15.45 | $9.95 (−36%) | $3.49 | $2.53 (−28%) |
| resolved | 58 (74.4%) | 55 (70.5%) | 59 (75.6%) | 60 (76.9%) |
| cost per resolved | $0.266 | $0.181 | $0.059 | $0.042 |
| LLM calls (median per attempt) | 6,514 (70) | 5,356 (54) | 2,527 (23) | 2,376 (26.5) |
| prompt tokens per call | 76.9K | 60.2K | 26.4K | 20.2K |
| prompt tokens in total | 501M | 323M | 67M | 48M |
| whole-file reads | 503 | 276 | 97 | 65 |
| spend after own tests green | $4.39 | $2.99 | $1.45 | $0.84 |
- Cost fell by 36% on Laguna (A → B): cheaper on 64 of 78 instances, median per-instance ratio 0.53, bootstrap 95% interval on the total ratio 0.49–0.83.
- Cost fell by 28% on MiniMax (C → D): cheaper on 50 of 78, median ratio 0.88, interval 0.59–0.89. Smaller in relative terms and much smaller in dollars ($0.96 against $5.50), as expected: MiniMax was already carrying a third of Laguna's context per call.
- The mechanism is the same on both models. Context per call fell (76.9K → 60.2K on Laguna, 26.4K → 20.2K on MiniMax) while the call count barely moved. The saving is prompt-token volume, not fewer turns.
- Resolve rate did not move measurably. 58 → 55 on Laguna, 59 → 60 on MiniMax; all four intervals overlap almost entirely. On 78 instances one instance is 1.3 points, and a five-instance plain re-run flipped one verdict on its own.
What the rules themselves cost
The rules ride in the system prompt, so they are re-sent on every call. The ledger shows exactly how much: the first task-carrying call of an attempt grew from 7,504 to 7,877 prompt tokens on Laguna and from 7,412 to 7,752 on MiniMax, 373 and 340 tokens for the block. Over every call in the arm, priced at the list rates (cached input after the first call of an attempt, uncached on it):
| Laguna | MiniMax | |
|---|---|---|
| rules tokens per call, measured | 373 | 340 |
| calls in the rules arm | 5,596 | 2,580 |
| tokens the rules added over the run | 2.1M | 0.9M |
| cost of sending the rules | $0.06 | $0.03 |
| saving on the bill | $5.50 | $0.96 |
| the rules' share of the saving | 1% | 3% |
Where the saving came from, by token kind: on Laguna, cache reads fell from $14.80 to $9.55, and uncached input and output together by $0.26; on MiniMax, cache reads from $1.92 to $1.38, uncached input from $0.78 to $0.57, output from $0.77 to $0.58. The rules cut what gets re-sent, and re-sending is the bill, so a block that costs cents to send saves dollars. The net is the 36% and 28% above; the rules' own cost is already inside those figures.
Which rules the models obeyed
| rule | Laguna | MiniMax |
|---|---|---|
| 1–3: search before reading, window the reads, bound output, use the repo's runner | obeyed: unbounded whole-file reads fell from 503 to 276; this is where the saving came from | obeyed: 97 to 65 |
| 4: verify once, then stop | partly: spend after the last edit halved, but the tail is still 36% of cost; every verdict that flipped to unresolved ran the covering tests green, then a broader module, then summarised | partly: post-edit spend fell 36%, tail still 48% |
| 5: do not repeat | no looping cases in either arm; redundant test runs stayed small (38 → 54 on Laguna, 3 → 5 on MiniMax) | same |
The stop rule is advisory in a prompt. The post-edit tail is proportionally the same on both models (36–55% of spend), which is the argument for enforcing "stop after green" in the harness rather than asking the model. That is the next experiment, not this one.
Verdict flips, instance by instance
| pair | both resolved | both unresolved | lost | gained |
|---|---|---|---|---|
| A → B (Laguna, rules) | 53 | 18 | 5: four wrong or incomplete fixes after green tests, one empty patch | 2 |
| C → D (MiniMax, rules) | 56 | 15 | 3 | 4 |
| A → C (Laguna → MiniMax, plain) | 55 | 16 | 3 | 4 |
The one empty patch (django__django-11555, arm B) is the only failure mode that looks
rule-related: 41 calls of reading and probing, no edit, then a final response containing reasoning
only, no text, no tool call, which OpenCode treats as the agent being finished. Nothing like it
occurred in the plain Laguna run (0 of 79 completed attempts) or on MiniMax. One case; flagged
rather than explained. Where the rules arms cost more on an instance, it was the hardest
instances getting more work (astropy__astropy-14096 135 → 268 calls, django__django-12965
113 → 185), all still resolved.
Caveats
- The 78 are the Laguna pilot's dispatch-order subset of Verified, 40 django, 18 astropy and the rest, not a random sample of the 500. The rule about using the repository's own test runner helps django-heavy work more than it would the full split.
- One run per arm. The paired deltas and the bootstrap intervals are the evidence; the resolve rates are reported with their intervals and are not distinguishable.
- Arm A ran on the framework commit before the image refresh; arms B–D ran on the current image. Both images have working search tools; the framework layer differs.
- The rules were written for OpenCode's tool surface. Presets for the other harnesses exist in the framework but were not run in this experiment.
The four arms as run
| harness | model | attempted | gradeable | resolved | rate / attempted | rate / gradeable | cost | $ / resolved | tokens in / out |
|---|---|---|---|---|---|---|---|---|---|
| OpenCode 21ebff89 | openrouter/poolside/laguna-xs-2.1 | 78 | 78 | 58 | 74.4% [63.7, 82.7] | 74.4% [63.7, 82.7] | $15.45 | $0.266 | 500.8M / 1.8M |
| OpenCode e95ccde6 | openrouter/poolside/laguna-xs-2.1 | 78 | 78 | 55 | 70.5% [59.6, 79.5] | 70.5% [59.6, 79.5] | $9.95 | $0.181 | 322.6M / 1.3M |
| OpenCode 883d2571 | openrouter/minimax/minimax-m2.5 | 78 | 78 | 59 | 75.6% [65.1, 83.8] | 75.6% [65.1, 83.8] | $3.49 | $0.059 | 66.7M / 640k |
| OpenCode 39ff3720 | openrouter/minimax/minimax-m2.5 | 78 | 78 | 60 | 76.9% [66.4, 84.9] | 76.9% [66.4, 84.9] | $2.53 | $0.042 | 48.0M / 487k |
Per-attempt metrics (calls, reads, test runs, spend after the last edit) are computed from the stored trajectory and the per-call ledger of each attempt with the same script for every arm; cost is the ledger, never a price-sheet estimate.
- run
- 01788984707065955750-e95ccde6 · 2026-09-09
- model
- openrouter/poolside/laguna-xs-2.1 (alias laguna-xs-2.1-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- 04a187d444a3f917b8e8642301f45cc28ed4769c
- harness image
- sha256:b1963fc83e1be63c3b25ca7688fac05a5ed9e6a7518333c7fe3cf6da03880dbb
- gateway config
- bb22fcd7bad15793e5814f4abbd4dfba760ea3c76a3d8a69d7ec935a3ad2df7e
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- limits
- max_tokens=not set · max_cost_usd_per_instance=2 · attempts_per_instance=1 · temperature=1