Set it up yourself
Two promises live on this page, and each has a status.
Every published number can be checked without our infrastructure. Kept. Every run is pinned, its export is committed in this site, and its full artifacts and prediction file are downloadable from a tagged release, so any run can be re-graded with the official harness on a laptop. The one thing held back is the gateway's request and response bodies; the trajectories carry the conversation.
The framework can be stood up in a fresh AWS account with low effort. Measured. On
2026-09-12, in a new account in a different region, with docs/SETUP.md as the only instructions:
72 minutes from an empty account to run-ready, two graded runs, and back to empty, with no hand
intervention. The record is below.
Check a number with no infrastructure
Three layers, each needing less than the next.
From this site alone. Every provenance block carries the framework commit, the harness image
digest and the agent CLI version baked into it, swebench 5.0.2, the dataset revision, the
per-instance image digests, the gateway configuration hash, the model as the router resolved it
with the alias's sampling and reasoning parameters, the limits, and the network posture. The
site's own data holds, per run, the export (verdicts, tokens, cost, timing per attempt), the
per-call ledger the Explorer replays (timing, tokens, cost and status of every inference call), and
the judge's verdicts with cited turns and its pass report.
From a bundle. Each run's release bundle holds every attempt's patch as the harness wrote it,
the normalised and native trajectories, the per-call ledger and the CLI's output, the official
harness's eval report and full test log, every judge pass, and preds.jsonl. Re-grade a run:
curl -LO <bundle url from the Downloads page>
curl -LO <SHA256SUMS url>
sha256sum -c SHA256SUMS --ignore-missing
tar xzf bundle-<run_id>.tar.gz
uv run python -m swebench.harness.run_evaluation \
--dataset_name SWE-bench/SWE-bench_Verified --split test \
--predictions_path <run_id>/preds.jsonl --run_id verify --max_workers 4
The resolve count should match the run's page, and the bundle's own eval_report.json files let
you diff any instance that differs. About 20 minutes for a 78-instance run once the images are
pulled, $0. Downloads.
Regenerated from artifacts. The judge, the leak scan and the export scrubber all run offline from the stored artifacts, so the judge findings and the integrity columns can be recomputed from a bundle rather than trusted.
Run it in your own account: the measured checklist
Fourteen steps, 0 to 13, from an empty account to a graded run and back to nothing. Each step in
the repository's docs/SETUP.md has the command, the verification, and the time and cost
measured in the validation account. Times are what the clean pass measured, not estimates.
| step | what | measured time | cost |
|---|---|---|---|
| 0 | re-grade a published run with the official harness, no infrastructure | 20 min | $0 |
| 1 | laptop tooling, test suite green | 30 min | $0 |
| 2 | one instance graded on your laptop: the official image at its pinned digest, the custom harness inside it, gold, broken and agent patches graded by the official harness | 5 min after the first image pull | under $0.05 |
| 3 | AWS account, region, credential profile | 30 min | $0 |
| 4 | quotas: Fargate vCPU, Spot vCPU, ECR storage, Lambda concurrency | 10 min to check, minutes to 2 days to raise | $0 |
| 5 | make doctor, make bootstrap: state bucket, two out-of-band secrets, settings file | 20 min | $0 |
| 6 | persistent tier and service images: VPC, Aurora, buckets, registries, queues, guard; the dataset mirror and the digest-snapshot check run inside it | 20 min (12 min apply, 8 min cached builds; a cold build adds 15 to 40 min) | ≈ $2/day from here |
| 8 | per-instance images: 10 for a trial, 500 for the split | 21 to 29 min for 10, one night for 500 | ≈ $2 per 10, ≈ $40 for 500 |
| 9 | UI and eval tiers, tunnel, dashboard | 11 min | ≈ $6/day while up |
| 10 | gates before a paid run: isolation probe, gold gate, alias pin, ceiling discovery | gate 6 min, pin 15 s, discovery 14 min | under $1 without discovery, ≈ $24 with the default discovery target |
| 11 | first run: 10 instances, judge, export; then the split | 15 min for 10; 2 h for 500 | ≈ $0.55 plus ≈ $0.04 judge for 10; $30 to 40 for 500 on a mid-priced model |
| 12 | tear down to the idle floor | 10 min | back to ≈ $2/day |
| 13 | leave nothing behind: make teardown-all, make account-empty | 15 min after step 12 | $0 from here |
Step 7 is folded into step 6. Steps 0 to 2 need no AWS account: a reader can stop after step 2
with a graded patch on their laptop and nothing else installed. Everything after step 2 spends
real money, and make doctor refuses to apply anything until tools, profiles, quotas and secrets
check out.
What is still manual
One line: creating the AWS, OpenRouter and Docker Hub accounts and their keys; MFA on root and
the account-level budget; quota-raise requests; confirming the SNS subscription email; launching
the isolation probe; setting the pacer rate and ceiling in the UI; and launching, judging and
closing runs from the UI. Everything else is a make target.
How the claim was validated
The claim "someone can pick this up and set it up" was measured rather than asserted: a
standalone new AWS account, us-east-2 instead of the originating us-west-2, a different name
prefix, and only the documents as the instructions. The rule for the clean pass was that any
deviation, a hand touch, a command not in SETUP.md, a code change, meant fix it, commit it, tear
everything down and start again from an empty account.
| first attempt | about 4 hours wall-clock, of which 1 h 20 was fixing 21 deviations as they were found; one hand intervention (releasing four drained Spot hosts from a termination hook) |
| restarts 1 to 4 | each stopped at a further deviation that became a fix: a quota figure, a type error in a test, a missing key for the alias pin, stale local images pushed under the current tag, a race between secret deletion and the empty-account sweep |
| restart 5, the clean pass | steps 1 to 13 straight through with no deviation and no hand touch: 72 min from empty to run-ready; a Claude Code run of 10 instances (5 resolved, $0.55) with a judge pass; a custom-agent run of 10 (7 resolved, $0.56); make down, make teardown-all, and an empty sweep. 2 h 51 min in all, including about 50 min of operator time between steps |
| findings recorded | 27, every one with its cause, its fix and its commit, in docs/FRESH-ACCOUNT-LOG.md |
| dollars | the API-side spend above; the account's own bill from Cost Explorer: $3.05 for 2026-09-12, the day of the clean pass (ECS $1.13, Aurora $0.55, EC2 and EBS $0.99, load balancers $0.19, VPC $0.15); Cost Explorer was enabled after the pass, and the discovery-pass days are still being prepared |
Two things the pass does not claim. It ran 10-instance runs, not the 500, because the images for the full split take a night to build and the point was the setup, not the result; the 500-run costs and times in the table come from the originating account. And it ran two harnesses of five, Claude Code and the custom agent, on one model; the other harnesses were not re-run, since the setup is the same for all of them.
Where this stands
- Code. The validated tree is published as a single-commit repository, tagged
v1.0, with the release attached; every code reference on this site is a link pinned to that tag, and step 1 is runnable from a clone. - Data. All twelve runs' bundles and prediction files are on the release with checksums. The eval reports inside each bundle are what the official harness produced here; a re-grade from a downloaded bundle by someone outside the project has not been reported yet. If yours differs, the feedback line is for that.
- Held back. The gateway's request and response bodies. The trajectories carry the conversation; the wire-level bodies may follow later.
- Bill. The validation account's own bill is read from Cost Explorer as its days become available; the figure in the table above is what has been read so far.