Downloads

Every published run ships as one bundle, attached to the v1.0-data release of the code repository: not a bucket, not git, since the bundles run to a hundred megabytes. The predictions file of each run is small and is also committed in the repository under published-runs/<run_id>/. Checksums for every bundle: SHA256SUMS.

run	harness · model	resolved	bundle	sha256	preds
b4338f67 five harnesses · 500	mini-swe-agent · minimax/minimax-m2.5	379 / 500	58 MB	86cabcadd66a…	preds.jsonl
05bd05b8 five harnesses · 500	custom minimal agent · minimax/minimax-m2.5	384 / 500	50 MB	a0ba3186a2b2…	preds.jsonl
c10df654 five harnesses · 500	Codex CLI · minimax/minimax-m2.5	383 / 500	59 MB	3a7bb3d93aa5…	preds.jsonl
1f722383 five harnesses · 500	Claude Code · anthropic/minimax/minimax-m2.5	384 / 500	102 MB	b7e4ac1bda57…	preds.jsonl
020169d5 OpenCode first 500-run · search tool broken · superseded	OpenCode · minimax/minimax-m2.5	388 / 500	73 MB	a539e30cc8bc…	preds.jsonl
21ebff89 78-instance experiment · arm A · Laguna · plain	OpenCode · poolside/laguna-xs-2.1	58 / 78	27 MB	89a8235972e7…	preds.jsonl
e95ccde6 78-instance experiment · arm B · Laguna · preset v2	OpenCode · poolside/laguna-xs-2.1	55 / 78	18 MB	03643dc8a393…	preds.jsonl
883d2571 78-instance experiment · arm C · MiniMax · plain	OpenCode · minimax/minimax-m2.5	59 / 78	9 MB	cd253caf0c66…	preds.jsonl
39ff3720 78-instance experiment · arm D · MiniMax · preset v2	OpenCode · minimax/minimax-m2.5	60 / 78	8 MB	68e761d43f1b…	preds.jsonl
f8f44aeb 78-instance experiment · arm E · MiniMax · mini-swe-agent · plain	mini-swe-agent · minimax/minimax-m2.5	58 / 78	8 MB	8ee7eaf53fd3…	preds.jsonl
c40be1f7 78-instance experiment · arm F · Laguna · mini-swe-agent · plain	mini-swe-agent · poolside/laguna-xs-2.1	55 / 78	19 MB	e4ac0f819b5d…	preds.jsonl
0c9303b3 five harnesses · 500	OpenCode · minimax/minimax-m2.5	385 / 500	61 MB	340fbfa1a63a…	preds.jsonl

Verify and re-grade

sha256sum -c SHA256SUMS --ignore-missing        # after downloading a bundle and SHA256SUMS
tar xzf bundle-<run_id>.tar.gz
uv run python -m swebench.harness.run_evaluation \
  --dataset_name SWE-bench/SWE-bench_Verified --split test \
  --predictions_path <run_id>/preds.jsonl --run_id verify --max_workers 4

The resolve count should match the run's page. The eval_report.json files in the bundle are what that command produced here, so a per-instance diff against your own run is one jq away. Each bundle's README.md repeats the provenance block, the expected count and this command.

What a bundle holds

Per attempt, on the harness side:

file	what
patch.diff	the agent's patch, byte for byte as the harness produced it (an empty file is "no patch")
trajectory.jsonl	the normalised trajectory: one turn per line with role, content and tool calls
native_trajectory.json	the harness's own log in its own format: mini-swe-agent's JSON, Codex's rollout, Claude Code's session log, OpenCode's export; the custom agent has none
llm_calls.jsonl	every inference call: timing, tokens, cost, status, the provider that served it; no request or response bodies
harness_stdout.log	the CLI's stdout and stderr for the attempt

Per attempt, on the eval side, straight from the official harness: eval_report.json (verdict and per-test status), test_output.txt (the full test log), run_instance.log, and resource_usage.json (the eval container's CPU and memory peaks).

Per run: every LLM-judge pass over it, one JSON per judged attempt under judge/, a README.md with the provenance block and the re-grade command, and preds.jsonl.

Attempts beyond the one the results page counts, retried after an infrastructure failure, are included where they exist; the export and preds.jsonl use the attempt the export recorded.

What was removed before publishing

The bundles are the artifacts the framework wrote, copied, with strings that belong to the rig taken out. Three kinds, all from attempts where the agent printed its environment or its host name, which happens on two Django instances and one pylint instance whatever the harness: VPC-internal host names (135), the per-run gateway key, revoked when the run closed (34), and a queue receipt handle (34). 203 replacements across eight of the twelve runs; four runs needed none. Patches were never rewritten. The raw and scrubbed trees were kept side by side with a per-line record of every change, and the scrubbed trees were scanned for secrets: 179 matches, every one an upstream test fixture, a git commit hash or a substring of a random provider id.

Not in the bundles: the gateway's request and response bodies. trajectory.jsonl carries the conversation as the harness recorded it; the wire-level bodies stay unpublished for now.

What is in the site's own data

• data/runs/<run_id>.json: one per run, the export contract every page renders from.
• data/timelines/<run_id>/: manifest with file hashes, fleet series, instance lanes, discovery steps, judge verdicts and report.
• static/timelines/<run_id>/calls/: one call timeline per attempt, fetched on demand.
• data/bundles.json: the table above, with full checksums.

The manifest's hashes let a reader confirm that the site's data matches a bundle. Code: psdhali/agent-evals.