Skip to main content

Evaluation & grading

What it is. The official SWE-bench harness (swebench 5.0.2, ADR-0043), run by an auto-scaled fleet of EC2 workers with Docker-in-Docker, against the official per-instance images pinned by a committed digest snapshot. The verdict is SWE-bench's, not ours.

Why a second fleet (ADR-0012)​

Grading needs to build and run a container per instance; that needs a Docker daemon, which Fargate does not provide. Agent tasks need isolation and fast horizontal scale, which Fargate provides well. Consolidating onto one fleet would have meant either agents on a shared host or grading without Docker. The two workloads cannot share a fleet, and one Terraform line says so: infra/terraform/modules/ecs-service-eval-worker/main.tf:215.

Images (ADR-0020, image-environment pipeline)​

The eval harness pulls one ~1.1 GB image per instance — 500 for Verified, about 330 GB — and Docker Hub rate-limits those pulls per source IP. So the images are mirrored once into a private registry by a dedicated warm job, tagged by instance and framework commit, and every worker pulls from there. The registry is the dominant idle cost of the whole estate.

Before a series runs, every image is gold-patch validated: the target tests must fail before the gold patch and pass after it, on our workers. The Verified set was validated on 2026-09-06 in three gate runs: 499 passed on the first pass and pylint-dev__pylint-4661 failed, because its gold patch adds a dependency that the offline image could not install; a pre-downloaded wheel fixed it and the third gate passed it — 500 of 500. Instances that fail validation would be excluded from the gradeable denominator and reported, not graded.

The eval scaler​

Desired grading tasks = queue depth + grades in progress + a feed-forward on live agent tasks (patches expected within the host-boot horizon); hosts = tasks / tasks-per-host. Scale-out is immediate; scale-in waits for five minutes of lower demand and only ever terminates an idle host.

What a verdict carries​

The full test report (FAIL_TO_PASS, PASS_TO_PASS, each with success and failure lists), the test output log, the run log, and whether the patch applied — all stored as artifacts and readable from the dashboard. An unresolved verdict is always explainable from these: in a two-instance Laguna smoke run before the Laguna series, for example, django__django-11532 was unresolved because the agent's partial fix broke eighteen pass-to-pass SMTP tests — an agent outcome, not an environment problem, and readable from the eval report alone.