Eval scaling and grading

The problem. Grading is the other half of a run and the half nobody watches. A patch has to be applied in the official per-instance image, the target tests run, and a verdict landed exactly once, while five hundred agents finish at unpredictable times and a Spot fleet boots five minutes after you ask for it. The rules below are what makes "resolved" mean the same thing on every row.

An attempt is two rows, one writer, one direction

Every attempt is two rows in instance_results, keyed (run, instance, attempt, phase): a harness row and an eval row. One process — the results-writer — writes both, and every write goes through a monotonic guard on state_rank: PENDING 0 < DISPATCHED 1 < RUNNING 2 < ABANDONED 3 < terminal 4. A late message can never move a row backwards, a duplicate verdict is harmless, and ABANDONED sits below terminal on purpose: a straggler verdict that arrives after the reaper gave up wins over the reaper.

Who writes what is fixed: the dispatcher owns DISPATCHED; the harness worker owns HARNESS_RUNNING and its terminal set; the results-writer seeds the eval row and enqueues the eval job — and only when its own upsert reports that the row actually advanced (an earlier version tested "was this an insert", which never fired for rows seeded at launch, so no eval job was ever enqueued: the one production blocker the state audit found); the eval worker owns EVAL_RUNNING and the verdicts; the reaper and the abort sweep own ABANDONED and the aborted states.

Two things never happen automatically: a retry, and a silent drop. An operator can /restart selected instances (the whole pipeline as attempt N+1, new spend) or /regrade (eval only, against the newest captured patch). Both are recorded with a reason and excluded from the resolve-rate denominator, which is always attempt 1. Image validation is the same path with the gold patch.

The eval fleet's control law

Eval workers are EC2 hosts with Docker-in-Docker on a Spot auto-scaling group (min 0, max 16), because grading needs a real Docker daemon. A daemon thread in the run-supervisor ticks every 30 s:

• Feed-forward: 0.35 × the number of agents currently running is added to demand, because a host takes about five minutes to boot and patches arrive in waves; waiting for the queue to fill first means every wave queues.
• Packing is the reservation: an eval task reserves 2048 CPU units and 3584 MiB, which is exactly four per 8-vCPU host with zero slack; the grading container itself gets a 3 GiB cgroup and no CPU cap. Measured peak memory per grade is median 248 MB, max 1.8 GB; a cold image pull is 100–140 s.
• Scale-in is damped (two ticks for hosts, ten for tasks) after a 60-second rule was seen to churn idle workers three times in thirty minutes; only hosts with zero running tasks are ever terminated, and a busy worker carries task scale-in protection so ECS cannot pick it as a victim.
• ECS managed scaling is off after it held a paid host at zero tasks; the loop above is the only thing that moves the group.

Grading: the official harness, unchanged

The worker pulls the instance's -inst image from the private registry by digest, re-tags it to the exact name the official harness looks up (Docker Hub is never touched), builds the official TestSpec from the pinned dataset row, and runs SWE-bench's own run_instance unmodified. Around it, the framework adds what the official harness does not:

• Contamination strip before grading: any model-authored test file that intersects the gold test patch is dropped from the patch before it is applied. This is the control that turned the django__django-10924 false positive into a discarded result (see Integrity).
• Two log tees, because the official harness buffers the whole execution stream and a 67-minute grade was silent until it ended.
• The grading container is named so the OOM watcher, the resource sampler and the SIGTERM handler all watch the right sibling; per-grade disk hygiene, after a 500-instance gate filled a host's disk at about 35 images and abandoned 16 grades.

What a verdict may say

Precedence, first match wins:

condition	verdict	in the gradeable denominator?
grading container OOM-killed	FAILED_EVAL, verdict voided	no — reported as the framework's failure
infrastructure failure (pull, daemon, disk)	FAILED_EVAL, verdict voided	no
grade exceeded its timeout	UNRESOLVED, category EVAL_TIMEOUT	yes — a hang is the patch's doing
official report invalid	grade invalid	no
patch did not apply	PATCH_APPLY_FAILED	yes
FAIL_TO_PASS all pass and PASS_TO_PASS all pass	RESOLVED	yes
otherwise	UNRESOLVED	yes
agent produced no patch	EMPTY_PATCH, never graded	yes — "the agent produced nothing" is a real outcome

Alongside the verdict, honesty fields land on the same row: touches_test_files, the stripped test paths, grade_invalid, and a gold-patch similarity computed on the eval side only (the harness never sees the gold patch). Leak detection (leak_detectable, the leaked nodes) comes from a separate offline pass, not from the live grade, and is null — "not scanned" — until that pass runs.

Failure handling and the reaper

Three rules keep rows from hanging forever: a deadline rule (harness timeout + 20 minutes with no live task and no live progress key), a dead-letter rule (DLQ consumers persist the message to S3 and emit ABANDONED), and a never-dispatched rule for the harness phase with three independent checks. On SIGTERM a worker hands its message back with visibility zero and force-removes the grading container; the orphan sweep skips containers with live execs, after a neighbour once swept a live 67-minute grade. One gap is acknowledged in the code: an eval row whose job never reaches a worker is reaped only by an abort.

What is not measured

Compute cost per run — the Fargate minutes and the Spot hours — has columns in the schema, the API and the export, and nothing in the framework writes them; the export's field stays null. The per-run compute figures on the Cost page are the site's own estimate, computed from what the run did record (billed task seconds, grading slots per tick, wall-clock) at list prices, and are labelled as an estimate rather than a measurement.

Code: swebench_eval/orchestrator/control_plane/results_writer.py, swebench_eval/orchestrator/control_plane/eval_autoscaler.py, swebench_eval/evaluation/swebench_runner.py.