Skip to main content

Orchestrator & dispatch

What it is. Four long-running services and one queue: the API (launch, control, read models), the run-supervisor (heartbeats and the reaper, the eval-fleet scaler, the capacity observer, key recovery), the harness-dispatcher (admission and the planner — it lives on the eval tier next to the gateway and is torn down with it) and the results-writer (landing rows, applying migrations). Together they own the run's state machine.

A launch, step by step​

  1. POST /runs validates the request, mints the run's provider key with its spend cap and the run's gateway key, writes the runs row and the pending instance_results rows, and enqueues one job per instance. The same launch can be triggered by dropping the run config into S3 (ADR-0024).
  2. The dispatcher (swebench_eval/orchestrator/control_plane/harness_dispatcher.py) pulls jobs and launches Fargate tasks up to a ceiling that is the minimum of the static cap, the per-run cap, the operator's override and the planner's projection. The task receives a job reference, not the job (ADR-0032), and the message stays in the queue until the task deletes it.
  3. The planner ticks every 15 s. It groups live tasks by gateway alias, projects each group's arrival tokens/s, in-flight tokens and calls/s from a fitted per-harness curve, and finds the largest fleet whose projection fits the alias's pacer budgets. It reports which constraint binds (arrival_budget, qps_budget, paced, cooldown, at_capacity, ceiling_override…) and grows the budgets 5% per clean window up to 1.5× the discovered seed.
  4. On a provider 429 the planner freezes new launches for a cooldown and lowers the budgets toward the seed; the pacer keeps admitting at the lowered rate. Nothing running is killed.
  5. Each task's progress key in Valkey drives the dashboard; the task's terminal row drives the ledger; a capacity observer writes one durable snapshot per pool per 30 s for the timeline.

Controls the operator has​

Pause and resume per pool (harness, eval, gateway); abort with scope; per-run task cap and ceiling override; the pacer's five constants per alias, editable live with an audit reason; and a close step that revokes both keys.

Things that were learned the hard way​

  • A planner that could not see booting tasks under-counted the fleet and held every ramp back; booting tasks are now modelled as fresh candidates.
  • A stale pacer config (older than six hours) halves the planner's utilisation; a config seeded four days earlier once produced a ceiling of 0 and a run that never started.
  • Recovery after a burst of 429s halves the token rate even when the 429s were a request-count cap; the operator can re-base the token rate live.