Admission control: keeping 500 agents inside a provider's limits

The problem. A coding agent makes a call every few seconds, each carrying tens to hundreds of thousands of prompt tokens. Five hundred of them at once is a fleet-scale token stream aimed at one provider pool that other people are also using, whose real limits are not published and move by the minute. Ask for too much and the provider sheds load with 429s, every harness retries in its own way, and the run's cost and wall-clock stop meaning anything. Ask for too little and a run takes a day. The framework treats this as a control problem with three layers, coordinated through one Redis hash per model pool.

L1 — the pacer: one atomic decision per call

Every call the agent makes passes through the local proxy shim in its own container. Before forwarding, the shim asks the pool's pacer for admission. The decision is one Lua script in Valkey, so five hundred shims never race:

1. Refill the token bucket (r_tok tokens/s up to c_burst) and the request bucket (r_qps up to c_req) from Valkey's own clock; prune finished in-flight entries and stale waiters.
2. Estimate the charge: prompt characters ÷ κ (a chars-per-token ratio learned per pool, EWMA, starting at 4.89), split into uncached and cached; charge = uncached + cached × cached_weight, the weight discovery measured for how much a cache-read actually costs the pool.
3. Fairness: waiters are ordered by first-denied time minus charge ÷ r_tok, so a large call is treated as having already waited its own refill time and small calls cannot starve it.
4. Admit only if tokens ≥ charge + the head waiter's estimate (clamped to c_burst, so an oversize call can still go alone), a request token is available, and in-flight tokens + this call ≤ k_inflight. The first failing axis names the denial; the shim sleeps a jittered fraction of the predicted wait and asks again.
5. Settle on release: the real prompt size replaces the estimate, refunding or debiting the bucket; κ is updated.

A call that cannot be admitted within its harness's hold cap (45 s to 240 s depending on the harness, each verified against that CLI's own HTTP timeout) is answered with a synthesised 429 carrying retry-after: 30, so the agent backs off instead of timing out. If Valkey is unreachable the shim falls back to local minimum-gap pacing — it never runs unpaced.

Provider overloads are told apart from our own 429s by one header: the gateway's self-imposed 429 always carries retry-after; in 2,651 real provider overloads observed, none did. A real overload is retried by the shim up to six times with min(30·n, 90) seconds of jittered backoff, releasing and re-acquiring admission around each retry (a re-send is a new arrival), and counted into the pool's overload window. The agent never sees it.

L2 — the planner: how many tasks may run

Every 15 s the dispatcher decides how many agent tasks the fleet may hold, per model alias:

• Ground truth is ECS ListTasks plus a scan of the live progress keys, never its own memory.
• Demand curves per (pool, harness) — tokens per call as a function of turn (capped at 235,929), latency, turn period, and a survival table — are fitted from the ledger. A pool without its own curve borrows one, with a latency floor and a hard ceiling of 30 tasks (binding = borrowed_curve).
• Projection over a 0–90 minute horizon of arrival tokens/s (cache-weighted), in-flight tokens and calls/s for the fleet plus each candidate extra task.
• Ceiling = fleet + booting + the largest extra that keeps arrival ≤ 0.9·r_tok, in-flight ≤ 0.9·k_inflight and qps ≤ 0.85·r_qps. Margins are halved when the pacer config is older than six hours. Per-alias ceilings are summed. The static cap (the account's Fargate quota, 145 tasks) wins in every mode; an operator override replaces the projection but never the static cap.
• Launches are staggered 2 s × U(0.5, 1.5) so five hundred first calls do not land in one second.

Every tick writes its whole decision — mode, binding constraint, projections, budgets and their source — to a Valkey key the dashboard reads and the capacity observer persists.

AIMD — what happens when the provider pushes back

Additive increase, multiplicative decrease, the same shape as TCP:

signal	reaction
any provider 429 in the trailing 60 s	cooldown: no new launches (nothing running is ever killed); on entry, recovery sets r_tok and r_qps to max(0.5 × seed, 0.85 × the rate actually admitted in the window), stamped as a fresh seed
paced back-pressure: > 5% of calls waited over 2 s, or any hold-cap timeout, or the head of the queue waited > 5 s	block new launches immediately; two clean ticks to release
a clean window at an armed peak, arrival > 0.8 × 0.9 × r_tok	growth +5% (hard maximum) up to 1.5 × seed

Growth is applied to one alias at a time; recovery is per alias — in a mixed fleet the other aliases can only shrink. That asymmetry is deliberate: the provider's pool is shared with strangers and its edge moves about twice a minute, so the loop is biased toward backing off.

The loop on the Laguna xs 2.1 pilot (OpenCode, 500 launched, 78 graded)

run 21ebff89 · alias laguna-xs-2.1-opencode · one point per 30 s

Top: agent tasks in flight against the planner's ceiling and the operator's override (the ceiling never exceeds the override, and drops below it whenever the loop is in cooldown or paced back-pressure). Bottom: the pacer's token budget (r_tok, tokens/s) against the planner's projected arrival rate for the fleet it has, with provider 429s in the trailing minute as bars. The budget starts at the discovery seed; every overload burst is followed by it stepping down (recovery to max(0.5 × seed, 0.85 × the admitted rate)); clean windows step it back up by 5%. When the projection sits above the budget, the planner's answer is a smaller fleet — which is what the top panel shows. Read from the committed fleet.json — no smoothing.

Reading the chart: the operator raised the override in steps while the planner's ceiling followed the budget; each burst of provider 429s (bars) is followed within one tick by the token budget stepping down, and the ceiling with it; clean stretches walk the budget back up. The provider was capping this pool at roughly 50 requests per minute for our key — the request axis, not the token axis, was binding, which is why the operator lowered r_qps by hand and let the loop find the rest.

L3 — discovery: where the seeds come from

Nobody publishes the number the pacer needs, so the framework measures it once per provider pool, before the first paid run, under its own key with a $30 backstop and the provider pinned:

• Phase A — burst edge: double the burst until the first 429.
• Phase B — ramp to strain from a 60-task target: a real 429 is hard strain; latency ≥ 1.2× the baseline is soft strain.
• Phase C — the request axis (calls per second) separately from tokens.
• Phase D — the cache-hit axis: how much a cached prompt token costs the pool, which becomes cached_weight.

The seeds are conservative — c_burst = half the edge, k_inflight = 0.95 × the edge, everything floored at one full-window call — and persisted to Aurora, because Valkey is torn down with the eval tier every night. At launch, restart and supervisor start they are copied from the pool to the run's alias.

What the operator can change, live

Per run: the task cap, the ceiling override, the ramp (≤ 5%), the cooldown, and whether the planner is enabled at all. Globally: the borrowed-curve cap, the growth cap, the utilisation targets, the eval fleet's limits. Per alias: the five pacer constants — an edit re-bases the seed and re-stamps its age, and can be persisted to the pool. Every edit is written to an audit table with a reason.

Decisions and their records

decision	why	rejected	record
arrival-paced admission per pool, one atomic script	five hundred independent retry loops cannot share a limit fairly	fixed concurrency; trusting retry-after alone	ADR-0041
a shim in every task	four of five harnesses cannot enforce a budget; the shim also meters	polling the gateway's spend log (lag, and a race with the last call)	ADR-0019, ADR-0037
planner inside the dispatcher, not a separate service	it needs the queue, the progress keys and the launch path in one place	the SQS-depth autoscaler	ADR-0030, ADR-0006 (superseded)
growth capped at 1.5× the seed, +5% per clean window	the pool is shared and its edge moves; the seed is a measurement, not a guess	unbounded growth to the first 429	ADR-0041
the enforced spend cap is the provider key's limit	the gateway's own budget counter over-counted 3.5×	gateway max_budget	ADR-0044

Code: swebench_eval/gateway/pacer.py (L1), swebench_eval/gateway/local_proxy.py (the shim), swebench_eval/orchestrator/control_plane/harness_dispatcher.py (L2), swebench_eval/orchestrator/control_plane/ceiling_discovery.py (L3).