Skip to main content

Capabilities

Everything the rig does today, grouped by what an operator is doing at the time. Each row names the page that explains the mechanism. Nothing here is planned or aspirational; a feature that exists only as a schema column with no writer is listed under not yet at the bottom.

Launching and configuring a run​

  • Start a run from the dashboard: dataset split and instances (all, pasted ids, or filtered by repository), harness (OpenCode, Claude Code, Codex CLI, mini-swe-agent, a custom minimal agent), model (any OpenRouter model registered as a rotatable, provider-pinned alias), limits (turns, dollars per instance, context window, timeout, attempts per instance) and the run's budget cap. Harness adapters, Inference gateway.
  • Validate the images before launch: the gold-patch gate proves every instance image grades the gold patch RESOLVED, offline, before a model dollar is spent. Images.
  • Per-run harness instructions, with presets per harness, appended to the agent's system prompt and logged per attempt. What a prompt can buy.
  • Cost previews before the optional spends: the judge pass and the discovery probe each show their estimated cost and wait for a confirm before anything is spent.
  • Scripted launch: drop the run configuration as JSON in S3 and a Lambda triggers the same path. Orchestrator.
  • Per-run keys: a provider key with the budget as its hard limit and a gateway virtual key, both revoked at close; alias rotation without a restart. Inference gateway.

Running​

  • One isolated container per instance on Fargate, no default route, non-root, no secrets in the task; up to the account's Fargate quota concurrently. Architecture, Integrity.
  • A local proxy shim in every task that meters every call, enforces turn, token, dollar and context-window caps, waits for pacer admission and retries provider overloads so the harness never sees a 429. Architecture → the shim.
  • Two-level admission control: a per-call token-bucket pacer per model pool, and a planner that sizes the fleet from projected demand, with AIMD recovery on provider 429s, a growth cap, cooldown and paced back-pressure. Admission control.
  • Ceiling discovery probes that seed the pacer per provider pool from a measurement, persisted across tear-downs.
  • Live operator controls on a running run: pause and resume per pool (harness, eval, gateway); abort with real sweep semantics; close; restart selected instances as attempt N+1; regrade an attempt; edit limits at run, global and pacer scope with an audit trail; raise or lower the concurrency ceiling; change the ramp and the cooldown. Controls.
  • Fail-closed control state: if the control snapshot is stale every pool reads as paused and every action is disabled until the store is confirmed. Observability.

Watching​

  • Run list with activity-gated polling; run page with progress by phase (expected and gradeable as two numbers), live per-instance turns, tokens and cost, the pacer ledger (buckets, wait queue, overloads), the autoscaler's decision record (binding constraint, projections, recovery, growth), the eval scaler's decision, queue depths including dead letters, cost so far, and the control-state banner.
  • Instance page: per-phase timing (queue, provision, image pull, boot, repository prep, agent, patch extraction, upload; eval queue, fetch, test, upload); the per-call table with the wall-clock decomposition (preflight, pacer wait, retries, backoff, provider latency); the live LLM-call view while the attempt runs; the trajectory viewer; the harness's native trajectory; stdout; the patch; the eval report; the test output.
  • Capacity charts for both pools, model ceilings with discovery and manual estimates, and a timeline export per run. Observability.

Grading​

  • Official SWE-bench 5.0.2 harness in the official per-instance images, pulled by digest from a private registry; verdict precedence OOM → infra → timeout → invalid → apply-failed → resolved; contamination strip before grading; touches_test_files; gold-patch similarity on the eval side only; grade-invalid detection; resource sampling; task scale-in protection; liveness heartbeat; orphan sweep. Eval scaling and grading.
  • The eval fleet scales itself on Spot EC2 with a feed-forward control law and damped scale-in; zero hosts when idle.

Judging​

  • A rubric-driven LLM judge over the trajectories — nine dimensions, cited evidence required, a score without evidence demoted; three prune modes; an efficiency profile from the ledger placed before the trajectory; a resilience cascade for empty judgments; a pass ledger with a lock; a live pass view; a synthesis report; calibration reviews with mandatory reasoning; a leak pass that runs first. The judge, findings.

Recording and publishing​

  • Per call: the full ledger row — tokens including cached and reasoning, cost with its source, latency tiers, pacer fields, retries, status. Per attempt: 56 columns including timing, honesty fields and artifact pointers. Per run: provenance (framework commit, harness image digest, dataset revision and image-digest pin, gateway configuration hash, model as resolved, limits), the summary with both denominators, 30-second timeline ticks, capacity decisions, events and limit edits. Observability.
  • Artifacts in S3 per attempt; an export endpoint with Wilson intervals and pass@k; a timeline export with scrubbing that fails on a hit and decimation; raw bundles. This site and the Explorer read only files. Downloads.

Operating cheaply​

  • Three Terraform tiers: the persistent tier is never destroyed; the UI and eval tiers are torn down nightly. Spot pools at zero by default; a nightly scale-to-zero guard that refuses unless the database, the queues and ECS are all idle; a budget and a billing alarm; an image refresh path that rebuilds only the framework layer. Cost, Images.

Not yet​

  • A cost estimate before a run is launched: not built. The run's budget cap is the control, and the cost is exact from the ledger once the run is going.
  • Per-run compute cost: schema, API and export carry the columns; nothing writes them. The site estimates it from the recorded usage at list prices (Cost).
  • The calibration summary endpoint exists; no dashboard surface reads it.
  • Alarms: two exist (billing, NAT); queue depth, dead letters and task failures are read-path rules in the dashboard, not alerts.
  • Test-gaming through configuration files outside the paths the static check covers (audit item G4) is not detected.