Decision record
Every non-obvious choice in the framework has an architecture decision record: the context, the decision, the alternatives rejected and why. There are 44 of them. They are internal working documents and are not published; the design pages on this site describe the system as built, and this index summarises the decisions those pages rest on. The ones cited on this site:
| ADR | decision | rejected |
|---|---|---|
| 0002 | a single gateway tier (LiteLLM, replicated behind a load balancer) in the VPC for every call | per-harness provider keys; trusting harness usage counters |
| 0006 | SQS-backed job dispatch with a hand-written closed-loop autoscaler (superseded for the harness pool by 0030 and 0041; still the shape of the eval pool) | ECS managed scaling |
| 0012 | harness workers on Fargate, eval workers on EC2 | one consolidated fleet |
| 0017 | uv for dependency management with a committed lockfile | pip |
| 0018 | scale shape for 20k concurrent instances: live state in Valkey, VPC sizing, dashboard design | Postgres as the live store |
| 0019 | a local proxy shim per task for budget enforcement | polling the gateway spend log |
| 0020 | per-instance eval images cached in a private registry by a dedicated warm job | pulling from Docker Hub per worker |
| 0024 | an S3-dropped run config triggers dispatch, alongside POST /runs | API-only launches |
| 0025 / 0029 | repository content served by a VPC-internal git mirror over git daemon, resolved by name | EFS; HTTP CGI |
| 0028 / 0033 | harness network isolation: no egress, one always-isolated posture, interface endpoints on the eval tier | allow-listed egress; a switchable posture |
| 0030 / 0032 | one Fargate task per instance, launched by sharded dispatchers; the task receives a job reference and reads the instance itself | a pooled worker per harness; the full job in the queue message |
| 0035 / 0044 | per-run keys; the spend cap is the provider key's limit, the gateway key carries no max_budget | shared keys with soft budgets; a budget monitor that observes rather than enforces |
| 0037 | measurement is a first-class plane: the shim meters every harness | trusting each harness's own counters |
| 0038 | contamination detection per instance, and the denominator rule (both denominators always, with intervals) | a single headline rate |
| 0041 | arrival-paced admission for shared-pool LLM capacity, seeded by a discovery probe | fixed concurrency; provider retry-after alone |
| 0042 | empty-judgment recovery cascade for a reasoning-model judge | dropping empty verdicts silently |
| 0043 | track upstream SWE-bench 5, pin dataset revision and image digests | staying on 4.1 |
note
The records themselves stay internal. What the system was required to do is on the Requirements page; how it is built is on the design pages.