Skip to main content

The LLM judge: a reader that has to cite

The problem. The grader says whether the tests pass. It cannot say whether the agent cheated, gamed the test, solved the wrong problem, lied about its own test run, or was defeated by a broken environment. A second model can read the trajectory and say so — but only if it is held to a standard that a human reviewer could check. The judge's design is mostly about that standard. Its findings on the published runs are on the Results → LLM judge page.

One pass​

The rubric​

Version 3 of config/judge_rubric.yaml, whose hash is written onto every result row (a malformed rubric fails the pass rather than judging with a default). Nine dimensions: contamination, hallucination, tool efficiency (a ratio), loop, environment problem, gave up early, test gaming, problem misread, and — added after the Laguna pilot — token efficiency: what share of the tokens and calls was avoidable, with a severity and a fixed list of causes, each cause carrying its share, one preventive recommendation and the turns it cites. Version 2 narrowed contamination after four false severity-3 flags on Claude Code: reading repository history at or before the checkout is code reading, not contamination.

The evidence rule​

  • Every dimension requires reasoning. A dimension scored above its baseline without a cited turn is demoted: the score is nulled and the row is marked evidence_missing. The judge cannot accuse without pointing.
  • Per-cause evidence is harvested before the demotion check runs — the first v3 pass demoted every efficiency score before that fix, which is why the order is fixed in code.
  • The judge never sees the gold patch. It sees the FAIL_TO_PASS test ids that were absent at the base commit, the agent's patch, and the trajectory.
  • Everything that was done to the input is recorded on the row: input_truncated, tool_output_pruned, events_elided, judge_parse_failed, missing_reasoning. A row with zero scored dimensions is the signature of an empty judgment, and is counted as one.

Inputs​

Three prune modes — full, pruned (the default), automatic. Long trajectories keep the first 20 and last 20 tool outputs whole and elide the middle, marked as such. A harness's benign boilerplate (Codex's "model metadata not found") is replaced by a harness-agnostic marker with an anchored pattern, after a review found the earlier substring match could discard prepended content. When the rubric has a causes dimension, an efficiency profile computed purely from the call ledger and the trajectory — prompt-size quantiles, cached share, spend after the last edit, unbounded and repeated file reads, test re-runs, large outputs, duplicate calls — is placed before the trajectory, labelled as hints, never as a verdict.

Resilience (ADR-0042)​

Reasoning models sometimes return an empty judgment. The cascade is: the primary call with forced JSON; a retry of the same call; a transcription of the model's reasoning content into the schema (serialise only, never re-judge); a call without JSON mode with the object extracted from prose; and finally judge_parse_failed = true. Each step is budget-gated; at most four calls per attempt. Transport retries are bounded (six, 120 s, Retry-After honoured). A 600-second timeout is a written row with no verdict at zero cost. Eight consecutive call failures trip a breaker and abort the pass as "provider down". On SIGTERM the pass stops submitting and releases its lock. The pass lock is a primary-key row used as a mutex; one database writer thread lands every result.

Selection, resumption, cost​

Candidates must have a stored trajectory (the aborted pilot had 416 of 500 without one — they are counted as not judged, not as clean). Sampling is stratified by harness × outcome with always-judge strata at 100% and a seeded generator; a resumed pass skips judged rows; retry_no_verdict re-tries timeouts and parse failures. Cost is estimated before launch from the measured mean input (116k tokens full, 79k pruned) × candidates at the judge model's price, recorded per row and per pass, and walled off from the run's own cost.

Calibration​

A reviewer can approve or deny any judged dimension, with reasoning required in both directions, pinned to the exact judged row (a re-judge is a new row). The summary this produces is an endorsement rate, and the code says so — it is not blind agreement, because the reviewer sees the judge's verdict first. The summary is global with a 20-instance threshold, and no dashboard surface consumes it yet; that is listed as a gap, not implied to exist.

Code: swebench_eval/analysis/judge.py, swebench_eval/analysis/judge_llm.py, swebench_eval/analysis/efficiency.py, config/judge_rubric.yaml.