Testing AI judges against answers graded by code

When one AI model grades another model's answers — an “LLM judge” — how do you know the grades are any good? This experiment measures that directly. It takes 172 answers whose pass/fail labels were computed by plain code, with no model involved, then asks two AI models to grade the same answers and compares their grades to the code's. Six predictions about the outcome were committed to git before any data was collected, so the results could not be cherry-picked. One prediction held. The most important finding was not about the judges at all: the experiment's own reply parser threw away 79.0% of one judge's replies, and this page reports that failure rather than hiding it.

The judges

Claude Haiku 4.5 (J-A in the run artifact) and GPT-4.1-mini (J-B in the run artifact). Both graded every answer three times.

REF-BASED

The judge is shown the correct answer.

REF-FREE

The judge is not shown the correct answer.

Item classes

original = real answers from a prior bench-lab run; P-FMT = correct answer, deliberately broken formatting; P-NEAR = plausible but wrong; P-WRONG = clearly wrong.

kappa

Agreement with the code's labels, corrected for chance agreement (1.0 = perfect, 0 = no better than chance).

false-pass

How often the judge passed an answer the code had failed.

Every figure below was measured once, on 2026-08-22T14:51:45Z, against pre-registration v1.1 and a gold set (ada1ecf0f538…) whose every label is a pure function of the answer and the reference — written by code, never by a model. The run made 2064 calls and cost $0.5559. This page makes no API calls and the experiment is not re-run: the run cost real money, and more importantly it recorded a failure worth keeping. Re-running would replace that evidence, not refresh it.

The scorecard

1 of 6 predictions held — the one about cost. Four of the five failures cannot be blamed on the judges: the harness's own parser destroyed most of one judge's replies before they could be graded (see Finding 1), and those rows are marked with an asterisk. The rules allowed only two verdicts, CONFIRMED or REFUTED, and this page does not add a third after the fact.

Every prediction from the pre-registration, scored against the run.

prediction	what was predicted	verdict	basis
K1	REF-BASED kappa > REF-FREE kappa for BOTH judges (pooled, all 172).	REFUTED *	GPT-4.1-mini holds: 0.912 > 0.698. Claude Haiku 4.5 inverts — REF-BASED 0.848 against REF-FREE 1.000 — but that 1.000 is measured on 118 of 516 calls.
K2	On P-FMT (n=39) in REF-FREE, false-pass rate > 0.50 for both judges: the rubric says judge substance; the checkers reject these by construction.	REFUTED	Refuted on complete data. GPT-4.1-mini: 0.422 over a full denominator of 116, below the threshold. Claude Haiku 4.5: 0.000 (12).
K3	For both judges, both modes: agreement with gold on P-WRONG (n=40) > agreement on P-FMT (n=39). Disagreement concentrates where format and substance diverge, not where the answer is simply wrong.	REFUTED *	Holds in three of four cells. Fails on Claude Haiku 4.5 REF-FREE, where P-WRONG 1.000 (21) and P-FMT 1.000 (12) tie — both on collapsed denominators.
K4	Perturbations-only REF-FREE false-pass rate is strictly highest in reason (n=39) vs code (n=42) and extract (n=38), for both judges. REF-BASED per-category tables ship as diagnostics. Amended 2026-08-22, BEFORE ANY DATA WAS COLLECTED. The original K4 asked for kappa, and every perturbation is gold-FAIL by construction — so that subset has a single-class gold marginal and kappa is exactly 0 for any judge behavior. It could have been neither confirmed nor refuted by any run.	REFUTED *	GPT-4.1-mini holds exactly as predicted: reason 0.267 (116) > code 0.214 (126) > extract 0.000 (114). Claude Haiku 4.5 cannot — 0.000 in all three, on denominators of 3, 57 and 3.
K5	REF-FREE inter-judge kappa > each judge's REF-FREE kappa vs gold: judges share failure modes with each other more than with a checker.	REFUTED *	Inter-judge REF-FREE kappa 0.949 > GPT-4.1-mini's 0.698, but not > Claude Haiku 4.5's 1.000. The inter-judge figure itself rests on only 118 shared graded pairs.
K6	Total spend for all 2,064 calls < $1.20 (projection at committed prices ~$0.90; falsifiable if rubric/reason tokens run long).	CONFIRMED	$0.5559.

* Refuted on incomplete data: the parser discarded most of this judge's replies before grading. See Finding 1.

What the run measured, cell by cell

Do not read the two Claude Haiku 4.5 rows as that model's behavior. Its surviving calls are a selected subset rather than a sample, and every rate on those rows depends on which replies happened to parse. Each figure carries its denominator for that reason: the same column holds numbers resting on 3 and on 516.

Finding 1: the parse contract

Claude Haiku 4.5 had 815 of its 1032 replies (79.0%) discarded, none of them for a reason that has anything to do with judging. There were no provider failures and no truncations, and the other judge had no rejections at all.

The survivors are a selected subset, not a sample. The rejection rate varies along precisely the dimensions the predictions cut on:

This harness enforced a stricter output contract than the checker it was built to check. bench-lab's grader strips a Markdown fence and grades what is inside, treating the fence as a formatting habit rather than a wrong answer. This harness rejected the same thing outright. So an experiment built to test whether an AI judge grades substance or formatting was itself undone by exactly that distinction: 761 verdicts that were almost certainly fine as judgements were thrown away over their punctuation. The experiment made the mistake it was looking for.

The parse rule was not relaxed after the fact, and the reason is timing. K4 was amended before any data existed, because a dry run showed it could never have been decided either way. The parser's damage only became visible together with the results — and loosening the rule at that point would have meant changing the rules after seeing the answers. It would obviously have improved the study; that is exactly why it was not allowed.

Finding 2: the replies were not kept

The obvious secondary analysis — reparse the discarded replies with one fence stripped and nothing else changed — cannot be run. The harness stored the parse verdict and dropped the bytes. A discarded call in the artifact looks like this:

{ "itemId": "fmt-code-01", "judge": "J-A", "mode": "REF-FREE", "rep": 1,
  "verdict": "", "reason": "", "parseOk": false, "failed": true,
  "error": "reply is wrapped in a Markdown fence; the rubric asks for strict JSON and nothing else",
  "outputTokens": 50 }

Fifty output tokens were generated, paid for, and read by the parser. All that survives is a note saying they were rejected. bench-lab keeps every one of its 360 replies verbatim — which is the only reason this gold set could be built at all. judge-lab kept only rejection summaries, so the obvious follow-up — strip the fence and re-grade the discarded replies — is impossible for this run. Future runs keep every reply; this artifact predates that fix and cannot be repaired.

What the clean data shows

One judge had no rejections, so its four cells carry full denominators. These are observations, not predictions — none was pre-registered in this form, and each would need its own pre-registration before being claimed as a result.

• A reference answer changes how a judge grades. Kappa against the checker is 0.912 with a reference and 0.698 without — same model, same items, same rubric otherwise.
• Disagreement concentrates where format and substance diverge. Reference-free false-pass runs 0.422 (116) on format-broken-but-right answers, 0.075 (120) on plausible near misses, and 0.000 (120) on unambiguously wrong ones. A judge shown no reference passes a format-broken but substantively right answer 42% of the time, and passes a wrong answer never.
• By category, reference-free false-pass on derived answers: reason 0.267 (116), code 0.214 (126), extract 0.000 (114).

What none of this settles is whether the same holds for another model, because the study's second judge produced no usable cells.

What was committed, and when

The predictions were committed before the data: the commit containing them (fe38ead) is the parent of the commit containing the gold set (8253261), and the run artifact came after both. This is a weaker guarantee than bench-lab's, where the spec predates the harness entirely. Here the gold set was generated first, on purpose: two earlier drafts of the predictions quoted sample sizes that turned out not to exist. The trade-off is that the class sizes were known when the six predictions were written, and readers should weigh them accordingly.

Method

• The gold set is 172 items: distinct responses from a bench-lab run, plus derived answers built to be wrong in three specific ways. Every label is a pure function of the answer and the reference, computed by that repository's checkers.
• Two judges, two modes, 3 repetitions. Reference-free shows the judge the question and the answer; reference-based adds the gold answer. The judge never sees the model, the lane, the provider, or whether an item is original or derived.
• The rubric is fixed in the pre-registration, verbatim, and a test asserts the strings in the code still appear in that document.
• All three inputs are pinned by digest in the run artifact — the gold set, the task text and the lane sidecar — so a run names the exact files it read rather than the ones a provenance line claims.
• Spend was capped at $2.00 with a halt at $1.50; the run cost $0.5559 and never approached either.
• All 2064 calls, with their request parameters, token counts and outcomes, are in the repository. They are not reproduced here because a megabyte of markup would support a claim the tables above already make.

Where this sits next to bench-lab

Bench Lab's page says no model judges another model there — every grade comes from code. That is still true. This experiment is the other half: here models did act as judges, and the code graded the judges. Every label they were scored against came from bench-lab's checkers, so checking these judges required no second AI judge. Neither page weakens the other's claim.

The harness, the pre-registration, the gold set, the full results and the scorecard are in pigfox/judge-lab.