Three models, one fixed suite, and a prediction written first
Every figure below was measured once, on 2026-08-20T04:13:15Z, against a committed suite 1.0.0 of 40 tasks whose answers are checked by code rather than judged by a model. The whole run cost $0.0433 across 360 calls. This page makes no API call — re-running on each visit would spend real money per pageview and show a different scoreboard every time, underneath a prediction that is fixed.
The scoreboard
| model | graded | accuracy | median TTFT | tokens/sec | total cost | USD per correct |
|---|---|---|---|---|---|---|
claude-haiku-4-5-20251001 |
120 | 98.3% | 568 ms | 360 | $0.01582 | $0.000134 |
gpt-5-mini |
120 | 100.0% | 1579 ms | 1551 | $0.02338 | $0.000195 |
gpt-4.1-mini |
120 | 100.0% | 469 ms | 80 | $0.00408 | $0.000034 |
Do not read the tokens/sec column across models. The run showed that metric is broken for reasoning models: it divides all output tokens — including the ones spent thinking — by the window after the first visible token, and thinking happens before that. The number is left exactly as the committed definition produced it, and the defect is written up below rather than quietly fixed.
What was predicted, and what happened
The predictions were committed before the harness, the suite and the run existed. 3 of 6 were refuted. That is the point of writing them down first: a prediction that cannot lose is not a prediction.
| prediction | predicted | measured | verdict |
|---|---|---|---|
| P1_costPerCorrectOrder | C < B < A | C < A < B | REFUTED |
| P2_ttftOrder | A < C < B | C < A < B | REFUTED |
| P3_throughputLeader | A fastest | B < A < C | REFUTED, AND THE METRIC IS AT FAULT AS WELL AS THE PREDICTION |
| P4_accuracyBandAndFloor | all lanes above 70%, within 15 points of each other | A 98.3, B 100, C 100 | CONFIRMED |
| P5_reasoningSpendShape | gpt-5-mini spends reasoning tokens on `reason`, near zero on `extract` | map[code:3584 extract:3584 reason:1920] | HALF CONFIRMED, HALF REFUTED |
| P6_totalSpend | under $3.00 | 0.0433 | CONFIRMED |
P1_costPerCorrectOrder: C is cheapest as predicted, but A and B are the other way round. gpt-5-mini spent 9,088 reasoning tokens across the run, billed at the output rate, and that outweighed its cheaper input price. The prediction reasoned that B would have the highest effective output volume and then failed to carry that through to the ordering.
P2_ttftOrder: B slowest as predicted, and for the predicted reason. But C beat A rather than the reverse: the assumption that the small fast model would lead on first-token latency did not hold against this provider on this run.
P3_throughputLeader: B measures fastest at 1551 tokens/sec against A's 360, which is an artefact of how SPEC.md 3 defines the metric rather than a fact about the models. See errata: reasoning tokens are counted in the numerator but are emitted BEFORE the first visible token, so they fall outside the window in the denominator.
P5_reasoningSpendShape: It did spend reasoning tokens, so the adaptive half holds. The SHAPE is wrong and inverted: spend was near-uniform across categories and `reason` drew the FEWEST, not the most. Extraction cost it as much thinking as code did.
P6_totalSpend: $0.0433, about 70x under the prediction and 460x under the $20 cap.
What the run showed was wrong with the method
SPEC.md 3 — the tokens/sec definition is wrong for reasoning models
methodology defect, found by the run
The specification said: tokens/sec is output tokens divided by (total wall time - TTFT).
The problem: On a reasoning model the two halves disagree about which tokens they cover. The numerator is the provider's OUTPUT token count, which includes reasoning tokens. The denominator is the window AFTER the first visible token — and reasoning tokens are emitted BEFORE it. So tokens generated during the TTFT window are divided by a window that excludes them.
The effect: gpt-5-mini reports 1551 tokens/sec against gpt-4.1-mini's 80. That is not a 20x throughput advantage; it is the metric crediting pre-TTFT work to a post-TTFT interval.
What is actually true: Throughput is only comparable between lanes when neither spends reasoning tokens. On this run only A and C are comparable, and there A leads at 360 against C's 80.
A v2 metric divides VISIBLE output tokens by the post-TTFT window, or reports throughput only for calls with zero reasoning tokens. NOT changed retroactively: the figure in this artifact is what the committed definition produced.
The suite is near-saturated at this difficulty
suite limitation, not a defect
The problem: Two of three lanes answered every graded call correctly. A benchmark whose tasks are all solved cannot rank the things that solved them.
What is actually true: This run distinguishes cost and latency well and capability barely at all. The one accuracy signal is lane A missing rsn-08 twice.
Suite v2 needs harder tasks. The version is committed and results name it, so a later comparison across versions stays honest.
The pre-registration file was not edited to match any of this. A prediction quietly corrected once the data arrives is worth nothing, so the corrections live beside the measurements that forced them.
The one wrong answer
A answered task rsn-08 as
60000 instead of 60, 2 times out of 3.
The only wrong answer anywhere in the run. The prompt fixes the unit conversion explicitly ('Treating 1 GB as 1000 MB') and asks for seconds, so 3 GB / 50 MB/s is 60. The answer is off by exactly 1000, which is the conversion the prompt pinned. Checked as a possible suite defect per MODEL.md P4 and it is not one: the other two lanes answered 60 on all three repetitions, and this lane answered 60 on one of its own.
Method
- Suite 1.0.0, 40 tasks: 14 code, 13 extract, 13 reason, each with a gold answer and a checker that is a pure function. No model judges another model.
- 3 repetitions per task per model. One run cannot tell a model that is reliably right from one that is right by chance.
- Temperature is 0 where the provider allows it. It does not always:
gpt-5-minirejects temperature 0 outright and ran at the provider default, so it is not compared like for like. - Cost is computed from published per-token prices committed as constants, read on 2026-08-19. Reasoning tokens are output tokens and are billed as such.
- All 360 calls, with their request parameters, token counts, timings and
grades, are in
bench/results.jsonin the repository. They are not reproduced here because a megabyte of markup would support a claim the table above already makes.
What this does not measure
- This is not a general capability ranking. It is 40 tasks sampling three narrow skills, chosen because they can be graded without judgement. A model better at things this suite does not contain will look worse here, and that is a property of the suite.
- The accuracy column barely discriminates. Two of the three models answered every graded call correctly. This run separates cost and latency well and capability almost not at all.
- It is not advice about which model to use for anything that does not resemble these tasks.
- No model judged any answer. Every grade came from code.
The harness, the suite, the pre-registration and the full results are in pigfox/bench-lab.