The two numbers that decide whether you can trust a model

Update, 2026-10-02: the three judges named in this post, grok-4-6, z-ai-glm-5-3 and deepseek-v4-flash-0731, are no longer in our judge roster. Their measured results stand as dated records. The current roster is on the Judges page, xlnc.co/judges.

Your engineering team runs the same evaluation on three frontier models. All three ace the hardest reasoning tasks you throw at them. All three score within a point of each other on the leaderboard. You pick the cheapest one and ship.

Two numbers decide whether you can trust a model as a grader: its capability, the highest reasoning level it reliably reaches, and its judge severity, how far its verdicts lean from the reference at each level. In our early readings, grok-4-6 rated work 1.57 stages above the reference at level 9 and 1.03 stages below it at level 12, with at least 225 judgments per level, so a single average hides the lean.

Your team runs the same evaluation on three frontier models. All three do well on the hardest tasks you have. All three land within a point of each other on the leaderboard. You pick the cheapest one and ship.

Weeks later, a customer escalates. The model you deployed as a grader passed work it should have failed, on routine items your own reviewers would have caught. You rerun the eval. Same scores.

That is the puzzle this post answers. Nothing in the eval was wrong. It measured one number, and the decision turned on a second.

Two numbers, not one

Every model we measure gets two readings, each with its own error and its own date.

The first is capability: the highest order of reasoning complexity the model can reliably produce, on the Model of Hierarchical Complexity (MHC) ruler. The second is judge severity: how far the model's verdicts lean above or below the reference, in stages, when it grades the work of others.

These are different properties. One says what a model can reach. The other says how far its judgments drift from the reference, and in which direction. Neither is a verdict on quality. Both are facts you need before you route work to a model.

Most evaluation pipelines report only the first. The second is where a routing mistake hides.

Severity bends with difficulty

In this post, a positive severity means the judge rates work above the reference (lenient). A negative severity means it rates work below the reference (severe). Every figure below is early evidence, dated, with its sample size.

grok-4-6, measured 2026-08-22 across bands 8.0-12.0 (minimum n=225, SE at or below 0.07 of a level; unit: one judgment), does not have one severity. It has five.

It does not have one severity. It has five. At level 8 it rates above the reference by 1.43 stages (n = 342, SE 0.07). At level 9, by 1.57 (n = 230, SE 0.06). At level 10, by 1.07 (n = 230, SE 0.06). Then the sign flips. At level 11 it rates below the reference by 0.38 (n = 232, SE 0.03). At level 12, by 1.03 (n = 225, SE 0.02).

Easy work gets inflated. Hard work gets discounted. Same judge, same instrument, same targets. The standard errors are small next to the effects, so the bend is not noise.

Three judges, three patterns

grok-4-6 is not an outlier. Two more judges were read on 2026-08-19, in a smaller probe.

grok-4-6 is not an outlier. It is one of three distinct severity signatures on the ruler: grok-4-6 measured 2026-08-22, glm-5-3 and deepseek-0731 fresh registry entries from the 2026-08-19 probe.

deepseek-v4-flash-0731 is lenient, and more so on harder work: 1.75 stages above the reference overall (95% CI 1.63 to 1.86, n = 1,336). The lean is about 1.2 stages at level 8, dips to about 0.4 at level 9, then grows to about 2.9 at level 12. The harder the task, the more it flatters. Note, 2026-09-26: this entry has been marked stale since 2026-09-20, when its maker released a new version. We keep the readings, and the model is not used until it is measured again.

Same ruler, same targets, three different patterns. One average severity for any of these models would hide the pattern that decides whether you can trust it at the difficulty you actually face.

What capability adds

Severity only matters where a model can see the task. Capability sets that limit.

Correction, 2026-09-26: this post gave a ceiling reading for grok-4-6. That reading is withdrawn and waits on a re-screen of the new Grok 4.7 on the corrected ceiling test.

glm-5-3 and deepseek-0731 both credential at ceiling 11 with a monotonicity break at O12. Hits above the break are spurious flatten-and-enumerate and do not count.

The interaction is now visible. The model with the worst severity profile (grok-4-6) sits at the bands where the cheaper model (glm-5-3) is neutral. The routing choice is a real trade, not a no-brainer.

Every reading here is early evidence, not yet certified. Entries go stale after 90 days or a new model version, whichever comes first, and stale entries are marked, never deleted or routed on. The readings in this post are dated 2026-08-19 and 2026-08-22, so they will need re-measuring for use after mid-November.

Two items at level 8 sit in the wrong place for grok-4-6 in both runs, and we disclose them as a property of the item set rather than of the judge.

Our failure-mode labels (for example, a retrieval mismatch or a refusal to commit) are early. They have not been checked against human-adjudicated cases, and their thresholds were set in advance rather than fitted to real failures.

Measure both, or trust neither

A model's capability tells you what it can reach. Its severity tells you how far its verdicts lean. The two interact level by level, and a single score hides the interaction.

If a grader is already scoring your releases, find out where it leans before the next escalation does it for you. The Design-Partner Readout takes a redacted slice of the evaluation data you already keep, traces included, and in about 30 days returns a written early read of how much of a score change came from the model, the judge, the dataset, the task, and the prompt. Ask for one at xlnc.co/solutions. An email address is enough to start.

Different teams need different things, from a hosted service to a self-hosted harness that keeps your data on your side. Tell us which fits yours at xlnc.co/solutions.