NIST AI 800-3
“Raw benchmark scores do not have this property.”
NIST AI 800-3, February 2026. Quoted verbatim.
What NIST AI 800-3 highlights as promising.
AIM implements at measurement grade the latent-trait/GLMM approach that NIST AI 800-3 highlights as a promising foundation for AI evaluation statistics. AIM is not a NIST standard. It claims no endorsement.
What the sentence asks for.
NIST AI 800-3 asks that the distance between two models' capabilities on the scale keep its meaning whichever tasks were picked for the benchmark. On an interval scale, a two-point gap means the same amount everywhere. That is what the sentence asks for. It is what our gate checks.
A worked example, not measured data. The raw score gap moves with the task mix. The gap on the scale stays put.
Where each part sits.
- A shared scale for questions and models: the Rasch model puts both on one scale.
- Distances that mean the same thing everywhere: the scale's unit is the logit.
- A stated error for each result: every measure carries its own error bar.
- Results that do not hang on one task mix: the questions are locked, and the mix is recorded.
What alignment is not.
AIM is not a NIST standard. NIST has not reviewed, tested or endorsed it. The sentence above describes a property. It does not describe our product.
The dated record.
Simulation
Set to an error bar of 0.20. Delivered at 0.21 once judge harshness is corrected at each level.
Simulation: 15,000 decisions on judge data from our own runs, seed 20260908. With harshness left in, the same run came in at 1.16. Error bars in logits.
The method is published as Chapter 3 of a book on models and metrology, edited by Fisher and Pendrill. De Gruyter published it in 2024.
