Current events and new studies, read on a calibrated ruler.
Stanford's free AI Measurement Science textbook gets validity, Rasch and LLM judges right. The one chapter we hope a future edition adds is metrological traceability.
We paid to measure three judges twice. Here is what moved, and what it does not prove.
In a field experiment, students who practiced with an unrestricted AI tool scored 17 percent lower on the later exam. Here is why that happens, and how to measure learning instead of borrowed answers.
Binary pass/fail evals are right for decisions and wrong for measurement. Score LLM output on a calibrated continuous scale with an error bar, then set the pass/fail threshold.
Your eval tools ask the questions and run the gates. AIM corrects LLM-as-a-judge bias and puts every score on one calibrated scale, with its error bar beside it.
An AI agent kill switch is fast. The threshold behind it is usually operator-set. A proposed five-rung design trips on calibrated measures with error bars.
Before a score gates a release or a hire, ask for its papers: the reference it was compared with, the chain back to that reference, and the uncertainty at the end.
Anthropic's new eval and hillclimbing guidance gets the judge's independence right. Here is what a calibrated judge adds, with early data.
We measured five LLM judges at eight levels of reasoning complexity. Their fit and where they qualified changed from level to level, and price did not predict either. Here are the early numbers, and three figures to ask any judge vendor for.
The model that scores your work can change under the same name. Here is how to tell a worse product from a changed judge, and the checks we are testing.
AIM is building a way to read your OpenTelemetry traces and write calibrated judge scores, with standard errors, back onto your own spans. Trace ingest is in development and offered as early access. It complements your observability stack; it does not replace it.
Benchmarks leak into training data, and a leaked benchmark measures memory. AIM's pipeline is built to generate fresh, calibrated items at measurement time. Here is what exists today, and what does not yet.
An LLM judge's score is only as good as the judge. Here is what it means for a judge to be calibrated, why judge drift differs from model drift, and where AIM's judge credentialing stands today: credentials at levels 6 to 10, with re-credentialing under a stricter procedure under way.
Defect work has a documented history. An AI release row counts outcomes, while a release decision turns on a location with a stated error.
We measured one of the judges we use at eight levels of task complexity. Two cleared. Here is the rule that follows, free to copy.
When you want to know if an AI system is right, you ask a human, and the human's rating is called ground truth. Raters have measurable biases, and an LLM judge's bias can change sign across capability levels. Whether the human reference does the same is rarely checked.
NIST just published a 64-page guide on evaluating large language models, and nowhere in it does the word "traceability" appear. The national measurement laboratory wrote a measurement manual that never mentions measurement's own vocabulary. That silence tells you where the field stops, and where it has to go next.
Capability tells you what a model can reach. Severity tells you how far its verdicts lean. One average hides the lean, and the lean changes with the difficulty of the work.
An agent swarm spent four days attacking infrastructure it did not need to attack, chasing a grader requirement that did not exist. The near-miss was not grader manipulation. It was a grader blind to provenance, and that blindness is a closable attack surface.
The same model, the same benchmark, a different score every run. That instability is your instrument telling you what it cannot do.
