AIM is not an eval tool. It is the ruler your LLM evals run on.
Your eval tools ask the questions and run the gates. AIM corrects LLM-as-a-judge bias and puts every score on one calibrated scale, with its error bar beside it.
The answer up top
A release ships, a vendor is renewed, a model is promoted. Each decision rests on an eval score, often from an LLM judge, that looked confident. The cost arrives later, when the score turns out to have been wider than anyone could see.
AIM does not replace the tools that produced that score. It sits underneath them and supplies the one thing they assume and none of them provides: a calibrated ruler, so every score carries a known error bar, lands on one scale, and has a record behind it.
If you stop reading here, hold two things. Your tools stay. The number underneath them needs a ruler.
What your eval stack does well
Your stack is good at its job. Tracing captures every call, prompt and tool step. Dataset tools curate and version your test sets. Pass-rates count what passed. AI judges score open-ended output at a volume no human team could match. CI gates block regressions, and dashboards catch the trend you would have missed.
None of that needs to change, and nothing below argues that it should.
The question none of them answers: how wide is the error bar?
Ask any of those tools two questions about a score. How wide is the error bar on this number? And what scale is it on, so I can compare it to last quarter's, or to a different model's?
The trace cannot tell you. Neither can the dataset, the pass-rate or the dashboard. Each one works as designed, and the error bar on the number can still be about six times wider than its label.
The exhibit: what LLM judge bias does to the error bar
We ran the same 15,000 certification decisions three ways, on the same responses, with one AI judge. In that simulation the measurement was set to an error bar of 0.20 logits.
- Judge harshness left in: delivered at 1.16. About six times the label, and nothing in the score shows it.
- Subtract one average, the standard fix: delivered at 0.79. The middle improves and the ends break. Weak work keeps 1.34 points of leniency; strong work gains 0.73 to 1.44 points of harshness that was never there.
- Correct at each level of work: delivered at 0.21 in that simulation.
Simulation. One judge, 13 to 35 responses per cell. Results are model output, not field measurements. The correction was estimated in-sample. Multi-judge generalization is a separate, gated study.
The full method and all three treatments are in the whitepaper.
What sits underneath
| What your eval and observability tools do (and keep doing) | What AIM supplies beneath them |
|---|---|
| Tracing: capture every call, prompt, tool step and output | A calibrated ruler: one interval scale with a fixed origin, so a step means the same at the bottom and the top |
| Datasets: curate, version and slice test sets | Frozen, fingerprinted question sets, so a move reads as a move in the model, not in the test |
| Pass-rates and metrics: count what passed | An error bar on every score, so "better" means better by more than the noise |
| LLM-as-judge: score open-ended output at scale | Judges used only at the levels of work where they were measured and admitted; steady harshness corrected, uncorrectable distortions excluded and recorded |
| CI gates: block a change that regresses | A gate that knows its own noise: pass when the improvement exceeds its error bar, hold when it does not |
| Dashboards and alerts: watch trends | One scale across models and versions; human experts join it after a calibration study that comes next |
| Root-cause views: which segment moved the metric | Which construct moved |
| (scores as they come) | Traceability: any decision traces to the question, bank version, scoring rule, judge and correction applied |
The root-cause row changes what a regression costs you. A blended score that drops tells you something got worse. It does not tell you what. AIM measures one construct per instrument, so when a regression lands, it names the capability that dropped. Your team then debugs one thing instead of guessing across the blend.
What AIM does not do
- It does not trace, log or store your production traffic.
- It does not manage your datasets, prompts or experiments.
- It does not author your eval suite or decide what you should test.
- It does not run your CI or host your dashboards and alerts.
- It does not replace human review, and it is not a fairness audit.
- It does not make you compliant with any law or standard. An error bar does not by itself meet a legal duty.
- It does not yet publish multi-judge panel results. The per-level correction result above is one judge, in simulation; the panel result is a separate, gated study.
Where it plugs in
Before release, certify a model, prompt or agent against the limits you set, with the error bar printed beside the result. In CI, gate on the measure rather than the raw score. In production, re-measure on a locked ruler at a set interval.
AIM reads the exports your tools already produce. Adapters for common eval-framework exports and generic CSV ship today; trace-file input is in build. Measures come back through a REST API, a Model Context Protocol server, or a harness that runs inside your boundary with your own model endpoints. The harness is in build.
LLM-as-a-judge calibration: three quick answers
Why do LLM-as-a-judge scores need calibration?
Because a judge's harshness shifts the scores it gives, and nothing in the score shows it. In our simulation of 15,000 certification decisions with one AI judge, leaving harshness in delivered an error bar of 1.16 logits against 0.20 set, about six times the label.
Is subtracting the judge's average bias enough?
No. In the same simulation, subtracting one average delivered 0.79 logits: the middle improves and the ends break. Correcting at each level of work delivered 0.21 in that simulation.
Does AIM replace my eval tools?
No. Tracing, datasets, pass rates, LLM judges, CI gates and dashboards stay. AIM sits underneath them as the measurement layer: one calibrated scale, an error bar on every score for eval reliability you can check, and a record behind each decision.
Read the method
Read the whitepaper: the full method, the three-way comparison, and every limit.
Or send us the judge you trust least, and we will show you where it bends.