What probabilities mean in AI decisions, and what calibration is

An answer like "urgent": 0.9 feels more informative than "urgent": true. It is, but only if the 0.9 means something. This post explains what a probability from a model is, what it means for one to be calibrated, and how to check that yourself before you build business rules on top of it.

Three different numbers that all get called "confidence"

When people talk about an LLM confidence score, they usually mean one of these:

A self-reported number. You ask the model "how confident are you, from 0 to 1?" and it writes 0.85. That's generated text. It can be useful as a rough signal, but it isn't derived from the model's internal probabilities.
Token probabilities (logprobs). Language models assign a probability to every possible next token. If the allowed answers are a closed set of labels, you can read the probability the model gives each label and renormalise over that set. This is a direct reading of the model's output distribution.
A calibrated probability. This is a probability, from either source, that has been checked, and usually adjusted, so that it matches observed frequencies on real data.

Only the third kind lets you say "when the model says 0.8, it's right about 80% of the time." The first two are raw signals.

What calibration means

A model is calibrated if, among all the cases where it says 0.8, the answer turns out to be correct about 80% of the time, and the same holds for 0.3, 0.95 and every other value.

Calibration is different from accuracy:

A model can be accurate but overconfident: it's usually right, but when it says 0.99 it's only right 90% of the time.
A model can be calibrated but not very useful: it says 0.5 for everything on a balanced yes/no task. That's perfectly calibrated and tells you nothing.

You want both: answers that separate the cases well, and probabilities you can take at face value.

How calibration is measured

Reliability diagram

Group predictions into bins (0–0.1, 0.1–0.2, …). For each bin, plot the average predicted probability against the fraction of predictions that were correct. A calibrated model sits on the diagonal. Points below the diagonal mean overconfidence, and points above it mean underconfidence.

Expected Calibration Error (ECE)

ECE is the weighted average gap between predicted probability and observed accuracy across the bins. Lower is better. It's easy to read, but it depends on how you choose the bins, so always report the binning scheme alongside the number.

Brier score

The Brier score is the mean squared difference between the predicted probability and the outcome (1 if it happened, 0 if it didn't). It rewards being both right and appropriately confident. Lower is better.

Log loss

Log loss heavily penalises being confidently wrong. It's useful when a confident mistake is much worse than an unsure one.

Why calibration drifts

Even a model that's well calibrated on one dataset can drift on yours:

Different domain. Legal emails don't look like retail chat.
Different language. Calibration measured on English may not carry over to French or German.
Different question wording. Changing your instructions or criteria changes the task.
Different label balance. If 2% of your tickets are urgent, a model tuned on a 50/50 split can over-predict urgency.

That's why calibration figures from a provider, any provider, are a starting point, not a guarantee for your use case.

How to check it yourself

You need a few hundred examples with known correct answers from your own data.

import numpy as np

# p: predicted probability of "yes" from the API (the noul value)
# y: the true outcome, 1 or 0, from your labelled data
p = np.array(predicted_probabilities)
y = np.array(true_labels)

brier = np.mean((p - y) ** 2)

bins = np.linspace(0, 1, 11)
idx = np.digitize(p, bins) - 1
ece = 0.0
for b in range(10):
    mask = idx == b
    if mask.any():
        ece += mask.mean() * abs(p[mask].mean() - y[mask].mean())

print(f"Brier: {brier:.3f}  ECE (10 bins): {ece:.3f}")

If the probabilities are off in a consistent way, you can fit a simple correction on a held-out set, such as temperature scaling, Platt scaling or isotonic regression, and apply it in your own code.

Using probabilities well

Set thresholds from data, not intuition. "Auto-approve above 0.9" should come from measuring what 0.9 means on your examples.
Use the whole distribution. In a choice answer, a close runner-up is a signal to send the case to review.
Re-check when anything changes: model version, criteria wording, the language of your inputs, or your traffic mix.
Log the probabilities alongside the decisions so you can audit them later.

Where Strike stands

Strike returns probabilities for every answer type: the yes value for noul, a distribution over options for choice, and a distribution over levels for score. Strike is in preview and its API currently runs in mock mode, so we make no calibration claims for strike-1. When we publish calibration results, we'll include the method and data so you can judge them. In the meantime, the checks above work with any model's probabilities, including ours.