Evaluation
Probability calibration: when model probabilities can be trusted
A forecast is calibrated when, of all the events given a probability of 70%, about 70% actually happen. It is checked by grouping forecasts into bins by probability and comparing the average forecast with the observed frequency in each bin. The weighted average gap is the expected calibration error (ECE): the smaller it is, the more reliable the probabilities.
Contents
Key terms
- Calibration
- Agreement between forecast probabilities and observed frequencies. A model is calibrated if about 60% of its “60%” calls happen.
- Calibration curve
- A plot of forecast probability (horizontal) against observed frequency (vertical), bin by bin. For a calibrated model the points lie on the diagonal. It is also called a reliability diagram.
- Bin
- A group of forecasts with a similar probability, for example everything between 60% and 70%.
- ECE
- Expected calibration error: the gap between forecast and frequency in each bin, weighted by the number of forecasts in the bin.
- Overconfident forecast
- The model gives a higher probability than the observed frequency of the outcome.
How is calibration checked?
- Forecasts are grouped into bins by probability, usually 10 percentage points wide.
- In each bin, compute the average forecast and how often the outcome actually happened.
- Show the gaps on a calibration curve or add them up into one number, the ECE.
ECE = Σ (forecasts in bin / all forecasts) × |average forecast − frequency|A thousand forecasts in four bins:
| Bin | Average forecast | Forecasts | Happened | Frequency | Gap |
|---|---|---|---|---|---|
| 20–30% | 25% | 200 | 54 | 27.0% | 2.0 |
| 40–50% | 45% | 300 | 141 | 47.0% | 2.0 |
| 60–70% | 65% | 300 | 177 | 59.0% | 6.0 |
| 80–90% | 85% | 200 | 132 | 66.0% | 19.0 |
ECE = 0.2 × 2.0 + 0.3 × 2.0 + 0.3 × 6.0 + 0.2 × 19.0 = 6.6 percentage points
Up to 50% the model is well calibrated, above that it is overconfident: when it says 85%, the outcome happens 66% of the time. The model is most overconfident exactly on the picks that look safest.
Why calibration matters in betting
A model's probability is compared with the implied probability of the odds, so a calibration error becomes a betting error.
A model that says 85% treats 1.18 (1 / 0.85) as fair odds, so any price above it looks like value. If such picks really win 66% of the time, fair odds are 1.52 (1 / 0.66). Odds of 1.40 then have an expected value of 0.66 × 1.40 − 1 = −7.6%, even though the model sees +19.0%.
An uncalibrated model keeps finding expected value that is not there.
Calibration is not accuracy
A forecast that gives every match the same probability, equal to the historical frequency (the base rate), is perfectly calibrated but cannot tell matches apart. A good forecast has to be calibrated and sharp: it gives different matches different probabilities, and those differences are right. The Brier score and log loss measure both properties together, a calibration curve only the first.
How are probabilities calibrated?
A curve “forecast → observed frequency” is learned on past forecasts, and every new forecast is passed through it. In the example above a new 85% becomes about 66%.
- The curve has to be increasing (monotonic): a higher forecast must never end up lower than a smaller one.
- Isotonic regression gives a step-shaped increasing curve, Platt scaling a smooth logistic one.
- The curve is learned on one set of data and checked on data it has not seen. Otherwise it only memorises the sample.
StatsAI calibrates its probabilities separately for each kind of pick (1X2, goals, BTTS and others): the curve is learned from settled predictions of the last 90 days, with at least 150 predictions per bin, it has to be increasing, and it is refreshed every 6 hours. On the AI model performance page each day's curve is learned only from earlier days, so the check is out of sample.
As of 31 October 2026, matches from 1 September to 31 October 2026:
| Market | Raw model | Calibrated model | Margin-free odds |
|---|---|---|---|
| 1X2 | 1.3 | 0.6 | 0.7 |
| Over/under 2.5 | 9.6 | 4.6 | 1.7 |
| BTTS | 9.6 | 3.4 | 1.8 |
The raw model was overconfident on goals: when it gave about 83% for over 2.5 goals (472 events), it happened 67.8% of the time. Calibration more than halved the error, but margin-free odds are still better calibrated. The figures change daily on the AI model performance page.
Limitations
- Calibration needs a large sample. A bin with few forecasts has a wide interval, so its gap may be chance.
- A model calibrated on one league or season is not necessarily calibrated on another, which is why the curve is relearned regularly.
- ECE depends on the number and width of the bins, so figures from different sources are only comparable when computed the same way.
- Good calibration does not mean profit: margin-free odds are well calibrated too, and the bookmaker still keeps its margin.
Related guides
Sources
- Niculescu-Mizil, A., Caruana, R. (2005). Predicting good probabilities with supervised learning. Proceedings of the 22nd International Conference on Machine Learning, 625–632.
- Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning.