Evaluation
What is the Brier score and how is it calculated
The Brier score is the average squared difference between a forecast probability and the outcome, where the outcome is 1 if the event happened and 0 if it did not. Lower is better: 0 is a perfect forecast, and a constant 50% on two-outcome questions scores 0.25. For 1X2 the squares are summed over all three outcomes, so the range is 0 to 2.
Contents
Key terms
- Forecast
- A probability given in advance, for example 70% that the home team wins.
- Outcome
- 1 if the event happened, 0 if it did not.
- Brier score
- The average squared difference between forecast and outcome. Lower is better.
- Base rate
- How often the outcome has happened before. A forecast that always gives the base rate is the reference a model has to beat.
The formula
BS = (1 / N) × Σ (p − outcome)²Squaring punishes large errors much more than small ones. A miss at 90% confidence costs 0.81, while a miss at 60% costs 0.36.
An example with four forecasts
| Forecast | Outcome | (p − outcome)² |
|---|---|---|
| 0.70 | 1 | 0.0900 |
| 0.20 | 0 | 0.0400 |
| 0.55 | 0 | 0.3025 |
| 0.90 | 1 | 0.0100 |
| Total | 0.4425 |
0.4425 / 4 = 0.1106 → 0.111
A forecast that always says 50% on the same four questions scores (0.5)² = 0.25 on each one, so its Brier score is 0.250.
Comparison with a reference
The number on its own says little without a comparison. That is why the improvement over a reference on the same events is calculated (Brier skill):
BSS = 1 − BS(forecast) / BS(reference)1 − 0.1106 / 0.250 = 0.5576 → 55.8%
The forecasts in the example are 55.8% better than a constant 50%.
The same calculation appears on the AI model performance page as the Brier improvement over the base rate, for the model and for the margin-free market.
The 1X2 Brier score
When an event has more than two outcomes, the squares are summed over all of them. For 1X2 that is three terms, so the range is 0 to 2.
Forecast 1 / X / 2 = 50% / 30% / 20%.
| Match result | Calculation | Brier |
|---|---|---|
| 1 | (0.5 − 1)² + 0.3² + 0.2² | 0.380 |
| X | 0.5² + (0.3 − 1)² + 0.2² | 0.780 |
| 2 | 0.5² + 0.3² + (0.2 − 1)² | 0.980 |
An even forecast of 33.3% for each outcome scores 0.667 whatever the result.
The AI model performance page calculates it the same way: summed over outcomes, then averaged over matches. Under the same rule a two-outcome question scores twice as much as the single-probability formula: a constant 50% scores 0.500 there, not 0.250, and the four forecasts in the example score 0.221. The monthly reports in Research track the same Brier score over time.
Limitations
- The Brier score depends on how common the outcome is. On rare events even a weak forecast can score low, so it is compared with a reference on the same events.
- Figures from different sources cannot be compared if some sum over all outcomes and others use a single probability.
- Over a small number of matches the difference between two forecasts can be down to chance. The performance page therefore shows a 95% interval next to the difference.
- A good Brier score does not mean profit. Odds already contain the market's estimate, so a model has to beat the market, not just chance.
Related guides
Sources
- Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.