Score Binary AI Probabilities with Brier Loss

A classifier that assigns 0.51 to an event and one that assigns 0.99 can make the same binary decision while expressing very different confidence. If your AI application consumes probabilities, evaluate those probabilities directly. This tutorial implements binary Brier loss using only Python's standard library and explicit input policies.

Prerequisites: Python 3.12 and a small evaluation set with independently established binary labels. No model, SDK, package installation or network connection is needed. The numbers below are synthetic fixtures, not measured model predictions.

Define the evaluation contract

Each record contains an integer outcome, either 0 or 1, followed by the predicted probability of outcome 1. The evaluator averages squared prediction errors. Perfect predictions score zero; confidently reversed predictions score one. A constant probability of 0.5 scores 0.25 on every record. Lower loss is better under this convention. See the official Brier loss definition.

Keep the label direction fixed. If 1 means escalation required in your dataset, every probability must refer to escalation required. Mixing probabilities of opposite classes can produce a plausible but meaningless report.

Complete executable example

Save this as brier.py. The evaluator rejects empty datasets, Boolean labels, nonbinary labels, nonfinite probabilities and probabilities outside the unit interval.

import math

def brier(rows):
    rows = list(rows)
    if not rows:
        raise ValueError("empty evaluation")
    errors = []
    for y, p in rows:
        if type(y) is not int or y not in (0, 1):
            raise ValueError("binary integer label required")
        if type(p) not in (int, float) or not math.isfinite(p) or not 0 <= p <= 1:
            raise ValueError("invalid probability")
        errors.append((p - y) ** 2)
    return math.fsum(errors) / len(errors)

score = brier([(1, .8), (0, .2), (1, .6), (0, .4)])
assert math.isclose(score, .1)
assert brier([(1, 1), (0, 0)]) == 0
assert brier([(1, 0), (0, 1)]) == 1
assert brier([(1, .5), (0, .5)]) == .25
for bad in ([], [(2, .5)], [(True, .5)], [(1, float("nan"))], [(0, 1.1)]):
    try:
        brier(bad)
    except ValueError:
        pass
    else:
        raise AssertionError(bad)
print(f"brier={score:.3f}; baseline=0.250; checks passed")

Run and interpret

python3 brier.py

Expected output:

brier=0.100; baseline=0.250; checks passed

The four errors are 0.04, 0.04, 0.16 and 0.16, giving a mean of 0.10. This arithmetic provides a hand-checkable oracle independent of the implementation. Exact endpoint checks also expose an accidentally inverted label convention.

Edge cases and troubleshooting

Do not substitute a model's verbal confidence rating for a probability without defining and evaluating that conversion. Keep unavailable probabilities separate from zero: zero is an extremely strong prediction. If invalid records exist, either fail the evaluation or report exclusions and their counts explicitly rather than silently dropping them.

A lower Brier loss does not establish better calibration by itself; discrimination and outcome uncertainty also contribute. The official calibration guide explains this limitation. For comparisons, preserve the same held-out records and label meanings. The executable tests establish arithmetic and rejection behavior only; they make no claim about model quality or statistical significance.

Continue with AI tutorials and data-science tutorials.