Before using human judgments to evaluate an AI classifier, check whether the annotators apply the labeling guide consistently. A high agreement percentage can conceal a dataset dominated by one label. Cohen’s kappa adds a comparison against agreement expected from the two annotators’ observed label frequencies.
This example evaluates annotation consistency. It does not establish that either annotator is correct. Keep an adjudicated reference set separately when correctness matters.
Prerequisites
Use Python 3.12 or later. No packages, model server or credentials are required. Save the following program as kappa.py. Inputs are two lists containing labels for the same records in the same order.
Complete example
from collections import Counter
from fractions import Fraction
def agreement(left, right):
if not left or len(left) != len(right):
raise ValueError("paired nonempty labels required")
if any(type(x) is not str or not x for x in left + right):
raise ValueError("nonempty string labels required")
n = len(left)
a, b = Counter(left), Counter(right)
observed = Fraction(sum(x == y for x, y in zip(left, right)), n)
expected = sum((Fraction(a[k] * b[k], n*n) for k in a), Fraction())
kappa = None if expected == 1 else (observed - expected) / (1 - expected)
return observed, kappa
assert agreement(["a", "b"], ["a", "b"]) == (1, 1)
assert agreement(["a", "b"], ["b", "a"])[1] == -1
assert agreement(["a"], ["a"])[1] is None
for x, y in [([], []), (["a"], []), ([None], ["a"])]:
try:
agreement(x, y)
except ValueError:
pass
else:
raise AssertionError("invalid labels accepted")
p, k = agreement(["yes", "yes", "no", "no"], ["yes", "no", "no", "no"])
assert (p, k) == (Fraction(3, 4), Fraction(1, 2))
print(f"agreement={float(p):.2f} kappa={float(k):.2f}")
print("fixture checks passed")
Run and expected output
python3 kappa.py
agreement=0.75 kappa=0.50
fixture checks passed
Understand the calculation
Three of the four paired judgments agree, producing observed agreement of 0.75. The first annotator uses each label twice. The second uses yes once and no three times. Multiplying corresponding label proportions and adding them gives expected agreement of 0.50. Kappa is therefore (0.75 − 0.50) / (1 − 0.50), or 0.50.
Fraction keeps these small calculations exact until display. The function returns both measures so a report can show the directly understandable agreement fraction alongside the adjusted statistic. Labels appearing only on the right contribute zero to the expected-agreement sum.
Edge cases and troubleshooting
When both lists contain only the same single label, expected agreement is one and the denominator becomes zero. The program returns None for kappa; do not silently convert this case into perfect reliability. Empty lists, unequal lengths and invalid labels raise an error.
Align records by a stable identifier before constructing the lists. Equal lengths cannot detect accidentally shuffled annotations. Keep label spelling and whitespace consistent through an explicit annotation vocabulary. This implementation uses nominal labels and gives every disagreement equal weight; it does not implement weighted kappa for ordered ratings.
If kappa changes substantially between annotation rounds, inspect label frequencies and disagreement examples before changing your release threshold. A tiny fixture demonstrates arithmetic, not the reliability of a production annotation process. These tests use synthetic labels and make no AI calls.
References and related tutorials
The scikit-learn kappa reference documents the statistic and its marginal-frequency calculation. Browse the AI tutorials and Python tutorials for related evaluation work.