Before using human judgments to evaluate an AI classifier, check whether the annotators apply the labeling guide consistently. A high agreement percentage can conceal a dataset dominated by one label. Cohen’s kappa adds a comparison against agreement expected from the two annotators’ observed label frequencies.

This example evaluates annotation consistency. It does not establish that either annotator is correct. Keep an adjudicated reference set separately when correctness matters.

Prerequisites

Use Python 3.12 or later. No packages, model server or credentials are required. Save the following program as kappa.py. Inputs are two lists containing labels for the same records in the same order.

Complete example

from collections import Counter
from fractions import Fraction

def agreement(left, right):
    if not left or len(left) != len(right):
        raise ValueError("paired nonempty labels required")
    if any(type(x) is not str or not x for x in left + right):
        raise ValueError("nonempty string labels required")
    n = len(left)
    a, b = Counter(left), Counter(right)
    observed = Fraction(sum(x == y for x, y in zip(left, right)), n)
    expected = sum((Fraction(a[k] * b[k], n*n) for k in a), Fraction())
    kappa = None if expected == 1 else (observed - expected) / (1 - expected)
    return observed, kappa

assert agreement(["a", "b"], ["a", "b"]) == (1, 1)
assert agreement(["a", "b"], ["b", "a"])[1] == -1
assert agreement(["a"], ["a"])[1] is None
for x, y in [([], []), (["a"], []), ([None], ["a"])]:
    try:
        agreement(x, y)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid labels accepted")
p, k = agreement(["yes", "yes", "no", "no"], ["yes", "no", "no", "no"])
assert (p, k) == (Fraction(3, 4), Fraction(1, 2))
print(f"agreement={float(p):.2f} kappa={float(k):.2f}")
print("fixture checks passed")

Run and expected output

python3 kappa.py
agreement=0.75 kappa=0.50
fixture checks passed

Understand the calculation

Three of the four paired judgments agree, producing observed agreement of 0.75. The first annotator uses each label twice. The second uses yes once and no three times. Multiplying corresponding label proportions and adding them gives expected agreement of 0.50. Kappa is therefore (0.75 − 0.50) / (1 − 0.50), or 0.50.

Fraction keeps these small calculations exact until display. The function returns both measures so a report can show the directly understandable agreement fraction alongside the adjusted statistic. Labels appearing only on the right contribute zero to the expected-agreement sum.

Edge cases and troubleshooting

When both lists contain only the same single label, expected agreement is one and the denominator becomes zero. The program returns None for kappa; do not silently convert this case into perfect reliability. Empty lists, unequal lengths and invalid labels raise an error.

Align records by a stable identifier before constructing the lists. Equal lengths cannot detect accidentally shuffled annotations. Keep label spelling and whitespace consistent through an explicit annotation vocabulary. This implementation uses nominal labels and gives every disagreement equal weight; it does not implement weighted kappa for ordered ratings.

If kappa changes substantially between annotation rounds, inspect label frequencies and disagreement examples before changing your release threshold. A tiny fixture demonstrates arithmetic, not the reliability of a production annotation process. These tests use synthetic labels and make no AI calls.

References and related tutorials

The scikit-learn kappa reference documents the statistic and its marginal-frequency calculation. Browse the AI tutorials and Python tutorials for related evaluation work.