Prerequisites and goal
A support classifier may assign several tags to one ticket. Predicting billing when the correct set is billing and refund is partially useful, but it is not a complete routing decision. This example reports both exact set match and mean per-ticket Jaccard overlap. Use Python 3.12 or later with no packages. The rows are manually specified fixtures, not predictions collected from a model.
Choose the evaluation unit before reading the score. Here every ticket gets equal weight, regardless of how many tags it contains. A ticket with ten labels does not automatically count ten times as much as a ticket with one. The vocabulary is fixed by the application, so a spelling mistake cannot silently become a new class.
Understand the calculation
For each row, exact match contributes one only when both sets are identical. Jaccard contributes the number of shared labels divided by the number appearing in either set. The fixture contributes 0.5, 1.0 and 0.5 to Jaccard, giving two thirds overall. Only the empty-set row matches exactly, giving one third exact accuracy. Formatting to three decimal places changes presentation only.
Both empty sets receive a Jaccard score of one in this application: correctly predicting no tags is treated as success. This convention must accompany any reported number. If empty tickets dominate a production dataset, also report their count and inspect scores for tickets with at least one true label. Otherwise a system that frequently emits nothing can look more useful than it is.
Edge cases and troubleshooting
The evaluator converts each collection to a set, so repeated tags do not improve the score. That is appropriate for tag membership, but would discard information in a task where repeated events matter. Unknown labels raise an error rather than being removed. Removing them would conceal false positive predictions and make a malformed response appear cleaner.
An empty evaluation raises ValueError because there is no denominator. A one-row evaluation is valid, but provides very little evidence about future behavior. If scores unexpectedly fall, print the original truth and prediction for each mismatching ticket, then check tag spelling, whitespace and upstream parsing. Do not normalize labels differently in the two columns.
The assertions check the hand-calculated aggregate, empty sets, duplicate tags, an empty dataset and an unknown label. This is a metric implementation test. It does not establish annotation quality, model accuracy, confidence intervals or performance on real support traffic. Keep a separate frozen test dataset when comparing model versions.
Complete example
Save as multilabel.py.
def score(rows, vocabulary):
if not rows:
raise ValueError("empty evaluation")
exact = 0
total = 0.0
for truth, predicted in rows:
truth, predicted = set(truth), set(predicted)
if (truth | predicted) - vocabulary:
raise ValueError("unknown label")
union = truth | predicted
total += len(truth & predicted) / len(union) if union else 1.0
exact += truth == predicted
return exact / len(rows), total / len(rows)
rows = [({"billing", "refund"}, {"billing"}),
(set(), set()), ({"delivery"}, {"delivery", "refund"})]
exact, jaccard = score(rows, {"billing", "refund", "delivery"})
assert abs(exact - 1/3) < 1e-12
assert abs(jaccard - 2/3) < 1e-12
assert score([([], [])], set()) == (1.0, 1.0)
assert score([(["a", "a"], ["a"])], {"a"}) == (1.0, 1.0)
for bad in [[], [({"unknown"}, set())]]:
try:
score(bad, {"billing"})
except ValueError:
pass
else:
raise AssertionError("invalid evaluation accepted")
print(f"exact={exact:.3f} mean_jaccard={jaccard:.3f}")
Run and expected output
python3 multilabel.pyexact=0.333 mean_jaccard=0.667Primary references
Continue with AI tutorials.