What you will build

A ticket classifier can achieve 80% accuracy while missing every urgent ticket. Build a small offline report that makes this failure visible. The fixture contains eight routine tickets and two urgent tickets; its deliberately poor predictor labels everything routine. This tests evaluation arithmetic, with no model invocation or claim about model quality.

The report uses a fixed vocabulary supplied by the application. Keep that vocabulary constant when comparing runs: a model must not improve its headline number simply because a difficult class disappeared from a batch.

Prerequisites

Python 3.12 or a later supported Python 3 release. Only the standard library is used; do not install scikit-learn. Inputs are aligned lists of string labels. Save this complete example as f1.py.

from math import isclose

def report(truth, predicted, labels):
    if not truth or len(truth) != len(predicted):
        raise ValueError('nonempty aligned records required')
    if not labels or len(set(labels)) != len(labels):
        raise ValueError('unique label vocabulary required')
    if any(x not in labels for x in truth + predicted):
        raise ValueError('unknown label')
    scores = {}
    for label in labels:
        tp = sum(a == label and b == label for a, b in zip(truth, predicted))
        fp = sum(a != label and b == label for a, b in zip(truth, predicted))
        fn = sum(a == label and b != label for a, b in zip(truth, predicted))
        denominator = 2 * tp + fp + fn
        scores[label] = 2 * tp / denominator if denominator else 0.0
    accuracy = sum(a == b for a, b in zip(truth, predicted)) / len(truth)
    return accuracy, sum(scores.values()) / len(labels), scores

actual = ['routine'] * 8 + ['urgent'] * 2
accuracy, macro, scores = report(actual, ['routine'] * 10, ['routine', 'urgent'])
assert isclose(accuracy, 0.8) and isclose(macro, 4 / 9)
assert scores['urgent'] == 0
assert report(['a'], ['a'], ['a', 'b'])[1] == 0.5
for args in [([], [], ['a']), (['a'], [], ['a']),
             (['x'], ['a'], ['a']), (['a'], ['a'], ['a', 'a'])]:
    try:
        report(*args)
    except ValueError:
        pass
    else:
        raise AssertionError('invalid fixture accepted')
print(f'accuracy={accuracy:.3f} macro_f1={macro:.3f} urgent_f1={scores["urgent"]:.3f}')
print('fixtures passed')

Run and expected output

python3 f1.py
accuracy=0.800 macro_f1=0.444 urgent_f1=0.000
fixtures passed

How it works

For each label, count true positives, false positives and false negatives using a one-versus-rest view. Calculate F1 as twice the true-positive count divided by twice that count plus both error counts. This avoids separately calculating precision and recall when one of their denominators is zero. Macro F1 is the arithmetic mean of the class F1 values. These definitions follow the official metric reference.

Routine has eight true positives and two false positives. Its F1 is 16/18; urgent has no true positives and two false negatives, so its F1 is zero. Their mean is 4/9, printed as 0.444. Accuracy remains 0.800. Keep both numbers and the per-class values: neither scalar explains which workflow is failing.

The vocabulary check rejects unexpected predictions instead of silently removing them. A new label requires an intentional evaluation-contract change. If your application supports abstention, define a separate policy before using this single-label evaluator; do not quietly drop unanswered records.

Edge cases and scope

This example assigns zero when a class has no true or predicted instances. The extra two-label fixture demonstrates the consequence: one perfect observed class and one absent class yield macro F1 of 0.5. Another reporting design could exclude undefined classes, but that produces a different metric and must be named explicitly. Empty datasets and unequal list lengths are errors.

These are deterministic point estimates, not confidence intervals. Keep the same population, vocabulary and zero-denominator policy across comparisons. For costly urgent-ticket failures, inspect false negatives and decision thresholds as well as the aggregate.

Troubleshooting

If a score looks too high, print the three counts for every class and check that expected urgent cases remain in the input. If an unknown-label error appears, inspect the mapping from provider output to application labels. Do not repair spelling inside the evaluator, where normalization would hide integration defects. Run without Python's -O option so fixture assertions remain active.

Primary references

Continue with AI tutorials and all Java Samples categories.