Measure AI Abstention Separately from Answer Accuracy

A question-answering assistant can improve its apparent accuracy by refusing difficult questions. Conversely, an assistant that always answers may invent answers when the supplied material contains none. Record coverage and accuracy separately so that these behaviors remain visible.

This tutorial implements a small evaluation contract: each row contains an expected answer and an actual answer. An expected value of None means the question is unanswerable from the supplied evidence. An actual value of None means the assistant abstained. The example uses authored fixtures, not responses from a model.

Prerequisites

Use Python 3.12 or another supported Python 3 release. No packages, credentials or network connection are needed. Save the following complete program as eval.py.

def score(cases):
    if not cases:
        raise ValueError('empty evaluation')
    answered = correct = false_answers = abstained = 0
    for expected, actual in cases:
        if actual is None:
            abstained += 1
        else:
            answered += 1
            correct += expected is not None and actual == expected
            false_answers += expected is None
    return (answered, correct, false_answers, abstained,
            None if answered == 0 else correct / answered)

rows = [('Paris', 'Paris'), ('Tokyo', None), (None, 'Mars'), (None, None)]
assert score(rows) == (2, 1, 1, 2, 0.5)
assert score([(None, None)]) == (0, 0, 0, 1, None)
try:
    score([])
except ValueError:
    pass
else:
    raise AssertionError('empty accepted')
a, c, f, s, precision = score(rows)
print(f'answered={a} correct={c} false_answers={f} abstained={s}')
print(f'coverage={a/len(rows):.2f} answered_accuracy={precision:.2f}')

Run and expected output

python3 eval.py

answered=2 correct=1 false_answers=1 abstained=2
coverage=0.50 answered_accuracy=0.50

Interpret the result

The assistant attempted two of four questions, giving coverage of 0.50. One attempted answer was correct, so answered accuracy is also 0.50. The Mars answer is counted as a false answer because that question had no supported answer. The Tokyo abstention reduces coverage even though a reference answer existed.

The evaluator deliberately returns None for answered accuracy when every response abstains. Reporting zero would suggest measured incorrect answers; reporting one would reward answering nothing. An empty evaluation instead raises an exception because there is no population to summarize.

Edge cases and troubleshooting

Exact equality suits controlled labels or short canonical answers. It will reject equivalent prose, spelling variants and alternate date formats. Define normalization rules before evaluating, preserve the original answer, and add fixtures for every normalization rule. Do not silently convert an empty string to abstention unless your application contract explicitly defines that behavior.

If an assertion fails, inspect the corresponding tuple before changing the metric. Run without Python's optimization option so the included assertions execute. Before using this with model outputs, attach stable case identifiers, preserve the evidence used to establish answerability, and separate development cases from a held-out set. The four examples demonstrate metric arithmetic; they provide no evidence about a particular model's quality.

Primary references

Anthropic's evaluation guide explains evaluation tasks and graders. The metrics and fixtures above are an original, deliberately narrow design. Continue with the AI tutorials or Python tutorials.