When comparing a candidate assistant with a baseline, a single win percentage can conceal how the evaluation actually behaved. A judge may prefer A, prefer B, consider the answers equal, or fail to produce a usable judgment. Preserve all four outcomes before calculating an aggregate. This example implements an application policy in which a tie contributes half a win and an invalid judgment contributes no preference evidence.

Prerequisites and contract

Use Python 3.12 or a compatible maintained Python 3 release; no third-party modules are needed. Each input row contains a unique case identifier and one of A, B, tie or invalid. A and B must retain their model meanings throughout the dataset. If a judging interface randomizes answer positions, translate its left/right result back to these model labels before calling this function.

The score is (A wins + half the ties) divided by all usable judgments. Invalid records remain in the reported counts. An empty dataset, or a dataset containing only invalid judgments, returns None rather than an invented zero. Unique identifiers prevent accidental double counting after a retry or a file merge. Duplicate cases raise an error; deciding which retry to retain belongs in an explicit upstream policy.

Complete example

Save as pairwise.py.

from collections import Counter

def summarize(rows):
    counts = Counter({'A': 0, 'B': 0, 'tie': 0, 'invalid': 0})
    seen = set()
    for case, verdict in rows:
        if not isinstance(case, str) or not case or case in seen:
            raise ValueError('case identifiers must be unique nonempty strings')
        seen.add(case)
        if verdict not in counts:
            raise ValueError('unknown verdict')
        counts[verdict] += 1
    valid = counts['A'] + counts['B'] + counts['tie']
    score = (counts['A'] + counts['tie'] / 2) / valid if valid else None
    return counts, score

rows = [('q1', 'A'), ('q2', 'tie'), ('q3', 'B'), ('q4', 'invalid')]
counts, score = summarize(rows)
assert score == 0.5 and counts['invalid'] == 1
assert summarize([])[1] is None
assert summarize([('q1', 'invalid')])[1] is None
assert summarize([('q1', 'tie')])[1] == 0.5
for bad in [[('q1', 'A'), ('q1', 'B')], [('q1', 'unknown')]]:
    try:
        summarize(bad)
    except ValueError:
        pass
    else:
        raise AssertionError('invalid input accepted')
print('A=1 B=1 tie=1 invalid=1 score=0.500')
print('edge checks passed')

Run and expected output

python3 pairwise.py
A=1 B=1 tie=1 invalid=1 score=0.500
edge checks passed

Interpret the result

The fixture has one win for each model, one tie and one invalid result. Its adjusted A score is 0.500 over three usable cases. That is different from saying that half of all attempted judgments preferred A. Keep the raw counts next to the score so readers can reconstruct the denominator and see judge failures.

The assertions cover an empty collection, all-invalid input, an all-tie collection, duplicate identifiers and an unknown verdict. These are deterministic aggregation tests. No answers were generated, no human labels were collected and no model judge was invoked. The illustrative score says nothing about any actual model's quality.

Edge cases and troubleshooting

An unknown verdict should trigger investigation of the normalization adapter instead of becoming a tie. Keep dataset revision, rubric, judge identity and answer ordering with your evaluation artifacts. A large invalid count can make a seemingly stable score unrepresentative of the attempted workload. This small example supplies no confidence interval and makes no significance claim. For real comparisons, inspect per-task outcomes and repeatability before deciding that a small numerical difference warrants a release. The cited evaluation paper discusses stability of global and pairwise scores; the exact half-tie accounting here is our declared reporting policy.

Primary references

Continue with AI tutorials.