Measure RAG Retrieval Recall at k with Explicit Edge-Case Policies

A retrieval-augmented generation system can produce a poor answer because its retriever never supplied the necessary evidence. Measure that stage separately with recall at k: the number of relevant documents among the first k results divided by the number of relevant documents in the reference set.

This tutorial builds an offline evaluator, rather than a retriever or model client. Prerequisites are Python 3.12 or later and a small collection of query-level relevance labels. No packages, credentials or model calls are required. Start with document identifiers that mean the same thing in both your retrieval output and your labels.

Complete example

Save this as recall.py. The fixture contains two relevant documents, but only one appears in the first two results.

def recall_at_k(ranked, relevant, k):
    if type(k) is not int or k <= 0:
        raise ValueError('k must be a positive integer')
    if len(ranked) != len(set(ranked)):
        raise ValueError('duplicate document ID')
    if not relevant:
        return None
    return len(set(ranked[:k]) & set(relevant)) / len(set(relevant))

assert recall_at_k(['b', 'a', 'c'], {'a', 'c'}, 2) == 0.5
assert recall_at_k([], {'a'}, 2) == 0
assert recall_at_k(['a'], set(), 2) is None
for ranked, k in [(['a', 'a'], 2), (['a'], 0), (['a'], True)]:
    try:
        recall_at_k(ranked, {'a'}, k)
        raise AssertionError('accepted invalid input')
    except ValueError:
        pass
print('recall@2=0.50; edge cases passed')

Run and interpret

python3 recall.py

Expected output:

recall@2=0.50; edge cases passed

The denominator remains two even when fewer than k results are returned. An empty ranking therefore scores zero when relevant evidence exists. A query with no labeled relevant documents returns None, because its recall denominator would be zero. When aggregating a batch, report the number of such queries separately and average only the defined values under this policy.

Duplicate identifiers are rejected across the complete ranking. This makes a repeated result visible as a retrieval defect instead of allowing set conversion to conceal it. If your retriever returns chunks while labels describe whole documents, first define and test an explicit chunk-to-document mapping; otherwise you are comparing different evaluation units.

Edge cases and troubleshooting

Use hashable identifiers such as strings. A boolean is deliberately rejected as k even though Python treats booleans as integers in some contexts. A larger k can improve this metric while increasing context size and retrieval cost. It does not establish that the generator uses evidence correctly.

If recall suddenly falls, inspect identifier normalization, corpus revisions and stale labels before changing retrieval parameters. Incomplete relevance judgments also limit the interpretation of the score. The executed fixtures verify arithmetic and input policies; they do not measure any live retrieval system or answer quality.

Primary reference

The Stanford information retrieval textbook defines precision and recall. The code applies that recall definition to a ranked prefix with explicit application policies. Continue with more AI tutorials and Python examples.