Audit RAG Citation Identifiers Before Rendering an Answer

A generated answer can contain convincing-looking citations that do not refer to anything retrieved for the current question. Before rendering those markers as links, check that each claim has a recognized marker and that every referenced identifier belongs to the current retrieval result. This catches a concrete wiring failure without asking another model to judge the answer.

The application contract in this example represents an answer as a list of claim strings. Sources receive identifiers such as S1 and S2 from application code. The audit recognizes only the exact marker format [S1], using Python's documented regular-expression operations. It never follows a URL supplied by the generated answer.

Prerequisites

Use Python 3.12 or a newer supported Python 3 release. No packages or model server are required. Save this program as citations.py. The source descriptions and generated claims below are explicit fixtures.

import re

MARKER = re.compile(r'\[S([1-9][0-9]*)\]')

def audit(claims, sources):
    errors = []
    for position, claim in enumerate(claims, 1):
        ids = MARKER.findall(claim)
        if not ids:
            errors.append(f'claim {position}: missing citation')
        for source_id in sorted(set(ids)):
            if source_id not in sources:
                errors.append(f'claim {position}: unknown S{source_id}')
    return errors

sources = {'1': 'fixture: refund policy', '2': 'fixture: delivery guide'}
assert audit(['Returns take 7 days. [S1]'], sources) == []
assert audit(['A [S1] [S1]'], sources) == []
assert audit([], sources) == []
errors = audit(['Unsupported promise.', 'Delivery is free. [S9]'], sources)
assert errors == ['claim 1: missing citation', 'claim 2: unknown S9']
assert audit(['Bad marker [S01]'], sources) == ['claim 1: missing citation']
print('\n'.join(errors))
print('Citation fixture checks passed; factual entailment not checked')

Run and expected output

python3 citations.py

claim 1: missing citation
claim 2: unknown S9
Citation fixture checks passed; factual entailment not checked

Connect the audit to a RAG application

Build the source registry from the passages actually supplied to generation, scoped to one request. After receiving and structurally validating the claim list, call the audit. If it returns errors, withhold the linked answer and route it to a repair or review step. Keep the original registry attached to that answer so a later request cannot accidentally resolve the same identifier to a different passage.

When rendering a valid marker, look up its destination in the application registry. Escape claim text before displaying it as HTML. A generated marker should select a known source; it should not create a new destination. This program only produces diagnostics and deliberately does not implement an HTML renderer.

What the checks establish

Repeated references to S1 are accepted, and converting the matches to a set avoids duplicate unknown-source messages within one claim. S01 is outside the chosen grammar. An empty claim list returns no errors because there are no claims to audit; applications requiring a nonempty answer must enforce that separately.

A passing audit does not show that a cited passage supports a claim. The positive fixture demonstrates identifier membership only. A model could append a valid citation to an unrelated assertion and pass this check. Factual entailment, source freshness, quotation accuracy, and retrieval relevance require separate evaluation.

Troubleshooting

If every reference is unknown, check whether the registry keys are strings without the S prefix, as required here. If markers disappear during processing, audit the original claim strings before formatting. Other marker styles are ignored by this intentionally narrow parser; adopt one explicit format throughout the application.

Explore related AI tutorials and Python tutorials.