An embedding ingestion job needs to associate every returned vector with the correct document. Concurrent workers and retries can finish in a different order from the original input. Zipping completion-order results with input-order documents can silently attach a valid vector to the wrong text. Reconcile a complete batch by stable application identifiers before committing it to a search index.

Prerequisites and integration boundary

Use Python 3.12 and its standard library. The example receives normalized application records with id and vector fields. This is our adapter contract, not a claim about any provider's HTTP response. An upstream worker must attach the submitted document ID to its result using the provider's documented correlation mechanism or the worker's own request context. Do not manufacture an association by guessing from completion order.

The expected ID list defines output order. Each expected ID must occur exactly once; unexpected IDs, repeated IDs and missing IDs reject the entire batch. Every vector must be a list with the configured dimension count and finite numeric components. Python booleans are explicitly excluded even though they can behave like integers. Returning a new list for each vector avoids sharing its mutable container with the input records.

Complete example

Save as embeddings.py.

import math

def reconcile(expected, records, dimensions):
    if type(dimensions) is not int or dimensions < 1:
        raise ValueError('invalid dimensions')
    if any(not isinstance(key, str) or not key for key in expected):
        raise ValueError('invalid expected identifier')
    wanted = set(expected)
    if len(wanted) != len(expected):
        raise ValueError('duplicate expected identifier')
    found = {}
    for record in records:
        key, vector = record['id'], record['vector']
        if key not in wanted or key in found:
            raise ValueError('unexpected or duplicate result')
        if not isinstance(vector, list) or len(vector) != dimensions:
            raise ValueError('wrong dimensions')
        if any(type(x) not in (int, float) or not math.isfinite(x) for x in vector):
            raise ValueError('nonfinite or nonnumeric component')
        found[key] = vector[:]
    if set(found) != wanted:
        raise ValueError('missing result')
    return [found[key] for key in expected]

# Normalized application records; not a vendor response schema.
records = [{'id': 'b', 'vector': [0.0, 1.0]}, {'id': 'a', 'vector': [1.0, 0.0]}]
assert reconcile(['a', 'b'], records, 2) == [[1.0, 0.0], [0.0, 1.0]]
bad_batches = [records[:1], records + records[:1],
    [{'id': 'x', 'vector': [1, 0]}],
    [{'id': 'a', 'vector': [1]}],
    [{'id': 'a', 'vector': [float('nan'), 0]}],
    [{'id': 'a', 'vector': [True, 0]}]]
for batch in bad_batches:
    try:
        reconcile(['a', 'b'], batch, 2)
    except ValueError:
        pass
    else:
        raise AssertionError('bad batch accepted')
assert reconcile([], [], 2) == []
print('ordered=a,b rejected=6 empty=ok')

Run and expected output

python3 embeddings.py
ordered=a,b rejected=6 empty=ok

Connect the boundary to an ingestion job

The fixture deliberately returns b before a. Reconciliation returns the vectors in a,b order. Run this step after collecting the batch and before writing document-vector pairs. On failure, retain the batch identity and an application error reason for inspection; do not fill missing records with zero vectors or discard duplicates silently. Those shortcuts can create plausible search results with corrupted associations.

Hugging Face describes feature extraction as converting text into embeddings. The fixture supplies hand-written two-dimensional vectors solely to test association and validation. It does not test semantic similarity, model behavior, network transport or a provider SDK. The six rejected batches cover missing output, duplicate output, an unknown ID, a short vector, NaN and a boolean component. Empty input with empty results is accepted.

Edge cases and troubleshooting

This function assumes each normalized record already contains the named fields and a string ID; malformed envelope parsing belongs in the adapter. Keep batches reasonably bounded because the expected set, result dictionary and returned vectors occupy memory. Dimension agreement is necessary but insufficient: vectors from two different models can have the same length. Persist model revision and preprocessing identity alongside your index, and never combine incompatible embedding spaces merely because this shape check passes. If duplicates appear after retries, fix retry deduplication upstream instead of choosing the last completion arbitrarily. The caller should decide whether to retry a partial batch or quarantine it, without exposing a partially reconciled result.

Primary references

Continue with Python tutorials and AI tutorials.