When a retrieval application shows a supporting passage, it should be possible to locate that passage in the original document. Saving only copied text makes this harder when the same sentence occurs several times. Store a start offset, an exclusive end offset and the exact substring for every chunk.

This example implements the document-preparation stage of a retrieval-augmented generation pipeline. It does not generate embeddings, rank passages or call a model. Fixed character windows make the offset and overlap rules easy to inspect before introducing sentence boundaries or a tokenizer.

Prerequisites

Use Python 3.12 or another supported Python 3 release. Save this UTF-8 program as chunks.py. It requires only the standard interpreter.

def chunks(text, size, overlap):
    if type(size) is not int or type(overlap) is not int:
        raise ValueError('integer sizes required')
    if not 0 <= overlap < size:
        raise ValueError('require 0 <= overlap < size')
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        yield {'start': start, 'end': end, 'text': text[start:end]}
        if end == len(text):
            break
        start = end - overlap

text = 'A🙂BCDEFGHI'
parts = list(chunks(text, 4, 1))
assert [(p['start'], p['end']) for p in parts] == [(0,4),(3,7),(6,10)]
assert all(text[p['start']:p['end']] == p['text'] for p in parts)
assert parts[0]['text'] + ''.join(p['text'][1:] for p in parts[1:]) == text
assert list(chunks('', 4, 1)) == []
assert len(list(chunks('ABCD', 4, 1))) == 1
for size, overlap in [(0,0),(4,4),(4,-1),(True,0)]:
    try:
        list(chunks(text, size, overlap))
    except ValueError:
        pass
    else:
        raise AssertionError('invalid sizes accepted')
for p in parts:
    print(p['start'], p['end'], p['text'])

Run and expected output

python3 chunks.py

0 4 A🙂BC
3 7 CDEF
6 10 FGHI

Understand the boundaries

The first window covers positions zero through three. Its end offset is four because Python slices exclude the end position. Subtracting the overlap from that end starts the next window at three, repeating C. The reconstruction assertion removes one overlapping character from each subsequent chunk and recovers the original text.

Requiring overlap to be smaller than size guarantees forward progress. Breaking immediately when the end reaches the document length avoids an extra chunk containing only the final overlap. Empty documents produce no chunks. Because this is a generator, argument validation occurs when iteration starts; the invalid-input tests explicitly consume the generator.

Edge cases and troubleshooting

These are Python string indices, not UTF-8 byte offsets, browser UTF-16 indices or model-token counts. The emoji fixture makes that distinction observable. Combining marks can also be split across windows. If a frontend highlights passages, agree on an offset representation rather than sending these integers directly to an unrelated indexing API.

Store a document identifier and immutable revision alongside each chunk in a real index. Changing line endings, normalizing Unicode or trimming whitespace after recording offsets will invalidate the mapping. Perform preprocessing first and retain the exact text version against which offsets were calculated. Character windows can split words and do not enforce a model's context limit; tokenizer-aware segmentation and retrieval quality remain separate work.

Primary references

The original RAG paper supplies retrieval-generation context; it does not prescribe this chunker. Python's sequence documentation defines slicing. Explore more AI tutorials.