An AI evaluation can report a healthy overall result while failing an important group of requests. A release checklist should name the groups that must be represented and define what happens when their evidence is missing. This tutorial builds a small deterministic gate with three outcomes per slice: pass, fail and insufficient.

The fixture deliberately produces 80% overall success. Eight common examples pass, while two rare examples fail. A third required slice has no examples at all. The gate blocks release despite the headline result.

Prerequisites

Use Python 3.12 or later with its standard library. Save this code as slices.py. The input contract requires each row to have a unique string identifier, a configured slice name and a Boolean correctness result. Correctness labels are supplied by the evaluation process; this program does not judge generated answers.

Complete example

from fractions import Fraction

def gate(rows, required, minimum=2, floor=Fraction(3, 4)):
    if type(minimum) is not int or minimum < 1 or not 0 <= floor <= 1:
        raise ValueError("bad gate policy")
    if not required or len(set(required)) != len(required):
        raise ValueError("unique required slices needed")
    groups = {name: [] for name in required}
    seen = set()
    for row in rows:
        if row["id"] in seen or type(row["ok"]) is not bool:
            raise ValueError("duplicate id or nonboolean result")
        seen.add(row["id"])
        if row["slice"] not in groups:
            raise ValueError("unknown slice")
        groups[row["slice"]].append(row["ok"])
    report = {}
    for name, values in groups.items():
        n = len(values)
        score = Fraction(sum(values), n) if n else None
        report[name] = "insufficient" if n < minimum else ("pass" if score >= floor else "fail")
    return all(s == "pass" for s in report.values()), report

rows = [{"id": str(i), "slice": "common", "ok": True} for i in range(8)]
rows += [{"id": "x", "slice": "rare", "ok": False}, {"id": "y", "slice": "rare", "ok": False}]
passed, report = gate(rows, ["common", "rare", "new"])
assert sum(r["ok"] for r in rows) / len(rows) == 0.8
assert not passed and report == {"common": "pass", "rare": "fail", "new": "insufficient"}
assert gate(rows[:2], ["common"])[0]
boundary = [{"id": str(i), "slice": "common", "ok": i < 3} for i in range(4)]
assert gate(boundary, ["common"])[0]
try:
    gate(rows + [rows[0]], ["common", "rare"])
except ValueError:
    pass
else:
    raise AssertionError("duplicate accepted")
print("overall=0.80 release=False")
print(report)

Run and expected output

python3 slices.py
overall=0.80 release=False
{'common': 'pass', 'rare': 'fail', 'new': 'insufficient'}

How the gate works

Initialize every required slice before reading results. Otherwise an absent slice could disappear from the report and accidentally satisfy an all-passing condition. The minimum count is evaluated before the score, preventing a single successful example from being treated as adequate evidence.

The threshold comparison is inclusive. The extra boundary fixture verifies that three successes out of four meet a floor of three quarters. Duplicate identifiers are rejected so replaying the same successful record cannot inflate the evidence count. An unexpected slice name raises an error instead of silently creating an unreviewed group.

Edge cases and troubleshooting

The minimum of two is intentionally small for demonstration. Choose a production sample policy using the consequences of failure and the uncertainty you can tolerate. Passing a minimum-count rule does not establish statistical confidence or fairness. Keep these policy values in a reviewed evaluation configuration.

This implementation assigns one slice to each record. Overlapping groups need a different membership representation and careful counting. Missing fields raise a key error; validate incoming file schemas before invoking the gate. Preserve failed and insufficient outcomes separately so the next action is clear: fix behavior or gather evidence.

All results here are synthetic fixtures. No model was evaluated, no deployment was performed, and the example does not claim an 80% result for any real system.

References and related tutorials

Google’s ML Test Score rubric recommends testing important data slices. Continue with AI tutorials and data-science tutorials.