An AI cascade tries an initial model and sends selected requests to a fallback. A common accounting mistake charges escalated requests only for the fallback, forgetting that the initial call already happened. Another mistake assumes the fallback always improves the answer. This offline simulator makes both assumptions visible before you connect any model service.

Prerequisites: Python 3.12. All examples use synthetic outcomes and arbitrary integer cost units. These units are neither current prices nor token estimates. No model call, paid API, package installation or credentials are required.

The FrugalGPT research paper describes model cascades as a way to select combinations of models. The implementation here is an original accounting exercise, not a reproduction of that paper's learned gate or performance results.

Choose a clear gate policy

A record contains a gate score, whether the initial answer was correct, and whether the fallback answer was correct. A score greater than or equal to the threshold accepts the initial answer. Otherwise, the simulator selects the fallback answer and adds its cost. Gate scores are fixture values; they have not been calibrated as probabilities.

import math

# score, cheap_is_correct, fallback_is_correct
FIXTURE = [(.95, True, True), (.8, False, True),
           (.4, True, False), (.2, False, True)]

def evaluate(rows, threshold, cheap_cost=1, fallback_cost=5):
    if not math.isfinite(threshold) or not 0 <= threshold <= 1:
        raise ValueError("invalid threshold")
    if type(cheap_cost) is not int or type(fallback_cost) is not int or min(cheap_cost, fallback_cost) < 0:
        raise ValueError("nonnegative integer cost units required")
    if not rows:
        raise ValueError("empty fixture")
    cost = correct = escalated = 0
    for score, cheap_ok, fallback_ok in rows:
        if not math.isfinite(score) or not 0 <= score <= 1:
            raise ValueError("invalid gate score")
        if type(cheap_ok) is not bool or type(fallback_ok) is not bool:
            raise ValueError("boolean outcomes required")
        cost += cheap_cost
        if score >= threshold:
            correct += cheap_ok
        else:
            escalated += 1
            cost += fallback_cost
            correct += fallback_ok
    return cost, correct, escalated

assert evaluate(FIXTURE, .7) == (14, 2, 2)
assert evaluate(FIXTURE, .9) == (19, 3, 3)
assert evaluate(FIXTURE, 0) == (4, 2, 0)
assert evaluate(FIXTURE, 1) == (24, 3, 4)
for threshold in (.7, .9):
    cost, correct, escalated = evaluate(FIXTURE, threshold)
    print(f"gate={threshold:.1f} cost={cost} correct={correct}/4 escalated={escalated}")
try:
    evaluate([], .5)
except ValueError:
    print("empty fixture rejected")
else:
    raise AssertionError("accepted empty fixture")

Run the simulator

Save the code as cascade.py and run:

python3 cascade.py

Expected output:

gate=0.7 cost=14 correct=2/4 escalated=2
gate=0.9 cost=19 correct=3/4 escalated=3
empty fixture rejected

At threshold 0.7, four initial calls cost four units and two fallback calls cost ten more. One escalation replaces a correct initial answer with an incorrect fallback answer. Raising the threshold to 0.9 repairs a different record, producing three correct answers at nineteen units. Sending every record to the fallback costs twenty-four units because the initial stage still runs.

Edge cases and troubleshooting

The function expects a nonempty list of fixture tuples. Threshold zero accepts every valid score. Threshold one still accepts a score exactly equal to one, so use fixtures deliberately when testing all-fallback behavior. An empty evaluation raises an error rather than reporting zero cost as if a useful experiment occurred.

For a real study, record request-specific costs, gate computation costs and labeled outcomes for each candidate stage. Select thresholds using development data and reserve separate test data for reporting. Missing fallback observations cannot be invented: this toy dataset includes both outcomes by construction. Latency, failures, retries and gate overhead are outside this simulator and were not benchmarked.

Explore more AI tutorials and implementation examples in Python.