Sensor values can share a very large baseline while differing only slightly. Computing variance by subtracting a squared mean from a mean of squares can lose useful precision when those large terms are nearly equal. An online corrected-sum update keeps a running mean and a sum of squared deviations instead. It also consumes an iterator once rather than materializing the whole dataset.

Prerequisites and statistical contract

Use Python 3.12 with the standard math and statistics modules. This example computes sample variance, so its final divisor is n minus one and at least two observations are required. Population variance uses a different denominator. Decide which quantity your report needs before copying the implementation into a measurement pipeline.

For each new value, compute its difference from the previous mean, update the mean using the new count, then add the product of that old-mean difference and the new-mean difference to the corrected sum. The implementation accepts only built-in integers and floats, excludes booleans, and refuses NaN or infinity. It raises an overflow error if an update leaves the supported finite numeric range.

Complete example

Save as variance.py.

import math
import statistics

def sample_variance(values):
    n, mean, m2 = 0, 0.0, 0.0
    for value in values:
        if type(value) not in (int, float):
            raise ValueError('real numeric values required')
        x = float(value)
        if not math.isfinite(x):
            raise ValueError('finite values required')
        n += 1
        delta = x - mean
        mean += delta / n
        m2 += delta * (x - mean)
        if not math.isfinite(mean) or not math.isfinite(m2):
            raise OverflowError('numeric range exceeded')
    if n < 2:
        raise ValueError('sample variance needs two observations')
    return n, mean, m2 / (n - 1)

values = [1_000_000_000_000 + x for x in [1, 2, 3, 4]]
n, mean, variance = sample_variance(iter(values))
assert math.isclose(variance, statistics.variance(values), rel_tol=1e-12)
assert sample_variance([7, 7])[2] == 0
for bad in [[], [1], [1, float('nan')], [True, 2], [1, float('inf')]]:
    try:
        sample_variance(bad)
    except ValueError:
        pass
    else:
        raise AssertionError('invalid series accepted')
print(f'n={n} mean={mean:.1f} sample_variance={variance:.6f}')
print('reference, constant and five invalid-series checks passed')

Run and expected output

python3 variance.py
n=4 mean=1000000000002.5 sample_variance=1.666667
reference, constant and five invalid-series checks passed

Understand the reference comparison

The fixture values are a trillion plus 1, 2, 3 and 4. Their mean is 1000000000002.5 and their sample variance is approximately 1.666667. Passing an iterator demonstrates single-pass consumption. The result is compared with Python's statistics.variance using a stated relative tolerance rather than requiring identical floating-point bits. A constant pair checks a zero result.

The reference function documents sample-variance semantics. The historical corrected-sums paper is included for the method's background. Our executable checks establish agreement on this specific dataset; they do not establish an error bound across all possible magnitudes or observation orders. The algorithm retains three accumulators, while the surrounding demonstration keeps a short list only to run the independent reference comparison.

Edge cases and troubleshooting

Empty and one-element inputs raise instead of returning zero, which would confuse insufficient evidence with observed constancy. A NaN does not silently poison every later update. The explicit rejection policy also avoids skipping bad readings without informing the caller. If your workflow wants to omit missing values, filter them in a named upstream step and report how many were omitted.

Online updates still use finite-precision arithmetic. Extremely large integers may not convert to a finite float, and extreme opposite-signed values can overflow intermediate calculations. Use an appropriate numeric representation or scale when those ranges are legitimate. Weighted observations, merged partial accumulators and decimal arithmetic require separate implementations; this function does not provide them. The test covers a large-offset fixture, a constant pair and five invalid series. It makes no performance, universal numerical stability or statistical confidence claim.

Primary references

Continue with data-science tutorials.