Pyvorin for A/B Testing

Experiment Analysis

T-test, chi-squared, and bootstrap confidence intervals compiled for fast result computation.

def t_test(control, treatment):
    mean_c = sum(control) / len(control)
    mean_t = sum(treatment) / len(treatment)
    var_c = sum((x - mean_c) ** 2 for x in control) / (len(control) - 1)
    var_t = sum((x - mean_t) ** 2 for x in treatment) / (len(treatment) - 1)
    se = ((var_c / len(control)) + (var_t / len(treatment))) ** 0.5
    return (mean_t - mean_c) / se

Variance Reduction

CUPED and stratification for more sensitive experiments.

Early Stopping

Sequential testing and power monitoring to end experiments faster.