ABiasales — a module for A/B test analysis.
It includes:
Handling outliers: removing, cappingVariance reduction: stratification, CUPED/CUPACRatio metric estimation: linearization, delta method, and various unit weighting methodsStatistical tests: classical tests and bootstrap
pip install git+https://github.com/aviasales/ABiasales.gitfrom abiasales.results import calc_exp_results
results = calc_exp_results(
df=your_dataframe,
exp_group_col='exp_group',
metrics={
'CTR': {
'num': 'clicks', # numerator of metric
'den': 'views', # denominator of metric
'weight_method': 'size', # 'uniform', 'size', 'sqrt', 'intra_corr'
'apply_linearization': False, # False, True
'use_delta_method': True, # False, True
'stat_method': 't_test', # 'proportion', 'z_test', 't_test', 'bootstrap'
'var_reduction_method': 'cuped', # 'cuped', 'cupac'
'var_reduction_covariates': ['clicks_cov', 'purchases_cov'],
'uplift_type': 'rel', # 'abs', 'rel'
}
},
use_stratification=True, # False, True
strats_cols=['platform', 'country']
)cleaning.py: outlier handlingweights.py: experimental unit weightingstats.py: mean, variance, and confidence interval calculationsstratification.py: stratification utilitiesvariance_reduction.py: CUPED / CUPAC variance reductionmetrics.py: mean and standard deviation estimationinference.py: statistical testspower.py: statistical power tablesresults.py: A/B test result summariesdata_generation.py: synthetic data generationsimulation.py: A/A and A/B test simulations
Oleg Yaksin