Frequentist Testing

Frequentist testing is the dominant statistical framework for online experimentation, rooted in the Neyman-Pearson hypothesis testing paradigm. The approach formulates a null hypothesis (no treatment effect), computes a test statistic from the observed data, determines how extreme this statistic would be if the null hypothesis were true (the p-value), and rejects the null if the p-value falls below a pre-specified significance level (alpha). For growth teams, frequentist testing provides a well-understood framework with clear decision rules, established sample size formulas, and broad tool support. Most experimentation platforms including Optimizely, VWO, and Google Optimize default to frequentist analysis, and the vast majority of published experiment results use frequentist methods.

The standard frequentist analysis for an A/B test proceeds as follows: compute the point estimate of the treatment effect (difference in means or proportions between treatment and control), compute the standard error of this estimate, form the test statistic z = effect_estimate / SE, compare z to the critical values from the standard normal distribution (z = 1.96 for a two-sided test at alpha = 0.05), and construct the 95% confidence interval as effect_estimate +/- 1.96 * SE. If the confidence interval excludes zero, the result is statistically significant. For proportion metrics, the test statistic uses the pooled proportion under the null for the variance estimate. For continuous metrics with potentially non-normal distributions, the central limit theorem ensures the test statistic is approximately normally distributed for large samples, which is typically satisfied in online experiments. The power of the test depends on the sample size, the true effect size, the significance level, and the metric variance.

Frequentist testing should be the default analysis method for most online experiments, especially when the organization values standardized decision rules and straightforward interpretation. The framework's main advantages are its simplicity, the pre-experiment control over error rates through power analysis, and the extensive tooling available. Common pitfalls specific to the frequentist approach include misinterpreting the p-value (it is not the probability that the null is true), conflating statistical significance with practical significance, the peeking problem (checking results repeatedly inflates the false positive rate), and the difficulty of incorporating prior information about expected effects. Bayesian testing offers advantages in interpretability (posterior probabilities are more intuitive than p-values), natural handling of sequential monitoring, and the ability to incorporate prior information, but requires specifying prior distributions and can be sensitive to prior choice.

Advanced frequentist methods for online experimentation include sequential testing procedures (group sequential, always-valid p-values) that allow valid interim monitoring, robust variance estimation methods that handle heavy-tailed metric distributions common in digital data (winsorized estimators, trimmed means), stratified analysis that improves precision by accounting for known sources of variation, and CUPED-style covariate adjustment. The debate between frequentist and Bayesian approaches has largely converged in practice: modern experimentation platforms offer both, and the choice often depends on organizational preference and the specific use case. The key insight is that both frameworks answer slightly different questions (frequentist: how unusual is this data if the null is true? Bayesian: what should we believe given this data?) and both can lead to valid decisions when properly implemented.