Pre-Post Analysis

Pre-post analysis is the simplest quasi-experimental design, comparing the same population's metrics before and after an intervention. If conversion rate was 5% in the week before a website redesign and 6% in the week after, the naive pre-post estimate attributes the 1 percentage point increase to the redesign. For growth teams, pre-post analysis is often the only feasible evaluation method for changes that cannot be randomly assigned, such as a complete brand refresh, a platform migration, or a pricing structure change that must apply to all users simultaneously. While pre-post analysis is easy to implement and intuitively appealing, it is the weakest causal inference method because it cannot distinguish the treatment effect from time trends, seasonal patterns, or concurrent changes.

The basic pre-post analysis computes the average metric in the post period minus the average in the pre period, with statistical significance assessed using a paired t-test or a time series comparison. More sophisticated approaches include interrupted time series analysis, which fits a regression model to the pre-treatment time series and extrapolates the expected trend into the post period, then measures the deviation from this trend as the treatment effect. The model typically includes level change (immediate jump at treatment) and slope change (change in trend after treatment) parameters: Y_t = beta_0 + beta_1*t + beta_2*D_t + beta_3*(t - T)*D_t + epsilon_t, where D_t is an indicator for the post-treatment period and T is the treatment time. This approach accounts for pre-existing trends, making the estimate more credible. Seasonality can be addressed by including calendar indicators or using the same period from the previous year as the baseline.

Pre-post analysis should be used when randomization is truly infeasible and when the pre-treatment period provides a stable baseline without major confounding events. The primary pitfall is confounding: any factor that changes coincidentally with the treatment will be attributed to it. If you launch a redesign during the holiday shopping season, the traffic increase from holiday demand will inflate the estimated redesign effect. Mitigation strategies include using a long pre-treatment period to establish stable trends, controlling for known confounders in the regression model, and using a comparison group (even a non-equivalent one) to create a difference-in-differences design that removes shared time trends. Teams should always be explicit about the assumptions underlying pre-post analysis and present results with appropriate caveats about potential confounding.

Advanced pre-post methods include Bayesian structural time series models (implemented in Google's CausalImpact R package), which use a state space model to forecast the counterfactual post-treatment trajectory and compute the posterior distribution of the treatment effect. This approach can incorporate control time series (metrics from comparable but untreated units) as covariates to improve the counterfactual forecast. Synthetic control methods extend this further by constructing a weighted combination of untreated units that best matches the treated unit's pre-treatment trajectory. For digital experimentation, combining pre-post analysis with regression discontinuity (when treatment assignment has a sharp cutoff) or with instrumental variables (when an exogenous shock drives treatment adoption) can strengthen causal identification substantially.