Difference-in-Differences

Difference-in-differences (DiD) is one of the most widely used causal inference methods when randomized experiments are not feasible. The core idea is elegant: compare two groups across two time periods. If the treatment group was already different from the control group before the treatment (selection bias) and if both groups would have followed the same trend absent the treatment (the parallel trends assumption), then the difference in their changes over time removes both the selection bias and the common time trend, isolating the treatment effect. For growth and advertising teams, DiD is invaluable for evaluating geo-targeted campaigns, market-level rollouts, policy changes, and any intervention that is applied to identifiable groups rather than randomly assigned individuals.

The DiD estimator is calculated as: tau = (Y_treatment_post - Y_treatment_pre) - (Y_control_post - Y_control_pre). In regression form: Y_it = beta_0 + beta_1*Treat_i + beta_2*Post_t + beta_3*(Treat_i * Post_t) + epsilon_it, where beta_3 is the DiD estimate of the treatment effect. Treat_i indicates membership in the treatment group, Post_t indicates the post-treatment period, and their interaction captures the differential change. Standard errors should be clustered at the group level to account for within-group correlation over time. With multiple pre and post periods, the regression extends to include time fixed effects and group fixed effects, and the treatment indicator captures the staggered adoption of treatment across groups. Tools for DiD analysis include the R packages did, fixest, and the Python library differences, as well as general regression tools in any statistical software.

DiD should be used when you have a natural comparison group that did not receive the treatment and pre-treatment data for both groups. The critical assumption is parallel trends: absent the treatment, both groups would have followed the same trajectory. This assumption is untestable (since we cannot observe the counterfactual) but can be assessed by examining whether the groups followed parallel trends in the pre-treatment period. If pre-treatment trends diverge, the DiD estimate is biased. Common pitfalls include having too few treated or control units for reliable inference (clustered standard errors require at least 20-30 clusters), violating parallel trends due to differential pre-existing trends, and ignoring anticipation effects where the treatment group changes behavior before the official treatment date. When parallel trends are questionable, alternative methods include synthetic control (which constructs a data-driven comparison group) and changes-in-changes (which relaxes the parallel trends assumption to parallel quantile trends).

Advanced DiD methods have seen rapid development in recent years. Staggered DiD, where different units adopt treatment at different times, introduces complications because the standard two-way fixed effects estimator can produce biased estimates when treatment effects vary over time. Recent econometric research by Callaway and Sant'Anna, Sun and Abraham, and others provides corrected estimators for staggered adoption settings. Event study designs extend DiD to trace out the treatment effect dynamics over time, plotting the differential change for each period relative to treatment adoption. For advertising and marketing applications, DiD is commonly used in geo-experiments where campaigns are launched in some cities or regions but not others, with platforms like GeoLift (from Meta) providing end-to-end tooling for designing, analyzing, and interpreting geographic DiD experiments.