Triggered Analysis

Triggered analysis improves experiment sensitivity by focusing on the subset of users who actually experienced the difference between treatment and control. In many experiments, a large fraction of assigned users never encounter the changed feature during the experiment window. For example, if an experiment changes the checkout flow, users who never reach checkout during the experiment period add noise without contributing signal. Triggered analysis restricts the comparison to users who were triggered, meaning they reached the point in the product where treatment and control diverge. For growth teams, triggered analysis can dramatically increase the effective power of experiments, often reducing required sample sizes by 50-80%, enabling faster decisions and the ability to detect smaller effects.

The implementation of triggered analysis requires that the triggering event (the point where users would experience the change) is logged identically for both treatment and control users. This is critical: the trigger must be defined based on reaching a point in the user flow, not on seeing the treatment. If the trigger is only logged for treatment users who see the new UI, but not for control users who see the old UI at the same point, the analysis is invalid. The correct approach logs the trigger for all users who reach the relevant product surface, regardless of their assignment. The analysis then compares outcomes between triggered treatment users and triggered control users. Because both groups were randomly assigned and the trigger condition is the same for both, the comparison preserves the causal validity of the original randomization. The variance reduction comes from eliminating the dilution caused by users who were never exposed to any difference.

Triggered analysis should be used whenever a significant fraction of randomized users never encounter the experimental change. It is particularly valuable for experiments on features deep in the user funnel (checkout changes, settings page modifications, advanced feature updates) where only a minority of users navigate to the affected surface during the experiment. Common pitfalls include defining the trigger based on treatment-specific behavior (which creates selection bias), not logging trigger events for the control group, and not recognizing that triggered analysis changes the estimand from the full-population ITT effect to the treatment effect among the triggered subpopulation. Teams should always report both the full-population ITT and the triggered analysis results, with clear labels distinguishing them. The triggered effect will be larger than the ITT effect because it is not diluted by unaffected users.

Advanced triggered analysis techniques include dilution-adjusted triggered analysis, which multiplies the triggered effect by the triggering rate to recover the full-population ITT estimate with reduced variance. This approach, described in papers from Microsoft Research and others, provides the best of both worlds: the precision of triggered analysis with the interpretability of the full-population effect. For experiments with continuous exposure intensity (where some users are heavily exposed and others barely), inverse propensity of exposure weighting can adjust for differential exposure while maintaining the causal interpretation. Some experimentation platforms implement triggered analysis automatically when trigger events are defined in the experiment configuration. The concept extends to heterogeneous triggering: different user segments may have different triggering rates, and the triggered treatment effect may vary across these segments, providing additional insight into who benefits from the change and why.