Novelty Effect

The novelty effect is a pervasive threat to experiment validity that causes teams to overestimate the long-term impact of changes. When users encounter something new, whether a redesigned interface, a new feature, or a changed workflow, their behavior changes simply because it is different, not necessarily because it is better. They may click more because they are exploring, spend more time because they are relearning, or engage more because of curiosity. If an experiment is analyzed during this novelty phase, the treatment will appear to outperform the control, but the lift will fade as users habituate to the change. For growth teams, failing to account for novelty effects leads to systematically overoptimistic experiment results and a portfolio of shipped changes that collectively underdeliver on their promised impact.

Detecting novelty effects requires analyzing how the treatment effect changes over time within the experiment. The standard approach is to plot the daily or weekly treatment effect estimate and look for a declining trend. More formally, you can segment users by their exposure date and compare the treatment effect for users who were exposed early versus late in the experiment, or use a regression model that includes an interaction between treatment assignment and days since exposure. If the treatment effect is large in the first few days but diminishes over subsequent weeks, a novelty effect is likely present. Another diagnostic is to compare the treatment effect for new users (who have no baseline expectation and thus no novelty response) versus existing users (who experience the change as new). If the effect is much larger for existing users, novelty is a likely driver. Tools like Statsig allow you to view metric deltas over time within an experiment, making this analysis straightforward.

To mitigate novelty effects, teams should run experiments long enough for the novelty to wear off, typically at least 2-4 weeks for UI changes. Analyzing only the steady-state period after the initial novelty phase provides a more accurate estimate of long-term impact. For experiments where long duration is impractical, teams can focus on new users as a novelty-free segment, since they have no prior experience to compare against. Another approach is to compare the treatment effect trajectory against historical patterns from similar changes to calibrate expected novelty decay. Teams should be especially cautious about novelty effects when testing visual redesigns, notification changes, or any change to well-established user workflows where users have strong habits.

Advanced approaches to handling novelty effects include fitting parametric decay models (exponential or power law) to the time-varying treatment effect to extrapolate the steady-state impact, using difference-in-differences designs that compare the pre-post change for treatment users against the same period for control users to remove time trends, and implementing long-running holdout groups that measure the treatment effect months after launch. Some researchers have proposed Bayesian models that explicitly decompose the treatment effect into a novelty component and a permanent component, estimating both simultaneously. The mirror image of the novelty effect is the primacy effect, where users initially resist a change and perform worse with the treatment, but gradually improve as they adapt. Both effects argue for longer experiment durations and careful temporal analysis of treatment effect dynamics.