Sample Ratio Mismatch

Sample ratio mismatch (SRM) is one of the most important diagnostic checks in online experimentation, yet it is frequently overlooked. If an experiment is designed to split traffic 50/50 between control and treatment, the observed counts should be approximately equal, with some random variation. An SRM test uses a chi-squared goodness-of-fit test to determine whether the observed ratio deviates significantly from the expected ratio. A significant SRM (typically p < 0.001 given the large sample sizes involved) indicates that something in the experiment implementation is systematically biasing which users end up in which group, which violates the fundamental assumption of random assignment and can invalidate all causal conclusions. For growth teams, SRM is a canary in the coal mine: it does not tell you what went wrong, but it tells you that something did.

Common causes of SRM include bugs in the randomization code, differences in page load performance between variants causing differential bot filtering or user abandonment, redirect-based experiments where one variant's redirect fails more often, initialization timing differences where the assignment check happens at different points in the user flow, and browser or client-side caching that affects variant delivery inconsistently. For example, if the treatment variant loads 200ms slower than control, some treatment users may bounce before their visit is logged, creating an SRM where control has more recorded users. Another common cause is interaction between experiments: if experiment A's treatment causes some users to never reach experiment B's assignment point, experiment B will show an SRM among users exposed to experiment A's treatment. The SRM test is computed as chi_squared = sum((observed_i - expected_i)^2 / expected_i) across all groups, compared to a chi-squared distribution with k-1 degrees of freedom.

Every experiment should include an automated SRM check that runs continuously and alerts when a mismatch is detected. The check should be performed early in the experiment (often within the first day) so that problematic experiments can be stopped before they waste more traffic. When an SRM is detected, the experiment results should be considered invalid until the root cause is identified and resolved. Common debugging steps include checking for differential logging between variants, examining bot traffic patterns, verifying that the randomization hash function produces uniform distribution, checking for interactions with other concurrent experiments, and examining whether variant-specific errors or timeouts could cause differential data loss. Teams should never try to fix an SRM by reweighting or adjusting the data; the goal is to identify and fix the underlying implementation bug.

Advanced SRM analysis includes checking not just the overall ratio but also the ratio over time (a sudden shift may indicate a deployment or configuration change), across different platforms or geographies (a platform-specific SRM points to a client-side implementation issue), and across different user segments. Some experimentation platforms like Statsig and Eppo run SRM checks automatically and flag experiments with mismatches. The concept extends beyond simple two-group experiments: for multi-arm experiments, factorial designs, and layered experiment systems, SRM checks should verify all expected ratios including interaction cells. Research by Fabijan, Dmitriev, and others at Microsoft has documented that SRM affects a significant percentage of experiments at major tech companies and is often the first indicator of subtle bugs that would otherwise silently bias results for months.