Statistical Significance
A determination that an observed result is unlikely to have occurred by random chance alone, typically declared when the p-value falls below a predetermined threshold, usually 0.05.
Statistical significance answers the question: "Is this result real or just noise?" In A/B testing, you observe a difference between variants (e.g., variant B has a 3% higher conversion rate). Statistical significance testing determines whether this observed difference is likely due to the actual effect of your change or could plausibly be explained by random variation in user behavior.
The standard approach uses hypothesis testing. The null hypothesis assumes no difference between variants. You calculate a test statistic and corresponding p-value. If the p-value is below your significance level (typically 0.05, meaning a 5% chance of a false positive), you reject the null hypothesis and declare the result statistically significant.
Common pitfalls include peeking at results before the sample size is reached (inflating false positive rates), ignoring multiple testing corrections when evaluating many metrics simultaneously, and conflating statistical significance with practical significance. A result can be statistically significant but practically meaningless if the effect size is tiny. Always consider effect size and confidence intervals alongside p-values when making decisions.
Related Terms
Cosine Similarity
A measure of similarity between two vectors based on the cosine of the angle between them, ranging from -1 (opposite) to 1 (identical), commonly used to compare embeddings.
Dimensionality Reduction
Techniques that reduce the number of dimensions in high-dimensional data while preserving meaningful structure, used for visualization, compression, and noise removal.
Batch Inference
Processing multiple ML predictions as a group at scheduled intervals rather than one-at-a-time on demand, optimizing for throughput and cost over latency.
Real-Time Inference
Generating ML predictions on-demand as requests arrive, typically with latency requirements under 200ms for user-facing features.
Data Pipeline
An automated sequence of data processing steps that moves data from source systems through transformations to destination systems, enabling reliable and repeatable data flows across an organization.
ETL (Extract, Transform, Load)
A data integration pattern that extracts data from source systems, transforms it into a structured format suitable for analysis, and loads it into a target data warehouse or database.