What role does sample size play in statistical significance?

Official answer

With a valid design and comparable independent observations, increasing sample size generally reduces sampling uncertainty and increases the power to detect a fixed true effect. For a sample mean with fixed finite population variance and independent observations, its standard error scales as 1/sqrt(n). Four times as many observations halve that standard error; they do not make it zero.

A larger sample can make a very small effect statistically detectable, so report its size, uncertainty and practical importance. A small sample may miss an important effect; failure to reject the null is not evidence of equivalence.

More rows do not necessarily mean more independent information. Repeated events from the same user, clustered assignments and serial dependence must be reflected in the design and analysis. Increasing a biased sample does not remove selection bias, confounding or systematic measurement errors.

Plan sample size around a meaningful effect, expected variation, randomization unit and desired error rates before the experiment. Do not simply collect more data until an ordinary fixed-horizon p-value crosses a threshold; a valid sequential procedure needs its own pre-specified rules.

Assessment: Separate precision from accuracy, statistical significance from usefulness, and raw row count from independent sample size.