Statistical Significance Calculator
Check whether a control vs variant difference is statistically significant.
How statistical significance works
This calculator compares two proportions — for example a control group and a treatment or variant group in an A/B test — using a two-proportion z-test.
It reports the observed difference, the pooled standard error, the z statistic, and the two-tailed p-value. If the p-value is below your chosen significance level, the difference is statistically significant.
Statistical significance only tells you that the difference is unlikely to be due to random chance. It does not prove that one option caused the outcome, and a statistically significant result can still be practically unimportant.
The formula (pooled two-proportion z-test)
z = (p̂₁ − p̂₂) ÷ √(p̂(1−p̂)(1/n₁ + 1/n₂)) · p̂ = (c₁+c₂) ÷ (n₁+n₂)
p̂₁ and p̂₂ are the two observed conversion rates, p̂ is the pooled proportion under the null hypothesis, and the z statistic is compared to the standard normal distribution to get the two-tailed p-value.
Worked example
Control 200/1000 vs variant 250/1000
Rates are 20% and 25%. The pooled proportion is 22.5%, giving z ≈ 2.68 and a two-tailed p-value of about 0.007 — statistically significant at the 5% level.
Frequently asked questions
What assumptions does the test make?
Independent random samples and an adequate normal approximation (expected counts of at least 5 in each cell). Small samples can make the approximation unreliable.
What is the difference between statistical and practical significance?
Statistical significance means the result is unlikely under chance. Practical significance asks whether the size of the effect matters in the real world — a tiny but statistically significant difference may not be worth acting on.
Does a significant result prove causation?
No. An observational difference or even a controlled experiment only supports causation under strong design assumptions. This calculator only reports the statistical comparison.
Related research methods
- Hypothesis Testing
When to use this tool
Use this to check whether a p-value falls below your pre-specified significance level.
Assumptions
- A pre-specified significance level.
- The underlying test's assumptions are met.
Limitations
- Statistical significance is not the same as practical importance.
- The significance level controls the long-run Type I error rate, not the probability a specific result is true.
How to interpret the result
If p < α the result is conventionally called statistically significant: the data are unlikely under the null hypothesis.
How to report the result
State the significance level used and the p-value, and avoid overstating what 'significant' means.
Next steps in your analysis
Related tools
Related guides
- How to Choose the Right Statistical TestA practical framework for picking the right statistical test: start from your research question, data type and study design, then check the assumptions.
- Sample Size and Margin of Error: A Practical GuideHow sample size controls margin of error, the formulas behind both, the finite-population correction, and the practical caveats researchers should not skip.
- Understanding P-Values and Statistical SignificanceWhat a p-value actually means, what it does not mean, the role of the significance level, and why the field is moving away from the 0.05 bright line.
- Survey Research Statistics: A Practical GuideThe core statistics every survey researcher should understand — response, completion and incidence rates, sample size, margin of error and significance — and how they fit together.
Built by PanelRoster
PanelRoster is an enterprise operations management platform for organizations coordinating distributed teams, assignments, workflows, quality assurance and operational execution.
Explore PanelRoster