Research Guide

Understanding P-Values and Statistical Significance

What a p-value actually means, what it does not mean, the role of the significance level, and why the field is moving away from the 0.05 bright line.

A p-value is the probability of observing a test statistic as extreme as — or more extreme than — the one you got, assuming the null hypothesis is true. It is a measure of evidence against the null hypothesis: smaller values indicate less compatibility between the data and the null.

P-values are widely used and widely misunderstood. This guide explains what a p-value is, how significance testing works, and the specific misinterpretations statisticians warn against.

How a p-value is computed

You state a null hypothesis (for example, "the true mean equals 50"), compute a test statistic from your data (a t or z value), and then ask: if the null were true, how likely is a statistic this extreme? That probability is the p-value. The PanelRoster p-value calculator computes it from a z or t statistic for one- or two-tailed tests.

The significance level α

The significance level α is a pre-chosen threshold for declaring a result "statistically significant", conventionally 0.05. If p ≤ α, the result is usually called statistically significant — meaning the data are unlikely under the null hypothesis. Choosing α in advance, before looking at the data, is good practice and prevents threshold-hunting.

The 0.05 convention is exactly that: a convention. It is not a law of nature, and it does not mark the boundary between true and false effects.

One-tailed and two-tailed tests

A two-tailed test asks whether the statistic differs from the null in either direction and is the common default. A one-tailed test asks whether it differs in a specified direction (greater than, or less than). One-tailed tests have more power in the specified direction but require a directional hypothesis stated before data collection.

What a p-value does NOT mean

A p-value is not the probability that the null hypothesis is true, and it is not the probability that your result happened by chance. It does not measure effect size or practical importance. A tiny p-value with a trivial effect (for example, a difference of 0.01 units) can be statistically significant but practically meaningless.

The American Statistical Association's 2016 statement on p-values lists these misconceptions and recommends reporting effect sizes and confidence intervals alongside p-values. Statistical significance is not the same as practical significance, and no single result should be treated as a scientific conclusion.

Key takeaways

A p-value is the probability of data this extreme under the null hypothesis — nothing more.
Significance is a pre-chosen threshold (α), not an objective property of the result.
A small p-value says nothing about the size or importance of an effect.
Report effect sizes and confidence intervals alongside p-values.

Tools used in this guide

References

References are provided for further reading; PanelRoster is not affiliated with the linked resources.