Professional Guide

Effect Size and Statistical Power: Designing Studies That Can Detect Effects

Why statistical significance is not enough — how to measure effect size with Cohen's d and design studies with adequate statistical power before you collect data.

A significant p-value tells you an effect is unlikely to be zero — not that it is big, or even worth acting on. For research, engineering and scientific decisions you need the size of the effect and the confidence that your study could have detected it. Those are effect size and statistical power.

This guide explains Cohen's d, the power concept, and how sample size, effect size, significance level and power connect.

Effect size with Cohen's d

Cohen's d measures the standardised difference between two group means: d = (mean₁ − mean₂) ÷ pooled standard deviation. Expressing the difference in standard-deviation units lets you compare effects across studies with different scales. Conventionally, d ≈ 0.2 is small, 0.5 medium and 0.8 large — but context beats convention.

The effect-size calculator takes the two group statistics (means, standard deviations, sample sizes) and returns d, with its interpretation. Always report effect size alongside any p-value.

Statistical power

Power is the probability that a study will detect an effect of a given size, if it truly exists. Power = 1 − P(Type II error). A study with power 0.8 has an 80% chance of detecting the effect you sized it for. Underpowered studies waste money and produce noisy, non-reproducible results.

The statistical-power calculator returns power (or the required sample size) from the effect size, significance level and sample size — the same inputs the sample-size calculator uses from the other direction.

The four-way relationship

Sample size, effect size, significance level and power move together: to detect a smaller effect you need more power or a larger sample; to use a stricter significance level you need a larger sample. The calculators let you fix three and solve for the fourth, which is exactly how a pre-study power analysis should work.

Do the power analysis before data collection, state the target effect size and power (commonly 0.80), and report the resulting sample size in the methods.

Key takeaways

Cohen's d = standardised difference between means; report it with every p-value.
Power = 1 − P(Type II error); 0.80 is a common design target.
Sample size, effect size, significance level and power are linked — fix three, solve for the fourth.
Run the power analysis before collecting data and report the target effect size.

Tools used in this guide

References

References are provided for further reading; PanelRoster is not affiliated with the linked resources.