Research Guide

Survey Research Statistics: A Practical Guide

The core statistics every survey researcher should understand — response, completion and incidence rates, sample size, margin of error and significance — and how they fit together.

Survey research lives on a handful of numbers: how many people responded, how many finished, how many qualified, how precisely the results estimate the population, and whether differences between groups are real. Individually each is simple; the skill is reading them together.

This guide ties the standard survey statistics together and shows where each PanelRoster calculator fits.

Response, completion and incidence rates

The response rate is completed surveys divided by the number of eligible people invited. The completion rate is surveys finished divided by surveys started — it measures drop-off once someone begins. The incidence rate is the share of screened people who qualify for the study. Each rate has a different job: response rate speaks to representativeness risk, completion rate to survey usability, and incidence rate to how many people you must screen to fill a quota.

Sample size and margin of error

The sample size determines the precision of your estimates. The margin of error for a proportion is E = z·√(p̂(1−p̂)/n), and the required sample size for a proportion is n = z²·p(1−p)/E². Precision grows with the square root of the sample size, so quadrupling the sample halves the margin of error. The PanelRoster sample size and margin of error calculators implement these directly.

Statistical significance in surveys

When you compare subgroups (men vs women, region A vs region B), the difference you see could be sampling noise. A statistical significance test asks how likely that difference is under the null hypothesis that the groups are the same. The PanelRoster significance calculator performs this two-proportion comparison, and the p-value calculator converts any z or t statistic to a p-value.

Remember that significance is not the same as importance: with a large sample, a tiny and practically irrelevant difference can be "statistically significant".

The honest interpretation

Every survey estimate carries two kinds of error: sampling error (captured by the margin of error) and non-sampling error (coverage gaps, nonresponse bias, question wording, measurement error). Good statistics cannot fix a biased sample. Report response rates, state the margin of error, and describe the sampling design when you present results.

Key takeaways

Response rate measures representativeness risk; completion rate measures drop-off; incidence rate measures how many to screen.
Sample size drives precision: n = z²·p(1−p)/E² for a proportion at a chosen margin of error.
Significance tests compare observed differences to what sampling noise would produce.
Sampling error is only part of survey error — bias is not captured by any margin of error.

Tools used in this guide

References

References are provided for further reading; PanelRoster is not affiliated with the linked resources.