Lesson
Small P Is Not Proof
A small p-value says the data would be surprising under a null model, not that the claim is probably true or important.
By the Math Says Yes editorial team
Human-reviewed under our source and correction standards.
How we review content
The question p answers
A starts by assuming a specific null model, usually a world where there is no real effect, no difference, or no association of the kind being tested. It then asks how often that model would produce data at least as extreme as what you observed. The p-value is therefore a statement about the data under an assumption. It is not a direct statement about the assumption itself.
The swaps that mislead
The common mistake is to reverse the conditional . People read p = 0.03 as a 3% chance that the result is a fluke, or a 97% chance that the hypothesis is true. The does not say either thing. It assumes the null model and measures the surprise of the observed data. To judge whether a claim is true, you also need the plausibility, study design, measurement quality, , and whether similar studies agree.
Significant is not important
and practical significance are different. With a huge , a tiny effect can produce a very small , even if the difference is too small to matter in real life. With a small sample, an important effect may fail to reach a conventional cutoff because the study is too noisy. The p-value tells you how incompatible the data are with the null model, not whether the effect is large enough to change a decision.
How to read it
Use a as one diagnostic, not as a verdict. Read it beside the , the , the , the number of tests tried, and the study design. Ask whether the hypothesis was planned before seeing the data, whether the measurement is credible, and whether the same pattern appears elsewhere. A small p-value can be a useful signal; it becomes dangerous only when it is treated as proof by itself.
FAQ
Does p = 0.05 mean there is a 95% chance the effect is real?
No. It means that data at least this extreme would occur about 5% of the time under the tested null model. The probability the effect is real requires more information than the p-value alone contains.
Can a tiny p-value still describe an unimportant result?
Yes. Large samples can make very small effects statistically detectable. You still need the effect size and real-world context to decide whether the detected difference matters.
Core Concepts
Conditional Probability
The probability of something after new information changes what is still possible. The important move is to update the denominator: cases ruled out by the new information should no longer be counted. Many mistakes happen when people keep using the original odds after the situation has narrowed.
P-Values
A p-value is the probability of seeing data at least this extreme if a specific null model were true. It is not the probability that the claim is false, not the chance the result was random, and not a measure of how large or important the effect is.
See it in action
A p-value is not the chance your result is wrong
A p-value tells you how unusual the data would be under a chosen starting assumption—often 'no effect.' It does not tell you whether a claim is true or how large the effect is.
P-value vs confidence interval: what each one tells you
P-values and confidence intervals answer different questions. Use the p-value for evidence against a null model, and the interval for the size and uncertainty of the effect.
Try the idea
P-value vs. confidence interval
Move the estimate and watch both summaries change.
estimate: +1.5
null value
estimate
-0.5 to +3.5
95% interval
.134
p-value
At an estimate of +1.5, the 95% interval still spans the null value and p = .134. The two agree: an interval that includes 0 is exactly the case where p stays above 0.05.
Quick Check
A study reports p = 0.03. Which plain-language translation is correct?
A
There is a 97% chance the study's claim is true.
B
The effect is large enough to matter in practice.
C
Results at least this unusual would occur about 3% of the time if there were really no effect.