Lesson
The Sample You See Matters
Your conclusion can be wrong when the data you see excludes important missing cases.
By the Math Says Yes editorial team
Human-reviewed under our source and correction standards.
How we review content
A sample is a filter
Every dataset reaches you through a collection process. That process filters who is included, who is excluded, who responds, who survives, who is visible, and who is measured accurately. A can be huge and still biased if the filter is pointed in the wrong direction. The first question is therefore not how many observations there are, but whether the observations represent the you want to describe.
The missing cases carry information
is powerful because the absent cases are often exactly the cases that would change the conclusion. Failed companies are missing from founder advice. People who disliked a product may never leave a review. Damaged aircraft that did not return cannot be inspected. When missingness is related to the outcome, the visible data does not merely have gaps. It points in a systematically misleading direction.
Precision does not fix bias
A biased can produce neat charts, narrow confidence intervals, and repeatable results while still answering the wrong question. More data reduces random noise, but it does not automatically repair a distorted collection method. If a survey only reaches one audience, adding more people from that same audience makes the estimate look more confident, not more representative. Sampling quality is a design issue before it is a calculation issue.
How to inspect a sample
Before trusting a claim, trace the path from to data. Who had a to be included? Who had a reason not to respond? Which failures, quiet users, small groups, or edge cases are invisible? Then ask whether the missing groups would probably have different outcomes from the visible groups. If the answer is yes, treat the conclusion as a statement about the , not the whole population.
FAQ
Can a very large sample still be biased?
Yes. Size reduces random error, but it does not fix a collection process that systematically excludes or overrepresents important groups.
What is the simplest way to spot sampling bias?
Ask who is missing and whether their outcomes would differ. If the missing cases are not random, the visible sample may describe itself more than the wider population.
See it in action
Why your friends seem more popular
The friendship paradox comes from pooling friend-list entries: well-connected people appear more often, so the pooled-entry average is at least the random-person average and is higher when friend counts vary.
Publication bias hides the studies that found nothing
Publication bias is the sample bias of science itself: positive, surprising, or significant studies are easier to see than null results.
Berkson's paradox: hospitals can create fake correlations
Berkson's paradox is collider bias in action: conditioning on being selected can create relationships that are not present in the full population.
A trend can reverse when groups are combined
Aggregation can hide the different sizes and risk levels of the groups being combined.
The missing planes mattered most
The planes that made it home are the survivors. The fatal damage is more likely where returning planes had fewer holes.
Quick Check
Why can a name pooled from all friend lists look more popular than a random person?
A
Most people have fewer friends than they think.
B
People tend to choose less popular friends.
C
Popular people appear in many friend-lists.