Publication bias hides the studies that found nothing

If only exciting results get published, the visible research record can make an effect look cleaner and stronger than it really is.

Publication bias is the sample bias of science itself: positive, surprising, or significant studies are easier to see than null results.

By the Math Says Yes editorial team

Human-reviewed under our source and correction standards.

How we review content

THE TRAP

Treating the published literature as the full set of evidence, even though boring or null studies may be missing from view.

A stack of published positive studies on a desk and muted null studies hidden in a file drawer.

The missing evidence problem

A study with a striking positive finding is easier to write up, easier to place in a journal, and easier for journalists or readers to notice. A study that finds no clear effect can sit in a file drawer, remain as a conference abstract, or never be submitted. Research does not enter the public record evenly: the published literature becomes a selected , not a complete map of what researchers tried.

What the Numbers Show

published

in the file drawer

Illustrative: 3 striking results are published while 7 quiet ones stay hidden, so readers see a biased sample.

How it exaggerates effects

Imagine ten small studies testing the same weak effect. By , a few will look impressive and several will look boring. If the impressive ones are published and the boring ones disappear, the visible evidence will overstate the effect. None of the published studies need to be fake for this to go wrong. The missing studies carry information too, and silence can the .

Why intuition fails

Readers naturally treat search results and journal pages as what science knows. But a search can only find what was made visible. A pile of positive papers feels like repeated confirmation, even when it may partly reflect repeated filtering. This is the same sampling lesson as : the cases you can inspect are not automatically representative of the cases that existed.

How to use it

Look for pre-registered studies, systematic reviews, trial registries, and meta-analyses that search for unpublished work. Be cautious when a field has many small positive studies, few null results, and strong incentives to publish exciting claims. Ask not only what evidence is visible, but what evidence would have been visible if it had gone the other way.

Worked example

Imagine 20 small labs test a weak effect. Pure can make a few studies look impressive and many look unclear. If only the impressive studies are submitted, accepted, or shared, a reader later sees a neat pile of positive results. The missing studies are not neutral; they would have pulled the visible back toward a smaller or less certain effect.

When it applies

Publication is most concerning when studies are small, results are flexible, incentives reward novelty, and null results are hard to publish. It is less damaging when protocols are registered before data collection, outcomes are reported regardless of significance, and reviews search trial registries or unpublished work. The question is not only whether published studies are honest, but whether the visible set is complete enough to trust.

What people get wrong

People treat the published record as if it were the full experiment log. It is not. It is the visible subset that survived incentives, submission choices, editorial decisions, and attention. If missing null results are common, the visible literature can look more consistent and more impressive than the total evidence really is.

Source note

Rosenthal's file-drawer paper is the primary source behind the page's metaphor: unseen null results can change the apparent strength of evidence. The background source defines publication more broadly, while the trusted source anchors the specific idea that missing studies can systematically distort the published record.

FAQ

Is publication bias the same as fraud?

No. Publication bias can happen even when every published study is honestly reported. The bias comes from which studies become visible and which stay hidden.

How can publication bias be reduced?

Pre-registration, trial registries, registered reports, and systematic searches for unpublished studies all help because they make null results easier to count.