The File Drawer

A study that finds an effect is substantially more likely to be published than a study that does not. This has been known and named since the 1950s, formalised as the file drawer problem by Robert Rosenthal in 1979, and repeatedly measured since.

The mechanism is ordinary and requires no bad actors. Journals prefer positive findings because they are more interesting. Reviewers rate them more favourably. Researchers, knowing this, are less likely to write up a null result, and a career is built on publications. Every incentive at every stage points the same way, and no one in the chain is doing anything they would describe as wrong.

The consequence is that the published literature systematically overstates effects. If twenty studies are conducted, one finds an effect at the conventional threshold by chance, and only that one appears in print, then the literature reports an effect that is not there. Meta-analysis, which was supposed to solve this by aggregating studies, aggregates the published ones and inherits the bias.

The size of the distortion has been measured in specific cases. Work on antidepressant trials registered with the FDA compared registered outcomes against publication, and found that trials with favourable results were published at a far higher rate, while unfavourable ones were unpublished or published in a form that reframed the primary endpoint. The published literature and the regulatory record told different stories about the same drug.

The replication crisis is the same phenomenon arriving as a bill. Large coordinated attempts to reproduce published findings in psychology and in cancer biology succeeded at rates well below what the original literature implied. Effect sizes in successful replications were consistently smaller than in the originals.

What makes this belong in this archive is the shape. It is suppression with no suppressor. Nobody decided that null results should not be known. A distributed set of individually reasonable preferences produced a body of knowledge with a systematic and knowable distortion, and it did so for decades while everyone involved was following the rules.

That form is harder to address than an Index, because there is no one to argue with. It is also more common, and this archive should be more careful about it than about the dramatic cases, which are easier to spot precisely because they have authors.

The remedies are known and partially implemented: trial preregistration, registered reports where the journal accepts the design before results exist, mandatory results reporting, and open data. Where these have been applied, the rate of positive findings drops sharply, which is itself the strongest evidence that the problem was real.

The drop is worth dwelling on. When a field is required to declare what it will test before it tests it, the proportion of confirmed hypotheses falls dramatically. Nothing about nature changed. What changed was the opportunity to decide afterwards what the study had been about.

The published record of the twentieth century was assembled without that requirement, and it is still the foundation everything is built on.