| Abstract: |
Selective publication and reporting in individual papers compromise the
scientific record, but are meta-analyses as compromised as their constituent
studies? We systematically sampled 63 moderately large meta-analyses (at least
40 studies per meta-analysis) in PLOS One, top medical journals, top
psychology journals, and Metalab, an online, open-data database of devel-
opmental psychology meta-analyses. We empirically estimated publication bias
in each. Across all meta-analyses, “statistically significant” results in the
expected direction were only 1.20 times more likely to be published than
“nonsignificant” results or those in the unexpected direction (95%CI: [0.94,
1.53]), with a confidence interval substantially overlapping the null.
Comparable estimates were 0.82 for meta-analyses in PLOS One, 1.23 for top
medical journals, 1.54 for top psychology journals, and 4.68 for Metalab. We
estimated that for 87% of meta-analyses, the amount of publication bias that
would be required to attenuate the point estimate to the null exceeded the
amount of publication estimated to be actually present in the vast majority of
meta-analyses from the relevant scientific discipline (exceeding the 95th
percentile of publication bias). Study-level measures (“statistical
significance” with a point estimate in the expected direction and point
estimate size) did not indicate more publication bias in higher-tier versus
lower-tier journals, nor in the earliest studies published on a topic versus
later studies. Overall, the mere act of performing a meta-analysis with a
large number of studies (at least 40) and that includes non-headline results
may largely mitigate publication bias in meta-analyses, suggesting optimism
about the validity of meta-analytic results. |