Contact
Acting Head of Research Synthesis Methods
Publication Bias
Funnel Plot
Funnel plots can be used as a graphical indication for publication bias. It is a scatterplot of the estimated effect sizes from each study and a measure of study size or effect size precision, here the standard error of the estimation.
The precision in the effect sizes should increase, as the standard error decreases. That means, that results from smaller studies are expected to scatter widely at the bottom of the graph, while estimates from larger studies should be nearer to the average estimate.
In the absence of bias, the plot will therefore look like a symmetrical, inverted funnel.
In case of publication bias, we would expect smaller studies without statistically significant effects to remain unpublished. This would lead to a lack of small studies with small effects.
A statistical test for this asymmetry is the Egger’s test (Egger, Davey, Schneider, & Minder, 1997). The null hypothesis is, that there is no association between the estimated effect size and the study size or precision. As test statistic, a z-value is given. If the Egger’s test is significant (p<0.05), we can assume, that there is substantial bias in the reported results included in the meta-analysis.
Power enhanced funnel plot
An enhancement to the classical funnel plot is the contour-enhanced funnel plot (Peters et al., 2008). It allows to take into account the statistical significance of the outcomes for the evaluation of potential publication bias. The contour-enhanced funnel plot is centered at 0 (null hypothesis of no effect) and the colored regions indicate different levels of statistical significance. Findings within the white region are not significant and thus, in case of of a lack of small studies (with higher standard errors) in the white region, this would be an indication for publication bias, as small studies with non-significant results are expected to remain unpublished. If there is asymmetry suggesting missing studies in areas of statistical significance, publication bias is not likely to be the reason for the asymmetry.
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of clinical epidemiology, 61(10), 991–996. https://doi.org/10.1016/j.jclinepi.2007.11.010.
p-curves
The p-curve is a tool to assess the evidential value of a set of published findings by examining the distribution of statistically significant p-values (p<0.05). It assumes this distribution to be a function of the real underlying effect. If there is no real effect, the p-values are expected to be uniformly distributed, as the red dotted line in the plot indicates. If there really is an effect, smaller p-values are more likely to be observed, resulting in right-skewed p-curves. The green dashed line shows the shape of a hypothetical curve of 33 % power. In case of higher statistical power of the underlying results, the curve is even more right-skewed. The plot also provides a power estimation for the blue observed p-curve. A left-skewed curve indicates p-hacking, as researchers may stop collecting more data as soon as findings achieve statistical significance. This results in large significant p-values, causing a left-skewed distribution.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). p-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results. Perspectives on Psychological Science, 9(6), 666–681. https://doi.org/10.1177/1745691614553988
Learn more about us! 
