Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015
Being able to replicate scientific findings is crucial for scientific progress. The authors of this study published in Nature Human Behaviour tried to replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications followed analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications were high powered, with sample sizes on average about five times higher than in the original studies.
In summary, the authors could successfully replicated 13 out of 21 findings (62%), and the effect size of the replications was on average about 50% of the original effect size.
Importantly, however, the authors also estimated peer beliefs about replicability using surveys and prediction market approaches: the prediction market beliefs and the survey beliefs were highly correlated and both are highly correlated with a successful replication – that means that peers were very effective at predicting future replication success.
Interestingly, the described successful prediction market approach is in contrast to other reports in the cancer biology area. In a recent publication, Daniel Benjamin and his colleagues analysed the ability of cancer researchers to judge whether selected preclinical reports can be reproduced or not. On average, researchers forecasted a 75% probability of replicating the statistical significance and a 50% probability of replicating the effect size, yet none of these studies successfully replicated on either criterion (for the 5 studies with results reported).
At this point, it is not clear which factors caused these different findings and future studies will tell whether there is indeed a difference between the sociology and the cancer biology studies when it comes to predicting successful replication attempts.