After being published, research findings often become a subject of replication efforts. Such replication efforts may be especially important when the data represent a breakthrough and designate a new direction into which science may develop. This is certainly the case when the published work announces a hope for a new therapeutic to be developed. Studies are repeated not only because there is a lack of trust in the original reports but because the confirmatory study (preferably conducted by an independent research group) is a pre-requisite for securing financial and other resources or is an essential element of a drug discovery project (e.g. a positive control needs to be established).
High-impact studies are often the result of many years of work conducted by a multi-disciplinary team that used very sophisticated state-of-the-art technologies. This may be a challenge for replication efforts that are sometimes not possible because of lacking skills or access to certain resources or research tools. However, if the motivation is high enough, these obstacles can be successfully overcome (especially, for powerful research organizations with significant financial resources). Yet, there are cases when such challenges become nearly unsurmountable:
In a recent paper by Mills et al. (Cell Metabolism 24 (2016), 795–806), nicotinamide mononucleotide (NMN), a key natural NAD+ intermediate, “effectively mitigated age-associated physiological decline in mice without any obvious toxicity”. These results were taken to suggest “the significant potential of NMN as an effective anti-aging intervention in humans”. As exciting and promising as it may sound, this study will be difficult to repeat to confirm such claims because … NMN was administered over a period of 12 months.
Does it mean that the breakthrough research should not take place if replication studies are not easy to perform? Of course, not. However, in cases when replication studies may for some reason be difficult to execute, planning, conduct and reporting of the potentially critical studies should be done with extra rigor. To achieve this, all stakeholders should combine their efforts, from those who conceive, fund, approve and conduct the study to those who review and accept it for publication. Even if, at the beginning, such a study is discussed and viewed as an exploratory effort, it needs to be planned with the rigor of a confirmatory experiment.
By careful examination of the published report it is usually possible to estimate the level of data robustness. In this context, one may look for information on blinding, randomization, power, etc. For example, in the case of the Mills et al report, we felt that having the results presented as scatterplots or boxplots (rather than means ± SEMs) would help to understand how robust the presented data sets are. We have therefore approached professor Shin-ichiro Imai, the senior author on this paper, who responded promptly and was very collaborative and helpful. However, Prof. Imai told us that, according to the University policy, he is not allowed to freely distribute scientific data to a third party organization. A very unfortunate policy given that, without looking at the raw data, it is difficult to follow the data analysis performed. For those of our readers who are interested in this particular case study, we recommend to have a closer look at figures such as Fig. 3F and try to understand why and how the Wilcoxon’s test is applied to analyze experiments involving independent groups of unequal size.
The above mentioned report by Mills et al. presents an interesting case to illustrate the need for particularly rigorous design, analysis and reporting for difficult-to-repeat studies. If one finds these results to be not robust and decides not to run a follow-up study, a potentially important finding might remain unconfirmed and may not lead to further discoveries. On the other hand, if a replication study is performed and the results are negative, we will be told that the original findings did not look robust and it was not worth wasting time and resources.