In the biomedical field, a significant proportion of preclinical research data sets are not robust enough to translate into successful clinical programs and a lot of resources are wasted on research that turns out to be irreproducible.
In this paper, published in Cell Systems in July 2019, Mario Niepel and colleagues from five different research centers have attempted to reproduce the results of an assay in which cultured cells were treated with anti-cancer drugs. Their lack of success highlights the importance to identify which factors and variables most likely affect the outcome of an experiment and should therefore be considered when planning, conducting and reporting biomedical studies.
In the initial experiments, the results showed drug potencies that could vary up to 200-fold between research centers. The researchers identified several technical factors which could (partially) explain the obtained discrepancies. For example, different methods for measuring cell numbers were used and it was found that cell counts using a microscope did not correlate well with measuring ATP levels in lysed cells. Groups also used different image-processing algorithms to count live cells. Furthermore, cell culture plates in which cells were grown showed edge effect artefacts from uneven evaporation of culture media and temperature gradients, which led to variation in results between labs.
Once the teams addressed these issues using a more standardized protocol and randomization steps for cell culture plates, the within-group and between-group robustness of experiments increased but results were still more consistent between scientists from the same group than from different groups. According to the authors, this could be due to possible differences in pipetting technique, variations in equipment, or ignoring the protocol because “counting cells is such a simple procedure that different assays can be substituted for each other without consequence.”
Following the protocol exactly is only part of the story. Another important part of understanding reproducibility is that many experiments depend on some biological or technical factors that are unknown and therefore unmeasured. However, ultimately, the point is not to just get reproducible effects – the point is to get reproducible results that are robust to subtle changes in the experiment and are therefore able to support successful translational drug discovery programs.