Reproducibility in systems biology modelling

A recent report by Tiwari et al. investigated the reproducibility rate in systems biology modelling by reproducing the mathematical representation of 455 kinetic models. The authors tried to 

1.) reproduce the published model (step 1), 

2.) if failed, adjust their efforts based on experience (step 2), 

3.) if failed again, contact the authors of the original study for clarification and support (step 3).

When attempting to reproduce the selected models based on the information provided in the primary literature (step 1), only 51% of the models could be reproduced, meaning that the remaining 49% needed additional efforts (i.e. via steps 2+3). However, 37% of the total articles could not be reproduced by Tiwari and colleagues at all, even when adjusting the model system or asking the authors of the original study for support.

Notably, over 70% of the corresponding authors did not respond when contacted by Tiwari et al and in half of the responses it was not possible to reproduce the model, even with the support of the authors.

This low reproducibility rate, in combination with the very low response rate of the original authors makes it absolutely necessary to have very good reporting standards in the original study and to have them checked by the peer reviewers.

To improve the situation for systems biology, Tiwari and colleagues provided specific reporting guidelines in form of a checklist with eight points to increase the reproducibility of systems biology modelling.

Newly developed ‘Harmonized animal research reporting principles’ (HARRP) published

The International Council for Laboratory Animal Science (ICLAS) has recently published Harmonized Animal Research Reporting Principles (HARRP), the first step of a unified approach by ICLAS to improve research reporting standards worldwide.
ICLAS was established in 1956, under the auspices of UNESCO with the goal of advancing human and animal health by promoting the ethical care and use of laboratory animals in research worldwide.
There have been many sets of reporting guidelines published over the past 30 years. However, despite endorsement of these guidelines (in particular the ARRIVE guidelines) by hundreds of journals, their implementation by editors and authors alike remains quite variable.
As an international organization focused on laboratory animal welfare, ICLAS is concerned with the current crisis of reproducibility of animal-based research. If animal-based studies cannot be reproduced, if the quality of the information published is so poor that the studies cannot contribute to systematic reviews and if subsequently the studies are not translatable, then animals’ lives are being wasted.
ICLAS has always worked to encourage improvements in the quality of science where laboratory animals are involved. This occurs most notably through its Laboratory Animal Quality Network, as well as through harmonization exercises to define key principles for the oversight of animal-based research. With this in mind, ICLAS sought to encourage the uptake of reporting guidelines worldwide. The ICLAS working group (Members of the working group were representatives of groups with current reporting guidelines and/or experts in research reporting) charged with harmonizing reporting principles used the three main sets of reporting  standards available at the start of the exercise as a basis (the Gold Standard Publication Checklist, the ARRIVE guidelines; and ILAR’s Guidance on the Description of Animal Research in Scientific Publications). The structured analysis of these guidelines, subsequent development of the principles and their review is further described in Osborne et al (2018).
ICLAS’s intention is not to replace the excellent reporting guidelines already in existence. Rather, the intention is to improve the quality of animal-based research worldwide by providing a set of principles that can be easily translated into other languages. The HARRP can then be used as a minimum standard achievably globally, and upon which more technical guidance can be built.

Study Design

These articles are presenting ideas on how to improve experiments.

RIGOR guidelines

Comprehensive guidelines especially designed for research in animal models of stroke. However, it does contain basic aspects that can be applied to other animal experiments. LINK

MIATA guidelines

This webpage provides useful information on how to publish data derived from flow cytometry experiments in particular when analysing T-cells and NK-cells. It describes the set of information and data to be included in material and method sections to be able to understand and reproduce published experiments.

LINK: http://miataproject.org