eNeuro editorial: Scientific Rigor or Rigor Mortis?
Christophe Bernard, editor in chief of eNeuro, discusses in this editorial the issue of scientific rigor with a reference to two commentaries by Katherine Button and Oswald Steward. To show that it is not a novel phenomenon, he gives the fitting example of a dispute between Louis Pasteur and Claude Bernard. Interestingly, though, since that time, clear guidelines and proper training do not have the full attention from the scientific community they deserve. To overcome this problem, scientists should be more critical with their own observations and make a clear statement when they perform exploratory studies. Secondly, scientists should receive better training related to Good Research Practices. eNeuro is establishing a webinar series to tackle the scientific rigor issue.
Commentary by Katherine Button: Statistical Rigor and the Perils of Chance
Button discusses the role of chance in statistical inference and how poor study design can lead to a high number of false-positive data. Furthermore, she claims that the current publication and funding system perpetuates this problem by encouraging the selective publication of positive data.
Commentary by Oswald Steward: A Rhumba of “R’s”: Replication, Reproducibility, Rigor, Robustness: What Does a Failure to Replicate Mean?
The commentary by Steward points out the many issues that are common practice in the daily lab routine and follow-up publications. He refers to “Begley’s 6 red flags” and provides a second list of points which he suggests to be called “the 6 gold stars of rigor”. These gold stars are suggested to be implemented as common publishing practices and e.g. include reporting of statistical power, requirement to report timing on data collection and to report all analysis in this context.

Functional magnetic resonance imaging (fMRI) is 25 years old, however, most common statistical methods used for analyses have not been validated using real data. In addition, an investigation of 241 fMRI studies showed that 223 unique analysis strategies were used. This means that almost no strategy occurred more than once – although results can vary markedly depending on the analysis strategy (Carp, Neuroimage, 2012). In the context of a typical fMRI experiment, that could lead researchers to wrongly conclude that activity in a certain area of the brain plays a role in a cognitive function such as perception or memory.
In line with this, a new study ‘Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates’ published in PNAS now reports that common settings used in software for analyzing brain scans may lead to a high rate of false positive results. Researchers led by Anders Eklund analyzed fMRI data from several public databases. Based on a significance threshold of 5% found in most publications, they expected to get a false positive result in 5% of the cases. Instead, depending on the software and the settings, the team found a false positive result up to 70% of the time.