Search
Search Menu

Is N-Hacking ever OK?

It has been proposed repeatedly that adding samples based on results of initial experiments is a form of p-hacking (see e.g. new Instructions to Authors of journals of the Am Soc Exp Pharmacol Ther). While these recommendations were based on sound theoretical considerations, Pamela Rainagel from San Diego demonstrates in a manuscript not yet peer reviewed that the impact on false positives based on Monte-Carlo simulations of dynamically adjusting sample size, called n-hacking by her. Interestingly, her analysis shows that that n-hacking increase false positives and that effect sizes and prior probability are key drivers of this.

However, her simulations also suggest that the positive predictive value increases and is higher than that from non-incremental experiments. Apparently, the increase in false positives is more than offset by that in true positives. She proposes that post-hoc increases in sample size must be disclosed but could confer previously unappreciated advantages for efficiency and positive predictive value. However, she also warns that adaptations of sample sizes need careful considerations of correction of p-values because n-hacking essentially is a form of increasing the statistical alpha.

Therefore, the proposal by Pamela Rainagel does not invalidate the argument that findings resulting from n-hacking/adapted sample sizes are no longer suitable for hypothesis-testing statistical analysis unless careful pre-specification is in place to adjust for the increased alpha.

Leave a Comment

Required fields are marked *.


This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll Up