Understanding the variability of criteria and thresholds for promotion and tenure applied across institutions requires a systematic empirical assessment. Therefore, the authors aimed to identify and document a set of pre-specified traditional (for example, number of publications) and non-traditional (for example, data sharing) criteria used to assess scientists for promotion and tenure within faculties of biomedical sciences among a large number of universities around the world. The study shows that the evaluation of scientists emphasises traditional criteria as opposed to non-traditional criteria. This may reinforce research practices that are known to be problematic while insufficiently supporting the conduct of better quality research and open science. The authors conclude that institutions should consider incentivising non-traditional criteria.