There are a large number of ways how to analyze the same data set. Depending on the decision made in handling, analyzing and presenting the data the outcome may be very different. Very often, there is a large uncertainty about what variable should be modeled and how they are related. This ‘availability of alternative options in model selection’ was termed ‘vibration of effects’ (VoE) by John P.A. Ioannidis in 2008 (REF) and describes the extent to which an estimated association changes under multiple distinct analytical modelling approaches. Thus, the VoE is related also to the previously described concept of ‘multiple testing’ (LINK).

As an example, the VoE can be illustrated by evaluating the association between Vitamin E (a-tocopherol) uptake and mortality. Early publications of observational studies claimed large reductions in disease-related and mortality-related events in association with Vitamin E. However, clinical trials that followed were not able to support these early observations.
Figure 1 shows the range of estimates of the Hazard Ratio (HR) that can be obtained by varying sets of adjusting variables.

For this purpose a set of 13 varying adjustment variables (e.g. smoking, BMI, alcohol consumption, education, income, hypertension, etc.) were chosen, resulting in a total number of 2^13 = 8,192 different combinations of adjustments. Next, the HR and the respective p-value for the association of a variable of interest with all-cause mortality were calculated for all 8,193 models. The vibration plot (Figure 1) illustrates that HR can be both greater and less than the null value depending on the chosen adjusting variables. HR<1 suggests that the variable of interest is associated with increased mortality while HR>1 suggests a decreased mortality.

The phenomenon that the association between higher levels of a-tocopherol can point to both higher and lower risk for mortality depending on the choice of the adjustment variables in the model was called the Janus effect (REF) after the two-headed representation of the ancient Roman god (Figure 2).
This large VoE and the sensitivity of results towards model choice may explain, at least in part, the inability to replicate the discrepancies mentioned above between first observational associations and subsequent randomized trials. A VoE analysis (and whether or not a Janus effect exists) may therefore help avoiding surprises when some unstable associations can not be replicated in follow-up studies.

VoE per se are not a problem if models chosen for analysis were selected a priori and selective reporting (i.e. several models are tested and only those with the most impressive results are presented) and cherry-picking could be avoided. Here, incentives for scientists must change to support reporting of transparent and unbiased results. In addition, by continuously updating meta-analysis data sets when new studies or independent replication attempts become available, given enough time, the true risk estimate will eventually reveal itself…