Amyloid beta (Aß) accumulation in the brain is regarded as one of the histopathological hallmarks of Alzheimer’s disease. However, Aß build-up may not necessarily result in Alzheimer’s disease in all patients. A recent paper claimed that there is a unique combination of 15 proteins that distinguishes Aß build-up cases with and without a clinical picture of Alzheimer’s disease (AD and NDAN, respectively).
The purpose of this case study is NOT to reflect on the current discussions about Aß as a therapeutic target and is NOT about the scientific importance of efforts such as those described in this paper.
We present this case as an example that may be useful in discussions with students about importance of confirmatory experiments. We intentionally leave out all other details and scientific aspects of this study.
In this paper, post-synaptic densities isolated from frozen mid-hippocampus tissue samples from three groups (AD, NDAN, and controls) were subjected to proteomic analyses:
727 Seven hundred and twenty-seven individual spots were detected by Coomassie-stained 2D gel electrophoresis.
340 Three hundred and forty most abundant spots were collected for protein digestion. Following digestion, the resulting peptides were separated by liquid chromatography and the amino acid sequences were determined.
122 Using MS/MS, there were 122 proteins identified that had a p-value ≤ 0.05 in at least one comparison (i.e., AD versus control, NDAN versus control, or NDAN versus AD).
31 Since the goal of this work was to determine differences between AD and NDAN, there was a further subgroup of 31 protein identified that were changed (cut-off ≥ ± 1.5 fold) in NDAN versus AD, regardless of whether they were changed in either group as compared to controls.
15 Finally, there were 15 proteins identified with significant differential expression in NDAN versus AD:

To make calculations for this example easier, let’s forget about the initial number of analyzed proteins. There are still more than 300,000,000 ways to select 15 proteins out of a set of 31 (see HERE for a formula how to calculate such combinations).
So, if this were indeed a true unique combination of 15 specific proteins, it is difficult to get it just by chance. How is it possible to confirm that this is indeed not by chance? By running a similar analysis using a “validation” set of experiments (e.g. using independent samples of tissue).
In other words, the first proteomics approach was an exploratory study while the validation experiments should represent a confirmatory study proving that the observed pattern of changes did not happen by chance.
Fortunately, independent samples of tissue were available that could be used for these validation studies. However, to validate the protein changes observed by gel electrophoresis and immunoblotting, the authors selected only four (!) target proteins based on the fold change and availability of commercial antibodies. Additionally, a negative control was included – one protein that was not changed in the proteomic dataset.
As a result, the immunoblotting analysis confirmed differences between AD and NDAN samples for two out of four proteins tested.
Do these results de-validate the results of the original proteomics analysis? No, of course not.
However, the validation experiment presented in this paper demonstrate impressively the importance of proper design and careful planning of confirmatory studies to be able to generate meaningful results:

• The confirmatory study has used a sample size lower than the exploratory experiment (and the authors blame the inter-individual variability as the reason for failing to validate some protein changes)
• The validation set includes one protein of which four isoforms were presented in the proteomic dataset. Three isoforms had no change in NDAN versus AD that could mask the detection of changes in only one out of the four isoforms by the immunoblotting antibody (as the authors hypothesized)
• -omics studies have been repeatedly reported to be prone to generate false positive outcomes. Hence, it is essential to involve data analysis experts when planning such studies
• Perhaps most importantly: confirmation studies require a hypothesis to be stated very clearly BEFORE the study is conducted. Without such hypothesis presented, it is difficult to evaluate the outcome of these validation experiments. Does a confirmation for two out of four proteins validate the entire set of 15 proteins?

In the authors’ and reviewers’ mind, the answer to this last question is apparently “yes” and the conclusion is: “We describe fifteen proteins which comprise the unique proteomic signature of NDAN PSDs, thus setting them apart from control subjects and AD patients.”