Jonas Ranstam's website


2. The challenge in statistical analyses


Joshua Rothman stated in an essay about rationality (published in the New Yorker on 16 August 2021) that the real challenge isn’t being right but knowing how wrong you might be. The statement is applicable also to statistical analysis, and it can help understand why the typical emphasis on statistical significance is a mistake.

P-values are often erroneously believed to represent the probability that a tested hypothesis is true and that the p-value, therefore, is a valuable tool for finding the truth. The belief is wrong because the p-value does not say anything about the truth. It just indicates how incompatible data are with a specified statistical model. The mistake is usually combined with a failure to recognise that estimates of effect sizes and their inferential uncertainty are the interesting quantities.

Instead of trying to reveal the truth by testing observed data, the investigator should recognise that sampled data suffer from sampling variation, and that the effects seen in one sample are likely to differ from the effects in other samples from the same population. The relevant question is how much an observed effect can vary. This is where a confidence interval is useful.