So much that just this year the American Statistical Association put out a 12-page manuscript about p-values and it took them a year of discussion(!) before the manuscript was complete.
See also this very short 2006 article by Andrew Gelman and Hal Stern The Difference Between “Significant” and “Not Significant’ is not Itself Statistically Significant:
The error we describe is conceptually different from other oft-cited problems—that statistical significance is not the same as practical importance, that dichotomization into significant and nonsignificant results encourages the dismissal of observed differences in favor of the usually less interesting null hypothesis of no difference, and that any particular threshold for declaring significance is arbitrary…
In making a comparison between two treatments, one should look at the statistical significance of the difference rather than the difference between their significance levels. [Emphasis added].
And this related 2011 paper by Nieuwenhuis, Forstmann, and Wagenmakers, Erroneous analyses of interactions in neuroscience: a problem of significance, which found that half of the 160 papers reviewed, which all appear in top academic journals, used the wrong statistical procedure when evaluating p-values.
One thought on “How much disagreement is there about statistics?”