Another Proof Against P-Value Reasoning

This isn't so much an extra proof, but a clarification on one of the proofs used in "Everything Wrong With P-values Under One Roof" (the first two arguments).

Calculation of the p-value does not begin until it is accepted or assumed the null is true: p-values only exist when the null is true. Now if we start by accepting the null is true, logically there is only one way to move from this position and show the null is false. That is if we can show that some contradiction follows from assuming the null is true. In other words, we need a proof by contradiction in the following way:

• If "null true" then some proposition Q is true;
• Not-Q (Q is false in fact);
• Then "null true" is false; i.e. the null is false.

Yet there is no proposition Q in frequentist theory consistent with this kind of proof. Indeed, under frequentist theory, which must be adhered to if p-values have any hope of justification, the p-value assuming the null is true is uniformly distributed. This statement (the uniformity of p) is the only Q available. There is no theory in frequentism that makes any claim on the size of p except that it can equally be any value in (0,1). And, of course, every calculated p (except in some circumstances to be mentioned presently) will be in this interval. Thus we have:

• If "null true" then Q = "p ~ U(0,1)";
• p in [0,1] (note the now-sharp bounds).

We cannot move from observing p in (0,1), which is almost always true in practice, to concluding that the null is true. This would be the fallacy of affirming the consequent. On the other hand, in the cases where p in {0,1} (the set with just elements 0 and 1), which happens in practical computation when the sample size is small or when the number of parameters is large, then we have found that p is not in (0,1), and therefore it follows that the null is false. But this is an absurd conclusion when p=1.

Importantly, there is no statement in frequentist theory that says if the null is true, the p-value will be small, which would contradict the proof that it is uniformly distributed. And there is no theory which shows what values the p-value will take if the null is false. There is no Q which allows a proof by contradiction.

Think of it this way: you begin by declaring "The null is true!"; therefore, it becomes almost impossible to move from that declaration to concluding it is false.

There is no justification for use of p-values other than will or desire of the user.

from Climate Change Skeptic Blogs via hj on Inoreader http://bit.ly/2FxwSgY