New Paper! Everything Wrong With P-values Under One Roof

Here is a link to the PDF.

Briggs, William M., 2019. Everything Wrong with P-Values Under One Roof. In Beyond Traditional Probabilistic Methods in Economics, V Kreinovich, NN Thach, ND Trung, DV Thanh (eds.), pp 22–44. DOI 978-3-030-04200-4_2

Here is the Abstract:

P-values should not be used. They have no justification under frequentist theory; they are pure acts of will. Arguments justifying p-values are fallacious. P-values are not used to make all decisions about a model, where in some cases judgment overrules p-values. There is no justification for this in frequentist theory. Hypothesis testing cannot identify cause. Models based on p-values are almost never verified against reality. P-values are never unique. They cause models to appear more real than reality. They lead to magical or ritualized thinking. They do not allow the proper use of decision making. And when p-values seem to work, they do so because they serve a loose proxies for predictive probabilities, which are proposed as the replacement for p-values.

"Dude, that's harsh."

It is, indeed. Here are more words from the The Beginning of the End (i.e. the Introduction):

A book could be written summarizing all of the literature for and against p-values. Here I tackle only the major arguments against p-values. The first arguments are those showing they have no or sketchy justification, that their use reflects, as Neyman originally said, acts of will; that their use is even fallacious These will be less familiar to most readers. The second set of arguments assume the use of p-values, but show the severe limitations arising from that use. These are more common. Why p-values seem to work is also addressed. When they do seem to work it is because they are related to or proxies for the more natural predictive probabilities.

The emphasis in this paper is philosophical not mathematical. Technical mathematical arguments and formula, though valid and of interest, must always assume, tacitly or explicitly, a philosophy. If the philosophy on which a mathematical argument is based is shown to be in error, the "downstream" mathematical arguments supposing this philosophy are thus not independent evidence for or against p-values, and, whatever mathematical interest they may have, become irrelevant.

Trust me, you haven't seen many of these arguments against p-values.

"What's your favorite?"

Glad you asked, friend. It's the Infinity of Null Hypotheses, which is as damning a proof as can be. But it's not just a negative proof. It also constructively points the way toward the replacement (predictive methods) and it highlights the hidden notions of cause in statistics, which badly need our understanding. I'm working on a paper on that subject, to highlight the material in the award-eligible book Uncertainty, which all the better sort of people own, or will own.

Here's a quotation about that proof—but you have access to the full paper, too.

For every measure included in a model, an infinity of measures have been tacitly excluded, exclusions made without benefit of hypothesis tests. Suppose in a regression the observable is patient weight loss, and the measures the usual list of medical and demographic states. One potential measure is the preferred sock color of the third nearest neighbor from the patient's main residence. It is a silly measure because, we judge using outside common-sense knowledge, that this neighbor's sock color cannot have any causal bearing on our patient's weight loss. The point is not that nobody would add such a measure—nobody would—but that it could have been but was excluded without the use of hypothesis testing.

If we can exclude an infinity of hypotheses without hypothesis testing—based on causal decisions using probability notions, mostly—we can exclude the few more we put into a model without testing.

I've been getting emails from certain named persons in statistics who think they have found reasons to keep p-values (and mainly ignoring the arguments against in the paper). A popular thrust is to say smart people wouldn't use something dumb, like p-values. To which I respond smart people do lots of dumb things. And voting doesn't give truth.

I'm sympathetic to why it seems p-values seem to work—sometimes. When they do it's because they either mimic predictive methods, or they already agree with the causal knowledge we have in place. That's in the paper, too.

The moral of the story is: do not use p-values.

from Climate Change Skeptic Blogs via hj on Inoreader http://bit.ly/2C5MjZA