The Probability Distribution of P-Values

On his facebook page, Nassim recently posted links to a new short technical paper on the probability distribution of p-values and a video commentary. He wrote:

I was able to pull out the exact meta-distribution of p-values (i.e. p-values as random variables).
The point is that the same phenomenon will produce p-values all over the map.  A true p-value of .12 will produce p-values <.05 more than half the time, so people may never replicate and get the same result.
One Hundred Years of P-Value Bullshit!

Here is the text of the paper, which was originally posted on his website, Fooled By Randomness.

Disclaimer: This website is not affiliated with Nassim Taleb. Click here and here for his official website(s).

One comment

  1. P value is a standardized measure like the thermometer. It doesn’t tell you straight away, whether the weather’s good, but it gives some indication. What’s important, is that the P values are calculated in a standard way. There’s very little room for improvisation. Replacing the P values by Bayesian analysis will almost inevitably lead to a catastrophe. This is for two reasons. 1) Economic incentives compromise the integrity of researchers, and consequently, they tweak the priors to get the desired posteriors. 2) Nobody cares about the convergence of the MCMC. Most Bayesian analyses are terribly inaccurate.

    If you, Sir, have a better alternative for the conventional P values, the world would be happy to hear it. All I’m saying is that Bayesian analysis is very unlikely to be that alternative.

Leave a Reply

Your email address will not be published. Required fields are marked *