Randomness of Correlation & Its Hacking by Big Dataists


This tutorial presents the intuitions of the randomness of sample correlation (spurious correlation) and the methodologies in derivations. Some later sections are somewhat technical as Nassim rederived an old equation with more precise functions (in order to apply to fat tails) and showed the distribution of the maximum of d variables with n points per variable.
This paves the way to the real scientific work on random matric theory under fat tails and the failure of Marchenko-Pastur.

One comment

  1. I’d be interested then in a *proper* analysis of data concerning peace time and war time frequency throughout history. My intuition would be while intensity of each single war has inexorably increased (due to technological advance in weaponry), the duration of peace in relative terms has declined throughout history, spurious signal accounted.

    But this is also almost logical, not merely intuition, since if war times in relative proportion to peace (suitably gauged by numerical severity/length of battle) has not declined, then the obvious and inexorable increase in weaponry power will (would have) wipe (wiped) out humanity in short order long ago. Example: 100 tribes fighting with bow and arrow for 30 days each year somewhere on the globe is a local disaster, relatively speaking, but just two people fighting with nukes for one day can wipe out whole cities. 100 tribes each fighting somewhere with napalm for 30 days each year will probably cause mass extinctions. 100 tribes each fighting somewhere with nukes for one day one year year will probably wipe out human civilization.

    So it must be the case that in relative terms (relative to weapon power) human civilization has become more peaceful. Relative to bow & arrow power this is not necessarily the case, since people with bows & arrows might still be fighting as severe as in the past. However, this is where my intuition suspects this is not the case. If the aim is actual war, to win a battle, you do not use bow & arrow these days, unless that is the only weapon you have access to, but how likely is that on the world scale?

    I suspect there is something good in Pinker’s work, or his data, even though he seems to be a major douchebag or pervert (hard to avoid ad hominen in light of recent current affairs! I guess innocent until proven guilty). He just perhaps hasn’t uncovered the valid results that do legitimately argue for our “better angels.”

Leave a Reply

Your email address will not be published. Required fields are marked *