A Mini-MOOC Tutorial: The Randomness of Correlation and its Hacking by Big Dataists

Nassim says:

This tutorial presents the intuitions of the randomness of sample correlation (spurious correlation) and the methodologies in derivations.
Some later sections are somewhat technical as rederived an old equation with more precise functions (in order to apply to fat tails) and showed the distribution of the maximum of d variables with n points per variable.
This paves the way to the real scientific work on random matric theory under fat tails and failure of Marchenko-Pastur.

Foreword to Ed Thorp’s Memoirs (A Man for All Markets)

Nassim has written the foreword to Ed Thorp’s memoir A Man for All Markets:

Abraham de Moivre

 

Ed Thorp memoirs read like a thriller –mixing wearable computers that would have made James Bond proud, shady characters, great scientists and poisoning attempts (in addition to the sabotage of Ed’s car so he would have an “accident” in the desert). The book will reveal a thorough, rigorous, methodical person in search of life, knowledge, financial security, and, not least, fun. Thorp is also known to a generous man, intellectually speaking, eager to share his discoveries with random strangers (in print but also in person) –something you would hope to find in a scientist but usually don’t. But he is humble –he would qualify as the only humble trader on planet Earth –so, unless the readers can reinterpret what’s between the lines, they won’t notice that his contribution is vastly more momentous than he reveals. Why?

Because of its simplicity. Its sheer simplicity.


For it is the straightforward character of his contributions and insights that made it both invisible in academia and useful for practitioners. My purpose here is not to explain or summarize the book. Thorp –not surprisingly –writes in a direct, clear, and engaging way; I am here, as a trader and a practitioner of mathematical finance, to show its importance and put it in context for my community of real world scientists-traders and risk takers in general.

The context is as follows. Ed Thorp is the first modern mathematician who successfully used quantitative methods for risk taking –and most certainly the first mathematician who met financial success doing it. Since then there have been a cohort, such as the Stony Brook whiz kids –but Thorp is their dean. His main and most colorful predecessor, Girolamo (sometimes Geronimo) Cardano, a sixteenth Century polymath and mathematician who –sort of –wrote the first version of Beat the Dealer, was a compulsive gambler. To put it mildly, he was unsuccessful at it –not least because addicts are bad risk takers, and, to be convinced, just take a look at the magnificence of Monte Carlo, Las Vegas, and Biarritz, places financed by their compulsion. Cardano’s book, Liber de ludo aleae (“Book on Games of Chance”) was instrumental in the later development of probability, but, unlike Thorp, was less of an inspiration for gamblers and more of one for mathematicians. Another mathematician, a French Protestant refugee in London, Abraham de Moivre, a frequenter of gambling joints and author of The doctrine of chances: or, a method for calculating the probabilities of events in play (1718) could hardly make both ends meet. One can count another half a dozen mathematician-gamblers, in a line that include the great Fermat, Huygens — who were either indifferent to the bottom line or (for those who weren’t) not particularly good at it. Before Ed Thorpe, mathematicians of gambling had their love of chance largely unrequited.

Thorp’s method is as follows. He cuts to the chase in identifying a clear edge (that is something that in the long run puts the odds in his favor). The edge has to be obvious and uncomplicated. For instance, calculating the roulette momentum with the first wearable computer (with no less of a co-conspirator than the great Claude Shannon, father of information theory), he estimated a typical edge of roughly 40% per bet. But that part is easy, very easy. It is capturing the edge, converting it into dollars in the bank, restaurant meals, interesting cruises, and Christmas gifts to friends and family; that’s the hard part. It is the dosage of your betting –not too little, not too much –that in the end matters. For that, Ed did great work on his own, before the theoretical refinement that came from a third member of the Information Trio: John Kelly, of the Kelly criterion, which we discuss today because of Ed Thorp made it operational.

A bit more about the simplicity before we discuss the dosing. For an academic judged by his colleagues, rather than the bank manager of his local branch (or his tax accountant), a mountain giving birth to a mouse, after huge labor, is not a very good thing. They prefer the mouse to give birth to a mountain; it is the perception of sophistication that matters. The more complicated, the better; the simple doesn’t get you citations, H-values or some such metric du jour that brings the respect of the university administrators as they can understand that stuff but not the substance of the real work. The only academics who escape the burden of complication-for-complication’s sake are the great mathematicians and physicists (and from what I hear this is becoming harder and harder in today’s funding and ranking environment).

Ed was initially an academic, but he favored learning by doing, with his skin in the game. When you reincarnate as practitioner, you want the mountain to give birth to the simplest possible strategy, and one that has the smallest amount of side effects, the minimum possible hidden complications. The genius of Ed is demonstrated in the way he came up with very simple rules in Black Jack. Instead of engaging in complicated combinatorics and memory–challenging card counting (something that requires one to be a savant), he crystallizes all his sophisticated research into simple rules. Go to a Black Jack table. Keep a tally. Start with zero. Add one for some strong cards, minus ones for weak ones, and nothing for others. It is easy to just increment up and down mentally, bet larger when the number is high, smaller when it is low, and such a strategy is immediately applicable by anyone with the ability to tie his shoes or find a casino on a map. Even while using wearable computers at the roulette table, the detection of edge was simple, so simple that one can get it while standing on a balance ball in the gym; the fanciness resides in the implementation and the wiring.

As a side plot, Ed discovered what is known today as the Black Scholes option formula, before Black and Scholes (and it is a sign of economics public relations that the formula doesn’t bears his name –I’ve called it Bachelier-Thorp) . His derivation was too simple –nobody at the time realized it could be potent.

Now the money management –something central for those who learn from being exposed to their own profits and losses. Having an “edge” and surviving are two different things: the first requires the second. As Warren Buffet said: “in order to succeed you must first survive”. You need to avoid ruin. At all costs.

And there is a dialectic between you and your P/L: you start betting small (a proportion of initial capital) and your risk control –the dosage — also controls your discovery of the edge. It is like trial and error, by which you revise both your risk appetite and your assessment of your odds one step at a time.

Finance academics, as it has been recently shown by Ole Peters and Murray Gell-Mann, did not get the point that avoiding ruin, as a general principle, makes your gambling and investment strategy extremely different from the one that is proposed by their literature. As we saw they were paid by administrators via colleagues to make life complicated, not simpler. They invented something useless called utility theory (tens of thousands of papers are still waiting for a real reader). And they invented the idea that one could get to know the collective behavior of future prices in infinite detail –things such as correlation, identified today, would never change in the future. More technically, to implement the portfolio construction suggested by modern financial theory, one needs to know the entire joint probability distribution of all assets for the entire future, plus the exact utility function for wealth at all future times. And without errors! (I have shown that estimation errors make the system explode.) We are lucky if we can know what we will eat for lunch tomorrow –how can we figure out the dynamics until the end of time?

Kelly-Thorp method, requires no joint distribution or utility function. In practice one needs the ratio of expected profit to worst-case return — dynamically adjusted (that is, one gamble at a time) to avoid ruin. That’s all.

Thorp and Kelly’s ideas were rejected by economists — in spite of their practical appeal — because of their love of general theories for all asset prices, dynamics of the world, etc. The famous patriarch of modern economics, Paul Samuelson, was supposedly on a vendetta against Thorp. Not a single one of the works of these economists will eventually survive: your strategy to survive isn’t the same as ability to impress colleagues.

So the world today is divided into two groups. The first method is that of the economists who tend to blow up routinely or get rich collecting fees for managing money, not from direct speculation. Consider that Long Term Capital Management that had the crème de la crème of financial economists, blew up spectacularly in 1998, losing a multiple of what they thought their worst case scenario was.

The second method, that of the information theorists as pioneered by Ed, is practiced by traders and scientists-traders. Every surviving speculator uses explicitly or implicitly the second method (evidence: Ray Dalio, Paul Tudor Jones, Renaissance Technologies, even Goldman Sachs!) I said every because, as Peters and Gell-Mann have shown, those who don’t will eventually go bust.

So say you inherit $82,000 from uncle Morrie: now you know that there exists a strategy that will allow you to double the inheritance without ever going through bankruptcy.


Some additional wisdom I personally learned from Thorp. Many successful speculators, after their first break in life, get involved in large scale structures, with multiple offices, morning meeting, coffee, corporate intrigues, building more wealth while losing control of their lives. Not Ed. After the separation from his partners and the closing of his firm (for reasons that have nothing to do with him), he did not start a new mega-fund. He limited his involvement in managing other people’s money. Most other people do reintegrate in the comfort of firms and leverage their reputation by raising monstrous amounts of outside money in order to collect large fees. But such a restraint requires some intuition, some self knowledge. It is vastly less stressful to be independent –and one is never independent when involved in a large structure with powerful clients. It is hard enough to deal with the intricacies of probabilities, you need to avoid the vagaries of exposure to human moods. True success is exiting some rat race to modulate one’s activities for his peace of mind. Thorp certainly learned a lesson: the most stressful job he ever had was running the math department of the University of California Irvine. You can detect that the man is in control of his life. This explains why he looked younger on the second time I saw him, in 2016, than he did the first time, in 2005.

Read the original post on Medium.