Ivory Tower Wonks Help Traders Make a Quick Buck
We live in an empirical age. P-values, R-squareds, and other statistical terms have become a universal language for describing the world. So perhaps it was inevitable that the contrarians and skeptics of our age would turn their wrathful eye on the excesses and missteps of statistical analysis. In recent years, there have been a number of high-profile academic papers demonstrating that science is chock full of bogus statistical results.
My personal favorite is John Ioannidis’ 2005 paper, “Why Most Published Research Findings are False.” Ioannidis shows that because researchers try many different statistical tests, a few are always likely to come out looking statistically significant, even if there’s really nothing there. This phenomenon goes by many names -- p-hacking, data mining, data snooping and data dredging. Statistician Andrew Gelman points out that researchers don’t even need to try a bunch of tests for this to be a problem -- all that’s required is that the researchers look at the data and decide in advance which tests have a better chance of yielding an eye-catching result. This bias can even be unconscious. In any case, the upshot is that any scientific literature that relies on statistical testing is likely to have a lot of false positives.
Finance is no exception. In 2014, Campbell Harvey, Yan Liu and Heqing Zhu wrote a paper called “…and the Cross-Section of Expected Returns.” They argue that because of data mining, most of the factors that researchers claim predict investment returns will eventually turn out to be spurious.
Now, a new paper by Kewei Hou, Chen Xue and Lu Zhang provides some evidence in support of Harvey et al.’s proposition. Hou et al. study anomalies instead of risk factors, but the two are pretty much the same thing. An “anomaly” means a market inefficiency -- a way to get extra reward without taking on proportionally more risk. A “factor” gives you extra reward, but only in return for accepting more risk. But since risk depends on people’s preferences, it’s unobservable -- therefore, whether something is an anomaly or a factor depends on how much you want to believe in efficient markets.
In any case, Hou et al. study hundreds of anomalies recorded in the finance research literature, and find that they don’t replicate -- when up-to-date market data is used, most of the anomalies disappear. Many of the ones that remain involve small-cap stocks where it’s pretty clear that markets shouldn’t be very efficient, because of the difficulty of short selling.
So it looks like academic finance joins psychology and other fields with serious replication crises. This news will be heartening to fans of the efficient markets hypothesis, which holds that anomalies are few or nonexistent.
But efficient markets fans should be cautious about embracing this new result. It might not mean exactly what it seems to. Finance, especially the study of market returns, is fundamentally different than other fields of science, for one big reason -- when humans are the object being studied, they react and change the rules of the game. When scientists observe physical laws, it’s a good bet that those laws won’t change tomorrow, and certainly not in response to the scientist’s experiment. But in finance, human beings are always looking for ways to beat the market, so what looks like a law of nature one day can vanish tomorrow.
What this means is that anomalies may be disappearing not just because they were false positives to begin with, but because traders are finding them and making them disappear. If an academic team finds a way to beat the market, it’s very possible that traders, who tend to have better funding and are far more numerous than professors, are already onto it. It’s also possible that traders scan academic papers and trade on the anomalies as soon as they read about them, thus making those anomalies vanish.
In fact, this is the conclusion of another recent paper by R. David McLean and Jeffrey Pontiff. These researchers find that after academics publish a paper claiming to find a new factor that predicts stock returns, those returns tend to disappear. But the factors tend to disappear only after the paper is published -- between the end of the data sample used to identify the factor and the time the paper goes to print, the factors mostly hold up.
This finding suggests that much or even most of the disappearance of anomalies is due not to p-hacking and data mining, but to trading itself. Markets are sort of efficient, but not quite -- if there’s free money to be had, traders will eventually discover it, but it takes them a long time and sometimes requires a helping hand from academia. McLean and Pontiff’s result paints a picture of a market where inefficiencies are constantly sprouting up and constantly being discovered and traded away -- not a smoothly running machine, but a constantly evolving ecosystem of predators and prey.
So when you see an academic paper claiming to have found a way to make easy money, don’t dismiss it as a product of data mining -- it might be a way to make a quick buck.
This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.
To contact the author of this story:
Noah Smith at firstname.lastname@example.org
To contact the editor responsible for this story:
James Greiff at email@example.com