Why Machine Learning Models Often Fail to Learn: QuickTake Q&A

Hedge funds have been in the doldrums and face mounting pressure to justify their fees. Will artificial intelligence come to the rescue? A growing number of hedge funds are putting money behind the idea that a branch of AI called machine learning could provide a way to get back on top. The problems? It’s hard, expensive and prone to failure.

1. What’s machine learning?

A software program that searches for patterns in more data than even the most sleep-deprived junior analyst could examine, and then tests its hypotheses against even more data. What can satellite shots of mall parking lots tell you when combined with in-store sales data? Does a default premium of A to B and a yield curve slope of C to D have an E percent chance of boosting a stock price of F percent or above? Put me in, coach, the algorithm says, I’ll figure it out.

2. Is everybody trying this?

Lots of funds, big and small, are testing the waters. Fifty-eight percent of managers in one recent survey said machine learning will have a medium-to-large impact on the industry. Hedge fund giant Bridgewater Associates as well as Highbridge Capital Management and Simplex Asset Management in Japan are among firms developing machine learning or investing in it.

3. So, how are the machines doing?

So far, not so well. Finding patterns isn’t that hard; finding ones that work reliably in the real world is. One quantitative analyst, or quant, estimates the failure rate in live tests is about 90 percent. Man AHL, a quant unit of Man Group Plc, needed three years of work to gain enough confidence in a machine-learning strategy to devote client money to it.

4. What’s the problem?

There are several. If you let the programs roam too freely through the world of data, they can find meaningless patterns, such as U.S. GDP and the S&P 500 Index tending to gain or fall in lockstep with the homicide rate in England. But when quants complicate their models, adding too many parameters to get the results they seek -- a problem called over-fitting -- they can also flop.

5. When is failure clear?

Many algorithms wash out during so-called backtests, when their predictions based on historical data can’t be replicated using a new data set. And some that pass such tests run afoul of market realities. Tucker Balch of Lucena Research came up with an algorithm whose backtesting suggested impressive risk-adjusted returns. But the formula was focusing on thinly traded equities, and the act of buying in the real market caused prices to rise. “Ignore your market impact at your own peril,” Balch said. And then there’s the unexpected. Few algorithmic trading strategies developed so far would cope well with a Brexit vote or terror attack.

6. So why are hedge funds pouring money into machine learning?

Machine-learning has yet to deliver Earth-shaking returns. The Man AHL Dimension fund, which includes a machine-learning strategy, has returned an annualized 7.1 percent in three years through September, compared with a 3.2 percent gain for the average hedge fund. The Eurekahedge AI Hedge Fund Index, which tracks 12 pools that use AI as part of their core strategies, has returned an average of 9.3 percent from 2011 through 2015, and is up 13.8 percent this year through September. That compares with an average gain of 10.7 percent over the five-year span, and a 6 percent rise this year, for the S&P 500.

The Reference Shelf

  • A story on whether machine learning is more hype than revolution.
  • QuickTake explainers on artificial intelligence and computerized high-speed trading.
  • A story on how hedge funds are embracing machine learning.
  • Apple hired an artificial intelligence expert to compete against Google, Microsoft and Amazon in machine learning.
  • Bloomberg View columnist Noah Smith sees machine learning disrupting white-collar work, in a good way.
Before it's here, it's on the Bloomberg Terminal.