Investment strategies that use computer models to decide when to buy and sell securities based on historical market trends are usually unsupported scientifically because of back-testing flaws, according to the 126-year-old American Mathematical Society.
“We are not implying that those technical analysts, quantitative researchers or fund managers are ‘snake oil salesmen,’” David H. Bailey, a research fellow at the University of California, Davis, and three co-authors said in a paper in the May issue of the society’s magazine Notices. “Hedge-fund managers are often unaware that most back-tests presented to them by researchers and analysts may be useless.”
Strategies that use computer models to predict future market moves are often based on selective historical data, the authors said in the paper, “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance.” The results lead to “the proliferation of investment products that are misleadingly marketed as mathematically sound,” according to the researchers.
Back-testing applies a trading strategy to historical results in an effort to determine how well it may fare in the future. When testing an investment model, a researcher or money manager may refine and eliminate certain data, a process known as overfitting, to maximize the strategy’s performance.
“We strongly suspect that such back-test overfitting is a large part of the reason why so many algorithmic or systematic hedge funds do not live up to the elevated expectations generated by their managers,” the authors said.
Profits from computer-based investment strategies have lagged behind U.S. equities in recent years, as money managers said central bank intervention in global markets skewed the price trends their models typically follow.
Managed-futures funds lost an annualized 1 percent in the five years through March 31, according to data compiled by Bloomberg, while quantitative funds gained an average of 7.5 percent a year, according to data compiled by Chicago-based Hedge Fund Research Inc. The Standard & Poor’s 500 Index averaged total returns of 21 percent a year in the same period.
Managed-futures funds, sometimes known as commodity-trading advisers, use computer models to follow trends in futures markets such as commodities. Quantitative strategies employ more complex mathematical models to trade across all assets.
The authors said an analogous situation occurs in medical research when drugs are tested on thousands of patients yet only the best outcomes are publicized.
“Such behavior is unscientific -- not to mention dangerous and expensive,” they said.
Investors should be protected by a new review process, Bailey said in a telephone interview yesterday.
“An independent body needs to be created to audit and certify financial products that are backed by mathematical models,” said Bailey, who has received awards from the IEEE Computer Society and the Mathematical Association of America, according to his biography.
The American Mathematical Society is based in Providence, Rhode Island, and its members include almost 30,000 people and 580 institutions. Notices is the world’s most widely read magazine aimed at professional mathematicians, according to its website. “Pseudo-Mathematics” is one of the journal’s first articles on the subject of mathematical finance.
Popular terms used in finance such as Fibonacci ratio, Elliott wave and stochastic oscillator “evoke precise mathematical concepts” despite being used in “scientifically unsound” ways, the authors said.
The authors said they hope to call overdue attention to the deficiencies of historical simulation.
“As early as the 18th century, physicists exposed the nonsense of astrologers,” they wrote. “Yet mathematicians in the 21st century have remained disappointingly silent with the regards to those in the investment community who, knowingly or not, misuse mathematical techniques such as probability theory, statistics and stochastic calculus. Our silence is consent, making us accomplices in these abuses.”