Wall Street’s Big Banks Are Waging an All-Out Technological Arms Race
Sunlight bounced off Goldman Sachs Group Inc.’s glass-and-steel Manhattan headquarters on a warm August morning in 2013. Eyes locked on their screens, traders and engineers shifted in their seats as exchanges prepared to open.
Unbeknownst to anyone, the machines were about to revolt.
Bam. Bam. Bam. Dummy trade signals that were supposed to stay within the company’s electronic systems broke loose and slammed into computers at the New York Stock Exchange’s options markets. So many orders crashed through that by 8:44 a.m., safeguards within Goldman Sachs sprang into action, severing the connection between the company and the exchanges.
It took ages for anyone to notice the anomaly. At 9:01 a.m. an employee finally saw the blockage and lifted it. A river of mispriced orders surged through the restored connection. Minutes later, the volume triggered another stoppage. And another. And another.
By the time the trades were blocked for the last time, less than an hour after they began, Goldman Sachs executed orders to sell more than 1.5 million options contracts for $1. The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million.
The blunder, in one corner of the stock market, exposed a soft spot in the controls and technology at the company’s larger equities business. Despite being the dominant equities shop on Wall Street, its electronic operations had become almost an afterthought. Goldman Sachs had gotten to the top by catering to the types of companies that ruled financial markets for decades: long-short hedge funds and active asset managers. In the pre-crisis era, Goldman’s human traders made billions of dollars in profit and became the envy of their peers by using the company’s balance sheet to take risks.
But as one of the greatest bull markets in history took off, professional stockpickers struggled to generate adequate returns. Volatility vanished as central banks placated markets. Trillions of dollars moved from active strategies into index and exchange-traded funds, favoring electronic platforms, where commissions are a fraction of voice trading. Quantitative hedge funds such as Renaissance Technologies and Two Sigma Investments hoovered up assets while their old-school peers withered.
The 149-year-old investment bank wasn’t alone in being slow to anticipate the seismic shift in equities. But as excuses go, Goldman Sachs had a couple of good ones. Senior executives worried about cracks beginning to form in the market’s foundations. And for more than a decade, Wall Street’s bond traders enjoyed unrivaled prosperity, buoyed by the expansion of new markets for derivatives and a decline in interest rates that made it more lucrative to simply hold investments.
Few benefited more from this than Goldman Sachs, which became the most profitable company in Wall Street history under former commodities salesman Lloyd Blankfein, who may step down as chief executive officer as early as this year. Using house money to facilitate bond trades—or principal risk-taking—was far more lucrative than merely acting on clients’ behalf to execute stock orders. In 2009, Goldman traders reaped $33 billion in revenue, two-thirds of it from fixed income.
But constraints brought about by the financial crisis ended the leverage that had fueled the boom. Fixed-income traders felt the brunt of the changes, and in the years since, equities traders —especially those with a technology background—have enjoyed a renaissance. Their rise has touched off a battle for supremacy that’s come down to only three companies: Goldman Sachs, Morgan Stanley, and JPMorgan Chase. These rivals are now locked in a technological arms race to control a $58 billion-a-year industry. As they each jockey for an edge over the other, no one who trades on Wall Street is safe.
Around the same time as the options mishap in 2013, in another glass-and-steel office tower about 4 miles north of Goldman Sachs, Morgan Stanley traders were digging themselves out of a hole.
Clients had abandoned the company during the financial crisis as subprime losses pushed Morgan Stanley to the edge of failure. The bank had caught a case of Goldman envy and plowed into mortgage-bond bets at precisely the wrong time. To survive a share plunge driven by short sellers, then-CEO John Mack lashed out at some of the same clients who paid them lucrative fees. The bank was saved by a $9 billion investment from Mitsubishi UFJ Financial Group Inc., but in the years immediately after the crisis, it seemed a shadow of its former self. An equities executive at a rival U.S. bank puts it this way: “I thought we left them for dead in 2010. I remember telling people, ‘These guys are roadkill.’”
But the financial crisis had one gift for Morgan Stanley. Other banks that built a following with quants were damaged as well. Lehman Brothers Holdings Inc. went bankrupt. Three European banks—Credit Suisse, Deutsche Bank, and Barclays—didn’t adapt quickly enough to the new realities and were later forced to raise capital, sowing doubt among prime brokerage clients. So for hedge fund clients who wanted the latest trading technology, Morgan Stanley became the clear choice.
Under Ted Pick, who became co-head of the equities business in 2009, the company went about persuading clients to come home. The bank was back, hungry and open for business, flush with liquidity for hedge funds to place bets. It emerged with a sharper focus: The new CEO, James Gorman, made it clear that the equities division, along with the company’s wealth management business, was key to the bank’s strategic vision.
A paradox seen among some senior equities executives is that despite their high status and net worth, they often dress slightly shabbily. It might be intentional, part of the pitch: We don’t care what we look like. It’s all about you, valued client.
Pick, 49, fits this description. The day he meets with a reporter, he’s got on a pair of beat-up loafers and a well-worn suit. He could use a haircut and a couple more hours of sleep. The walls of his office are bare except for some grainy photos of his children. His only windows face the equities trading floor: Densely packed with employees and computers, the vast room feels 5 degrees warmer than comfortable. Instead of being sequestered in offices, his managing directors are scattered amid their charges, the better to stay close to clients and the thrum of markets.
Pick’s fervor for Morgan Stanley borders on the maniacal. Atop a mahogany cabinet—inherited from his predecessor, Vikram Pandit—are rows of manila folders filled with the minutiae of more than 30 quarters of equities results. Shortly after taking over the stock division, he divided the world into nine boxes—cash equities, derivatives, and prime brokerage across the Americas, Europe, and Asia—with the goal of improving in every segment. The full spectrum of offerings meant that whatever strategy or region a client needed in a given environment, Morgan Stanley was there. Pick keeps a close watch on all this data and, when the mood strikes, pulls out a folder to recite figures from a long-ago period.
The company made investments that let clients send orders with the least possible delay by moving servers closer to exchanges and using wires that shaved microseconds off the process. It updated its low-latency trading system Speedway and, in 2012, embarked on Project Velocity, which anticipated the rising needs of quants and other institutions that were embracing algorithmic trading. The improvements strengthened what was already a leading electronic platform; almost a decade earlier, Morgan Stanley had been the first major broker to create an electronic swaps system, the preferred mode for quants to trade in equities. Quants favor the system because it gives them leveraged exposure to stocks without owning them and having to pay taxes on dividends.
With the upgraded electronic system and revamped prime brokerage, Morgan Stanley enveloped clients in a cocoon of lightning-fast connectivity to markets everywhere and liquidity to short stocks. Its systems were robust enough for the biggest quants, and it could lease pipes and algorithms to smaller hedge funds that couldn’t afford the technological investments the company made.
It all paid off. In 2014 the company grabbed the crown from Goldman Sachs, exceeding its rival in equities revenue for the first time in almost a decade. Under Pick, Morgan Stanley had gone from the recovery ward to the summit of the world’s most iconic market in four short years. And yet reaching the pinnacle hasn’t dulled their drive: Pick says they’re as hungry now as they were in 2010.
In a sense, Morgan Stanley and Pick are reaping gains from a prescient bet made two decades earlier. When Pandit took over equity trading at Morgan Stanley in 1994, he noticed that the cost of computing power was collapsing. That, combined with abundant data on public exchanges that would enable automated trading, convinced Pandit of the need to prepare for a post-human equities market.
“We realized that the human touch was interesting but actually a hindrance to what it took to really trade these markets correctly,” Pandit, who served as CEO of Citigroup Inc. for five years, says in the Midtown office of his investment company, Orogen Group. “The only thing you could do is figure out how you automated all the human aspects of trading, understanding what drove stock prices, and then used those algorithms to make markets.” So Pandit gathered math and science experts to open the Equity Trading Lab, or ETL, whose initial mission was to automate the trading floor.
It helped that at the time Morgan Stanley was already known as a hotbed for innovation; in the 1980s, the bank instituted an early quant strategy called pairs trading. Employees from that era included a computer scientist named David Shaw, who later founded a legendary quant hedge fund. In the early 1990s, Peter Muller founded an in-house quant fund called PDT Partners that minted money for the bank until it was spun out after the financial crisis.
Heeding the needs of its early quant clients—several of whom were former employees—the ETL team built version 1.0 of its electronic-trading operation. The guiding principle was to reduce friction for hedge funds: The team later created the Trading and Position System, or TAPS, which helped prime brokerage clients to receive automated reports. Because of these innovations, quant funds such as Renaissance Technologies gravitated toward Morgan Stanley.
Morgan Stanley hugged its quant clients close. Instead of merely offering to execute trades, prime brokerage provides all-important leverage and custody of assets. “Prime brokerage is really the lifeblood,” says ETL co-founder Michael Botlo. “This is the oxygen. This is what you’re immersed in. If all of a sudden prime brokerage becomes terrible, then you’re toast. If you can’t short anymore, you’re dead. If you can’t access your swaps, you’re dead.”
As much as Pick talks up the bank’s people and culture, the trading algorithms—an early form of artificial intelligence—are its secret sauce, according to Pandit. “Imagine the accumulated learning of having done it for the last 20 years,” he says. The algorithms, he adds, are rules that essentially gather “all the things you did wrong that informs what you should do today that’s right. They’re so far ahead of the game, it’s going to give them an edge for a while.”
Goldman’s dominance among Wall Street’s active traders all those years ago didn’t happen by accident. In the 1950s and ’60s, the legendary trader Gus Levy helped establish the white-shoe investment bank in the rough-and-tumble world of trading. Known as “the Octopus,” because he wanted a tentacle gripping every transaction, he flew into a rage whenever a competitor won a coveted block trade. These were deals in which Goldman used its balance sheet to purchase chunks of stock to profitably peddle off to clients. He was also a pioneer in the practice of risk arbitrage, or using the company’s money to bet on takeover targets.
Levy, who died in 1976, instilled in the company an intensity and risk appetite that would live long after him. A generation of traders followed in his footsteps and burnished Goldman’s reputation in markets. Robert Rubin, who later became President Clinton’s secretary of the Treasury, got his start in risk arb under Levy and later ran the equities desk. Rubin, in turn, helped train Richard Perry, Eric Mindich, and Daniel Och, all of whom would go on to found successful hedge funds. Goldman alumni typically stay close to their old colleagues, and this group was no different, making Goldman the top choice for investors in search of market intelligence, smart sales coverage, or financing.
In 1998 the tectonic plates of finance shifted. That year the SEC allowed a new breed of alternative trading platforms to compete with traditional stock exchanges. In one stroke, regulators fostered the rise of electronic-trading networks and weakened the grip of human traders.
Perhaps recognizing that the low-margin, unfashionable business of electronic market making wasn’t a part of its heritage in the way it was at Morgan Stanley, Goldman just went out and bought the capability.
In 1999, CEO Hank Paulson, who later served as Treasury secretary during the financial crisis, bought a stake in Archipelago Holdings LLC, a relatively new trading platform beginning to gobble up market share. Six months later he spent $500 million for Hull Group Inc., a Chicago-based options broker and early proponent in a new strategy using algorithms. In 2000 he spent billions more for Spear, Leeds & Kellogg LP, a big employer of human market makers on the floor of the New York Stock Exchange. The Spear purchase included a stake in a small electronic-trading system called REDIBook.
Then, in 2001, the SEC took another whack at profit margins and further tipped the scales in favor of automation when it ordered all exchange trading to quote prices in pennies rather than fractions. Six years later, regulators demanded that trades take place on whichever venue offered the best price at a given time, sparking a proliferation of venues, which now number more than 80.
For a few years after the Spear purchase, Goldman was a top player in electronic trading as it steadily added capabilities. But the ever-shifting landscape would prove to be fertile ground for a new market inhabitant that would erode Goldman’s standing: the high-frequency trader.
In the following years, a battle would rage within Goldman about what to make of high-speed traders and their cousins, the quants. (Quants deploy strategies that are different from those of HFT companies, but they have similar technology requirements.) It often pitted men who’d made fortunes in personalized, or “high-touch,” trading against technologists who arrived at Goldman through its acquisitions.
Senior equities executives including Brian Levine were convinced that Goldman should stick to its historic strength of wielding risk capital for clients, according to people with knowledge of the situation. They questioned whether the returns generated by servicing quants, who use leverage to amplify returns from thousands of bets on tiny price movements, were adequate. Levine was also troubled by structural changes that he felt were weakening markets. Glitches, such as Goldman’s options mistake, and numerous others in the cash equities market were symbols of the market’s vulnerability. (By 2013, Levine had seen enough that he agreed to sit down with Michael Lewis for his 2014 book, Flash Boys: A Wall Street Revolt, and the company later wrote a Wall Street Journal op-ed calling for changes to the market.) It was because of this confluence of concerns that Goldman failed to anticipate how much the quant client base would grow. That decision looks unwise in hindsight, but Goldman had generated $13 billion in stock revenue in 2008, a high-water mark for the industry. By some accounts, they were fat and happy. “I just don’t think we were as concerned about electronic trading,” Levine recalls in December, his voice hoarse after holiday parties.
Part of Goldman’s weakness in automated trading was by design. Following the Spear purchase in 2000, Goldman had kept the electronic-trading unit in New Jersey, separate from the rest of the operations. Goldman’s penchant for proprietary trading was well-known, so clients fearful that their trades would be visible asked the company to keep the technology at arm’s length. But the separation prevented Goldman from developing the ability to do principal trades in the electronic unit, meaning they couldn’t offer what quants needed: equity swaps that bundle financing and execution costs together.
The company finally merged its broker-dealers in 2013—almost a decade after Morgan Stanley pioneered electronic swaps trading. Goldman had dithered too long.
In February 2013 it was announced that electronic-trading head Greg Tusar was leaving for a high-frequency trading company. Ronnie Morgan, a senior equities executive with years of serving clients in a high-touch capacity, was put in charge of “low-touch,” or electronic, trading. Decisions such as this reflected a cultural bias for risk takers over technologists, says Michael Dubno, a retired Goldman Sachs partner and chief technology officer. “They lagged for a long time simply because they really didn’t put their front-office tech leaders in ownership positions,” he says. “They weren’t comfortable doing that.”
It didn’t help that some in the company were devoted to doing as much as possible within its risk management platform, known as Securities DataBase, or SecDB. Years earlier, executives had decided that trading algorithms should reside inside the system. And for good reason: The strength of it was that every trading desk fed into the all-seeing program in real time, giving Goldman executives companywide views of risk. The system’s reputation grew during the financial crisis when it helped Goldman outperform rivals. So for some inside the bank, building a system outside of it seemed like sacrilege. But the platform wasn’t engineered to handle the data rates of hundredths or thousandths of a second needed to compete in the low-latency game.
Goldman continued to struggle with its place in the rapidly changing market. As Morgan Stanley’s rise became apparent, Morgan and Levine invited employees of Goldman’s rival in for job interviews, according to Lewis’s book. In the process, one of the things the Goldman execs learned was that Morgan Stanley was minting money with its high-speed offerings.
It got worse. Traditional Goldman clients such as multistrategy hedge funds started moving toward computer-driven strategies. Asset managers such as Fidelity Investments became even more sensitive about the quality of their trade execution with Wall Street firms. And regulators didn’t react to Lewis’s book and other efforts to sway public opinion. It took years for Goldman to realize it had been wrong on quants.
When it finally awoke—in 2014—it acted swiftly. The first step was hiring Raj Mahajan as a partner, a dramatic move in and of itself. Mahajan had begun his career in 1996 at Goldman’s commodities business and later founded a tech company with Marty Chavez, now the firm’s chief financial officer. Mahajan subsequently became CEO of high-speed company Allston Trading LLC, known as a member of a small cabal of HFT executives who knew their way around the increasingly complex corners of U.S. equity markets.
It’s clear that Mahajan, who finds a few minutes to talk between client meetings, is obsessed with speed. As he attempts to distill the complexities of market structure into simple analogies, over and again he cites milliseconds and microseconds, unfathomably tiny units of time. Once at Goldman, he went about making some of the very decisions the company had resisted—such as creating a parallel system outside SecDB that connects to it only when necessary.
Mahajan determined that SecDB’s data constraints meant it couldn’t capture the number of orders to buy or sell shares above or below the price on offer, what’s known as market depth. Without that, he says, “it was hard for us to be able to make a quick pricing decision on stocks we were trading algorithmically.”
He first tackled the plumbing—the wiring that connects dozens of systems, including those that keep track of inventory, compliance, and post-trade processing internally as well as the outside world of exchanges or broker-dealers. Each trade must be scrutinized in accordance with a lengthy checklist, so Mahajan had programmers write new code to speed up the process. “It’s a computer science and physics problem in how to perform a certain amount of work in a shorter amount of time,” he says.
As the number of exchanges and other trading systems has multiplied, it’s become increasingly important to discover which one offers the best price at any given moment. Mahajan had to make his systems fast enough to consume market data every 20 milliseconds to 50 milliseconds—less than half the time it takes for a human eye to blink—and make almost-instantaneous decisions about where to route orders.
As a last step in his makeover, Mahajan ordered a wholesale rewrite of the algorithms that decide how to break up an order. Clients now have a choice of using those instructions or Goldman’s plumbing to transmit orders to exchanges. “The guiding principle,” he says, “is that we wanted to be the No. 1 electronic intermediary in the business regardless of whether you are a large asset manager or a quantitative hedge fund.”
Mahajan says Goldman has created a system that verifies a trade, locates the best price, and moves quickly to process the order before someone else snatches it. It has a success rate of better than 99 percent. The bank is now one of the top three providers of fast access to European markets, Blankfein said in a March letter to shareholders touting the company’s efforts. The platform also works for futures, commodities, and Treasuries.
Goldman’s aspirations are modest for a company that teems with ambition. Knowing it’s difficult to get big-name quants to switch all their business at once, the bank aims to attract smaller shops or those active investors who are just starting quant strategies. The company reevaluates the rationale behind the project every six months. So far, Mahajan says, “everyone’s heard the arguments, it’s been debated, and it’s been unanimous: Keep your foot on the gas.”
While Goldman Sachs and Morgan Stanley tussled for supremacy, another threat emerged: Jamie Dimon. His bank, JPMorgan Chase & Co., emerged from the financial crisis as the most complete of banking franchises, helped by a pair of takeovers and a reputation for being a safe harbor amid the storm. As the fallout receded, it ended up having leading businesses in almost every major category of finance, including retail and commercial banking, asset management, and Wall Street advisory services. What’s more, it was sitting on almost $1 trillion in deposits to fund it all, more than its two rivals combined.
The glaring exception was in equities, the only major category in which it didn’t have a top-three position. Jason Sippel, a 16-year veteran of the company, remembers how hard it was to coax clients to trade with the bank. The belittling was relentless. “Clients used to beat us up when we saw them,” he says. “We didn’t have a strong risk offering for bigger trades, we were far behind in electronic trading, and we didn’t have a low-latency offering.”
Dimon, the aggressive leader who helped pioneer the financial supermarket with ex-mentor Sanford “Sandy” Weill, wouldn’t accept a subpar rank. In 2010 the bank hired Frank Troise, an electronic-trading specialist from Lehman Brothers and Barclays Plc and told him to build a world-class platform. At first, Troise and his new hires were a tribe unto themselves within JPMorgan. To protect the nascent electronic business from the voice traders whose livelihoods the unit threatened, Troise reported directly to Carlos Hernandez, the global equities chief at the time.
By the time Troise left JPMorgan in 2015 to run broker Investment Technology Group Inc., JPMorgan had tripled its market share in equities electronic trading. The company began pumping more capital into its prime brokerage for clients and ramped up investment in quant services, creating a devoted risk management team.
Sippel, who was head of prime services in 2015 and became co-head of equities in 2016 with Mark Leung, now had a different story to tell. Quants who needed the fastest access and ample liquidity were signing on. The company could market its growing inventory of stocks to other potential users, allowing them to match more buyers and sellers, as well as longs and shorts, internally, shaving expenses for everyone involved. Pricing their products competitively, they also persuaded their existing customers—JPMorgan has the world’s biggest fixed-income business—to use the bank’s equities desks. In this arena, scale is all that matters; those without it will find themselves unable to invest in the technology to stay relevant.
Despite being years late, JPMorgan grew by leaps and bounds. Sippel is blunt about how they got there: He calls it the Morgan Stanley playbook. Anchored by electronic trading and prime brokerage, the company has filled every major segment in every region of the world, a kind of convergent evolution with Pick’s nine boxes. That helped JPMorgan pull clients from weakened European investment banks. While the overall pool of trading fees shrank, JPMorgan managed to increase its market share. It had 10.3 percent of the global market in 2017, up from 5 percent in 2006.
Part of that story is the triumph of electronic trading. Even though per-trade commissions are a fraction of those from voice trading, electronic trading at JPMorgan has gone from a rounding error five years ago to pulling even with its higher-service counterpart. “The electronic side has won,” Sippel says. “And that’s not something we forced on them. Clients are choosing it because it’s more efficient, cheaper, easier.”
But now the game is changing again. The twin forces that have always shaped the markets—technology and regulation—are about to wreak havoc once more.
Investment banks are starting to unleash a new generation of learning machines on the markets to customize, hedge, and execute trades. It’s a step toward the post-human vision of markets that Pandit had at Morgan Stanley in the 1990s. Across the equities and fixed-income world, apart from a dwindling pool of human traders working on bespoke deals and the human minders of the machines, algorithms will be connecting sellers and buyers.
Then there are the regulations, the never-ending cascade of rules flowing from the wreckage of the financial crisis. The latest ones emanate from Europe in the form of the Markets in Financial Instruments Directive II (MiFID II), which requires unbundling trading commissions from research and increasing transparency—and is certain to accelerate the tilt toward machines.
And yet, for all the ways in which finance is becoming a place where machines transact with other machines, the race for trading riches will ultimately be won or lost by people such as Blankfein, Gorman, and Dimon, men driven to keep the throne—or claim it at last.
Son and Campbell cover finance at Bloomberg News in New York.
— With assistance by Sam Mamudi, and Polina Noskova