Server Crash Spurs 3-Hour Nasdaq Halt as Data Link Lost

The interruption that froze half the U.S. stock market last week began with a routine notice.

“NYSE Arca is currently experiencing an issue processing customer messages on routed orders in Tape C symbol range ‘TACT’ through ‘ZYY.C,’” read an alert for traders received by Bloomberg News at 10:17 a.m. on Aug. 22. “Will advise.”

The brevity of the message belied its significance. Computers at the world’s biggest exchange operators were having difficulty communicating. Within two hours, trading stopped in more than 2,000 U.S. stocks. The three-hour shutdown was the latest in a series of failures to disrupt increasingly complicated markets, prompting the Securities and Exchange Commission to push for rules requiring executives to improve the reliability of their technology.

“Every time we have a glitch like this, it just knocks a little bit more investor confidence off the table,” said Seth Merrin, chief executive officer of Liquidnet Holdings Inc., a private trading platform that competes with the public exchanges, in an interview on Bloomberg Television. “This kind of scenario should have been thought of and practiced between all of the different participants.”

Market Integrity

The disruption underscored how quickly the integrity of the U.S. stock market, which has a value of almost $20 trillion, can be threatened as orders to buy and sell shares are matched on more than 50 exchanges and alternative electronic venues. While the outage highlighted what can arise when exchange computers fail, past malfunctions have shown how faulty use of a trading algorithm by a single brokerage or investment firm can roil the entire system.

A fault today halted transactions on Deutsche Boerse AG’s Eurex unit, Europe’s largest derivatives market, for about an hour through 9:20 a.m. Frankfurt time.

One result of last week’s shutdown may be the development of a backup data feed to compile and disseminate price quotes, according to Robert Greifeld, the chief executive officer of Nasdaq OMX Group Inc. SEC Chairman Mary Jo White, a former prosecutor, responded to the failure by vowing to finish a rule proposed in March to require exchanges to test the reliability of their technology.

“We have to improve our defensive-driving ability,” Greifeld said on Bloomberg Television Aug. 23. “We have to always be thinking, what can go wrong in the external environment, how can we get body blows that don’t impact us.”

Aug. 22 started off without signs of trouble in the U.S. equity market, where an average of 6.2 billion shares traded each session this year.

Quiet Start

Benchmark indexes opened higher, with the Standard & Poor’s 500 Index rebounding from a six-week low, as reports on global manufacturing and the American labor market eased concern about the Federal Reserve’s plans to reduce its $85 billion in monthly bond purchases used to help stimulate the economy.

After the first warning by Arca, owned by New York Stock Exchange-operator NYSE Euronext, e-mails from exchanges on system abnormalities started to pile up in traders’ inboxes. Three minutes after the initial communication, Lenexa, Kansas-based Bats Global Markets Inc., the fourth-biggest U.S. stock exchange, dispatched an alert that said it stopped sending orders to NYSE Arca as a precaution.

A connection was disrupted between Arca’s machines and the Nasdaq Stock Market data facility in Carteret, New Jersey, according to two people with direct knowledge of the situation. Arca’s server tried to connect to a computer system known as the securities industry processor, or SIP, which is run and maintained by Nasdaq and overseen by a committee of experts from exchanges.

Tape C

The SIP compiles price quotes for Nasdaq-listed stocks and shuttles them to the rest of the market in what is referred to as Tape C. NYSE runs its own SIP to distribute quotes for its stocks, Tape A and Tape B.

Disruptions to the data feeds are rare. In 2007, Nasdaq’s SIP prevented some stock prices from updating for about 23 minutes, 10 of them during the regular trading session.

In last week’s instance, Arca’s servers attempted to connect with the Nasdaq SIP and failed. The computers tried to make a connection multiple times, according to people familiar with the matter who asked not to be identified because they were not authorized to speak publicly. The first issue was resolved temporarily and at 10:26 a.m., NYSE Arca sent another update saying, “all systems are operating normally at this time.”

More alerts followed.

Orders Canceled

NYSE Arca said in a message it would cancel live orders in stocks with symbols that fell between the letters TACT and ZYY.C. The server that handles those securities on Tape C was the computer that couldn’t connect to Nasdaq, a person familiar with the situation said.

Bats declared what’s called self-help against NYSE Arca again, allowing it to bypass that exchange as it sent orders around the market in search of the best available prices.

By 10:37 a.m., the machines at Arca and Nasdaq were having trouble communicating again, according to a person familiar with the matter. As the issue progressed, Nasdaq closed the port that connects to Arca, the person said. Arca switched to a backup server around this time in an attempt to restore the link.

Arca, the legacy of the New York Stock Exchange’s merger with Archipelago Holdings Inc. in 2006 to increase its electronic trading capabilities amid competition from Nasdaq, handles about 11 percent of American share volume. Now the ability of the world’s largest stock market to operate normally hinged on whether the rivals’ computers could communicate with each other.

Under Stress

Nasdaq’s server was under stress and running short of the memory it needed to process Tape C, according to two people familiar with the matter. As the memory ran out, the machine went down.

Greifeld, 56, was at a wake in New Jersey in the morning and rushed back to his office on the 50th floor of 1 Liberty Plaza in lower Manhattan, listening to updates on the phone in the car, according to a person familiar with his actions.

He was concerned about what he called “information asymmetry.” Some proprietary and high-frequency trading firms, whose lightning-fast strategies can execute thousands of trades a second, were still getting accurate data from exchanges, Greifeld said in the Bloomberg TV interview. Other investors, including individuals trading at home, had to rely on the prices from the so-called SIP and were now out of luck.

Conference Calls

Emergency conference calls were arranged and the lines were left open, Greifeld said. One was with regulators including officials from the Financial Industry Regulatory Authority and the SEC. Another had representatives of exchanges discussing the SIP, which is housed in the same Nasdaq data facility as the machines that match buyers with sellers.

“We couldn’t take any moves with the SIP because it’s really not ours to control without them being involved,” Greifeld said in the Bloomberg TV interview. “That open line went through the whole process, so it wasn’t like it was episodic. It was constant.”

By 12:14 p.m. Nasdaq said it was halting all of its listed stocks, which have a market value of more than $5 trillion. Its three options exchanges were then shut down and all its listed shares stopped changing hands on outside venues.

Before computers started playing a dominant role in securities markets, traders were accustomed to frequent interruptions. Markets closed on many Wednesdays in the 1960s and observed shortened hours as daily trading of 10 million to 12 million shares meant “brokers were literally buried in paperwork,” according to the Depository Trust & Clearing Corp.

Computer Trades

In today’s market, where computers can move more than 10 billion shares on a busy day and humans react to news in seconds, interruptions have become less common.

At First New York Securities LLC in New York, partner Andrew Ross was watching news that could move stocks. Billionaire investor Carl Icahn had said on Twitter that he spoke to Apple Inc. CEO Tim Cook and the two were planning to have dinner. Bloomberg News reported that Amgen Inc. was pushing to reduce its takeover offer for Onyx Pharmaceuticals Inc.

Ross was paralyzed, unable to trade on the news, as stock charts showing instant updates of prices turned into horizontal lines for Nasdaq-listed companies from Apple to Zillow Inc.

At JonesTrading Institutional Services LLC in Chicago, traders sought to relieve tension by squeezing rubber stress balls, said Christopher Rich, head options strategist.

Idle Hands

“If you take everybody’s toys away, then they have nothing to play with, so everybody’s going ‘I’m mad,’ but there’s nothing they can do about it,” Rich said in a telephone interview.

President Barack Obama was told about the malfunction and the SEC’s White said she planned to convene market officials to discuss ways of making trading more reliable. Treasury Secretary Jacob J. Lew was questioned about the breakdown by a master of ceremonies after a speech in Mountain View, California. Lew said he saw no reason to believe the halt had “any of the more frightening aspects to it, but we’re going to have to learn all the facts.”

The malfunction was the latest in a series of mishaps that have shaken confidence in American computerized equities trading since the faulty use of an algorithm briefly caused markets to erase $862 billion in share value in May 2010.

Past Breakdowns

Just two days before the latest incident, a Goldman Sachs Group Inc. technology error inundated options markets with unintended orders. Nasdaq agreed to pay a $10 million fine to settle SEC charges stemming from its mishandling of Facebook Inc.’s initial public offering last year as regulators cited “poor systems and decision-making.” A programming error almost bankrupted Knight Capital Group Inc. last year, and Bats had to cancel its IPO after failing to get the shares to start trading on its own platform.

Back in New York and Carteret, a New Jersey town across the Arthur Kill tidal straight from Staten Island, Nasdaq was making progress. While the SIP’s technical issue was resolved by 12:24 p.m., executives were reluctant to reboot in the middle of a session, concerned they would spur unpredictable price swings.

Shortly before 1 p.m., Nasdaq alerted clients that it would re-open trading “at a time to be determined,” and that it would not cancel open orders, so traders needed to withdraw their bids and offers themselves. At 2:20 p.m. it advised that quotes would start running on test stock symbols -- imaginary shares -- in 10 minutes with real trades of the fake stocks starting at 2:45. Real trading of real stocks would start at 3:10.

Landing Gear

“I call it really like preflight risk checks,” Greifeld told Bloomberg TV’s Betty Liu. “The system came down, we had it fixed within a half hour, we’re ready to go, we’re involved in intensive communications going through the preflight checks.”

That alert proved too optimistic and the exchange pushed back the open another 15 minutes. By 3:25 p.m., Nasdaq stocks started ticking again. With the close about 35 minutes away, both the Nasdaq Composite Index and the Nasdaq 100 Index began to rise.

There was more turbulence. At 3:37, Nasdaq declared self-help against NYSE Arca.

Finally at 4 p.m., James Hughes, CEO of Clinton, New Jersey-based Unity Bancorp Inc., a community lender with 166 employees and a market value of $60 million, stood on the dais of Nasdaq’s MarketSite in Times Square accompanied by a group of smiling colleagues to ring the closing bell.

Last Trades

The last trades of the session were tallied and at 4:11 p.m., the exchange said the end-of-day auction was done.

“The Nasdaq Closing Cross has completed as normal and the extended trading session has begun,” it alerted traders.

While only eight stocks in the Nasdaq 100 finished the session lower, Nasdaq’s own shares retreated. They closed down 3.4 percent at $30.46 for the biggest loss in more than four months. The next day, the shares recovered 1.2 percent to $30.83.

“It will come back,” Greifeld said before the market opened on Aug. 23. “It’s a great buying opportunity.”

Before markets opened today, two of Greifeld’s biggest competitors confirmed they were teaming up. Bats Global Markets and Direct Edge Holdings LLC said they would combine, uniting the third- and fourth-largest equity exchange owners in a transaction expected to close in the first half of 2014.

Nasdaq shares were up 0.3 percent at $30.91 as of 10:18 a.m. in New York and have risen 24 percent this year. Shares of NYSE Euronext, which is being taken over by IntercontinentalExchange Inc., added 0.4 percent to $42.83 and have rallied 36 percent this year.

Before it's here, it's on the Bloomberg Terminal.