Basically a computer in New Jersey.

Photographer: Spencer Platt/Getty Images.

Market Complexity Broke the NYSE Before Saving It

Matt Levine is a Bloomberg View columnist. He was an editor of Dealbreaker, an investment banker at Goldman Sachs, a mergers and acquisitions lawyer at Wachtell, Lipton, Rosen & Katz and a clerk for the U.S. Court of Appeals for the Third Circuit.
Read More.
a | A

So today the New York Stock Exchange published its explanation of why it shut down for much of yesterday. Here it is. Basically NYSE did a software update, and when it opened up the machine to put the updated software in, some bats got into the machine and flew around breaking stuff, and so NYSE had to open the machine back up and catch all the bats, and that took a while. I may not fully understand how computers work.

Here though I want to focus on the first sentence of NYSE's explanation, which is this:

On Tuesday evening, the NYSE began the rollout of a software release in preparation for the July 11 industry test of the upcoming SIP timestamp requirement. 

So the software update was not to like change the font for the stock symbols; it was to prepare for "the upcoming SIP timestamp requirement." Let's talk about that. The SIP is the Security Information Processor, which "links the U.S. markets by processing and consolidating all protected bid/ask quotes and trades from every trading venue into a single, easily consumed data feed."

We talked yesterday about how stocks trade. A bunch of electronic trading firms post bids to buy shares, and offers to sell them, at a bunch of places. Those places are all basically computers in New Jersey, but they go by names like "the New York Stock Exchange" and "Nasdaq" and "BATS Y." When you want to buy stock, you go to your broker, and your broker looks at all the exchanges (the computers in New Jersey) and sees which one has the lowest offer -- that is, which one has the electronic trader who will sell the stock the cheapest. Then he goes and buys the stock for you there.

Conveniently, all those exchanges provide their bids and offers to the SIP, so your broker can just look at the SIP to see what the cheapest price for the shares is. But as with everything involving electronic trading, this is controversial. The SIP is slow. All the exchanges send their information to the SIP, which aggregates it into one feed and then sends it back out to SIP users. If you are an electronic trading firm, you might instead subscribe directly to the information feeds from the exchanges. This costs you more money, but the information gets to you faster than the SIP does. So when the SIP says that shares of XYZ stock are trading at $10, you might know that price is out of date by, say, a whopping 0.0001 seconds,  and that the current price is $10.01. And if you can find someone who will still sell you the stock at $10.00, you can turn around and sell it for $10.01 and make an easy penny. 

This is not particularly supposed to happen: If the price is $10.01, brokers aren't supposed to sell their customers' stock for $10.00. But lots of people worry that it is happening, and that high-frequency traders who know that the price is $10.01 get to buy at $10.00 in the time it takes the SIP to update. The concern is that some brokers and venues execute customer orders using SIP data that is out of date, leaving their customers vulnerable to high-frequency traders who take advantage of them.  (It's "front-running," for some definition of front-running. It's also frequently called "latency arbitrage.") This concern is the big motivating intuition behind IEX, the dark pool at the center of "Flash Boys," which uses a magic shoebox to delay high-frequency traders' orders to make sure they're not trading on information before IEX reacts to it.

It is also the motivating intuition behind "the upcoming SIP timestamp requirement." The requirement is that exchanges will have to report to the SIP, not just the prices and sizes of trades and quotes, but also their times, to the microsecond, so that customers and regulators can check up on how much of this sort of thing is actually happening :

The Participants believe that adding timestamps to the elements that Participants must report in connection with trade reports and bids and offers will improve transparency regarding the latencies between the CTA and CQ Plans' consolidated data feeds and industry proprietary feeds. Users of the consolidated feeds would be better able to monitor the latency of those feeds and to assess whether such feeds meet their trading and other requirements.

If you keep seeing bids and offers change, and then trades executing at the old prices 50 microseconds later, you can figure out that something's up. 

The key thing to notice is that this is all an artifact of the current fragmented state of the U.S. stock market. If there were only one exchange, then the price on the exchange would be the price on the exchange, and you'd never need to worry about trading at a stale price. But instead there are 11 exchanges and dozens of dark pools and retail internalizers and other places where stocks trade, and everything is, or might be, just a bit out of sync. Those disparities create profit opportunities: If you fully understand the complex fragmented market, you can find ways to exploit it. And so in turn there is a complicated regulatory overlay to try to shut down those exploits and bring things back into sync. The whole system is dynamic -- new exploits beget regulatory changes which beget new exploits etc. -- and the regulatory structure sometimes has to be tweaked. And sometimes those tweaks break markets.

And then it's fine because of the fragmented market! The NYSE was down for about three and a half hours yesterday, but no one much missed it, because there were 10 other exchanges, and lots of other venues, where you could trade stocks. That fragmentation, and the desire to help customers deal with it in a fair way, is what got NYSE into yesterday's mess. But it's also what made yesterday's mess an embarrassment rather than a disaster.

  1. I'm going to quote the whole thing here because it is short and that link goes to NYSE's full Market Status History, not to the individual explanation:

    On Tuesday evening, the NYSE began the rollout of a software release in preparation for the July 11 industry test of the upcoming SIP timestamp requirement. As is standard NYSE practice, the initial release was deployed on one trading unit. As customers began connecting after 7am on Wednesday morning, there were communication issues between customer gateways and the trading unit with the new release. It was determined that the NYSE and NYSE MKT customer gateways were not loaded with the proper configuration compatible with the new release.

    Prior to the market open, gateways were updated with the correct version of software and stocks opened at 9:30am. However, the update to the gateways caused additional communication issues between the gateways and trading units, which began to manifest themselves mid-morning. At 11:09am, NYSE issued a Market Status message that a technical issue was being investigated. At 11:32am, because NYSE and NYSE MKT were actively trading but customers were still reporting unusual system behavior, the decision was made to suspend trading on NYSE and NYSE MKT. NYSE ARCA, Arca Options and NYSE AMEX Options were not impacted by this event and continued to trade normally.

    NYSE and NYSE MKT began the process of canceling all open orders, working with customers to reconcile orders and trades, restarting all customer gateways and failing over to back-up trading units located in our Mahwah, NJ datacenter so trading could be resumed in a normal state. In consultation with regulators and industry, we determined that we would implement a complete restart and that NYSE MKT primary listings would resume trading at 3:05pm and NYSE primary listings and NYSE MKT Tape C symbols would resume trading at 3:10pm. Trading resumed as scheduled and the closing auctions accepted orders and executed normally. All NYSE and NYSE-MKT listed securities traded for the entire day either on NYSE and NYSE MKT or other market centers.

    I think it is fair to say that this is not, like, a particularly specific explanation of what went wrong. Like:

    1. We installed a software update.
    2. Some bad stuff happened.
    3. We fixed it.

    There is not a lot about the actual bad stuff.

  2. That is, 100 microseconds. IEX's magic shoebox, described in "Flash Boys," uses a 350 microsecond delay. Nasdaq is planning to reduce SIP delay to 25-50 microseconds:

    As a result of upgrading the platform, Nasdaq will reduce latency to 50 microseconds within the initial introduction of INET in early 2016. It expects to attain 25 microseconds after one year of introducing INET.

  3. This obviously elides a bunch of stuff, particularly that the issue is more quotes than last-trade prices, but never mind.

  4. BATS got in some trouble for using the SIP on some of its exchanges. People also worry about banks' dark pools, which sometimes use the SIP, and about wholesalers who get retail orders and trade against them for their own account. Here is one such worrier:

    In the case of a Market Order, an order to immediately buy or sell a stock at the best available price, a Wholesaler may decide to trade against a retail order if they can offset a position already in inventory or if they can immediately profit by trading against the retail customer using a stale price on the SIP (Securities Information Processor, a.k.a the consolidated feed) at a time when the Wholesaler knows that the actual market price has already changed, according to its faster direct market data feeds.

    I should say that I don't really know how often this happens and get different answers from different people I ask. I also don't know if selling retail orders to a wholesaler who regularly did that sort of SIP latency arbitrage would violate the retail broker's best execution requirements, which are separate from the Regulation NMS order protection requirements that the SIP fulfills.

  5. Some translation: "The Participants" are the exchanges and Finra. "The CTA and CQ Plans' consolidated data feeds" are the SIP (for NYSE listed stocks). "Latency" means that the SIP is slower than the direct feeds.

This column does not necessarily reflect the opinion of Bloomberg View's editorial board or Bloomberg LP, its owners and investors.

To contact the author on this story:
Matt Levine at mlevine51@bloomberg.net

To contact the editor on this story:
Zara Kessler at zkessler@bloomberg.net