Nasdaq blames software bug for outage
30 Aug 2013
Nasdaq OMX Group's massive trading halt last week came due to a software bug and other internal technology issues triggered by problems at NYSE Euronext's Arca exchange that caused the failure of a backup system, according to the exchange operator.
Expressing its deep disappointment, over the three-hour outage on 22 August Nasdaq said it pointed to connection problems between NYSE's electronic exchange Arca and the Nasdaq-run system that received all traffic on quotes and orders for Nasdaq stocks.
Nasdaq took ultimate responsibility for the glitch.
Bob Greifeld, Nasdaq's chief executive said in an interview that the exchange's backup system did not work, adding there was a bug in the system.
He said the exchange needed to ensure it did not happen again, referring to the system failing to fully revert to backup mode.
Nasdaq added it was working on identification of potential design changes for increasing resilience of Securities Information Processor, or SIP, "including architectural improvements, information security, disaster recovery plans and capacity parameters."
Nasdaq plans to submit its initial recommendations for change to the SIP governing committee, made up of US exchanges and the Financial Industry Regulatory Authority, within 30 days.
Nasdaq's version of events clarified what Greifeld referred to obliquely when he said "another player" was part of the crash.
Greifeld said last week, Nasdaq had to improve "defensive driving", meaning what one did when another part of the ecosystem, another player, faced some adverse event that triggered something in one's own system.
Other sources, according to The Guardian newspaper, however did not agree with Nasdaq's analysis about the NYSE, and squarely pinned the blame on Nasdaq's system.
The newspaper cited a source familiar with the NYSE's systems, who questioned as to why Nasdaq did not cut off the NYSE's access to the quote system when the problem started.
According to the source, they needed to have the ability to manage this by throttling the input, and the system itself should not melt down.
According to Eric Hunsader, a market structure expert and CEO of Nanex LLC, who posted an independent analysis of the event, the problem started when the NYSE sent out "quote loops".
He theorised that the NYSE sent out 50 minutes' worth of quotes in three seconds, and went on to repeat the loop for stock quotes coming from each of the 13 exchanges.
Hunsader concluded that the problem, basically lay in Nasdaq's own software. "(NYSE), and anyone external to (Nasdaq's stock quote processing system) could not have caused the quote bursts which probably led to Nasdaq deciding to shut down the SIP," Hunsader wrote.
