We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message
RSS FeedTechnology

NYSE Euronext boosts network bandwidth

For the stock exchange, the flatter the network, the better

Article comments

NYSE Euronext has rolled out 10GbE throughout its data centers, and with the new networking technology the stock exchange boasts 2.4Tbps aggregate bandwidth or a sub-75 microsecond round trip for network messages.

"It's a screamer," said Andrew Bach, the stock exchange's senior vice president of network services. NYSE's previous gigabit ethernet network had a total aggregate throughput of .5Tbps. "We've been able to get our wire speed up to 10 gig sustained, regardless of packet size, and we've been able to drop our latency significantly.

The stock exchange, which daily executes tens of millions of trades for 1,200 broker-dealer clients, performed the multimillion-dollar upgrade in networks that are part of its existing infrastructure. The 10Gbps Ethernet was also built into two new data centers, one just outside of London and the other in Mahwah, N.J.

The exchange is in the middle of a data center consolidation project that would reduce the number of facilities from eight to four or five, according to Bach.

The 10GbE upgrade was required to handle the increasing amounts of trade messaging traffic, which is staggering. In the US alone, NYSE exchanges process more than 22.4 billion messages representing $70.8 billion in financial transactions every day. The stock exchange said it handles about 7.5 times more traffic than the number of Internet searches that Google handles daily.

The stock exchange also sells market data to broker-dealers, which represents a large portion of its network traffic.

Using network switches from Juniper, the exchange deployed more than 1,000 Solarflare SFN5122F server adapter cards from Solarflare Communications as well as the vendor's OpenOnload application acceleration software. NYSE Euronext has about 6,000 servers, but many of them have up to six 10GbE ports, Bach said. Only about 15 percent of server ports are used for inter-switch linking, something the exchange reduced significantly over the past three to four years.

"We're headed for a data center fabric with top of rack switching methodology. It's much more scalable on the bandwidth ... lower latency, and more importantly, ever lower jitter," he said. "In short, the flatter the network, the better."

For its main trading platforms, NYSE explored using InfiniBand, which has a maximum quad data rate of 40Gbps, but it is nowhere near as ubiquitous as ethernet.

"We still use InfiniBand in some areas ... but the vast majority of the install base is ethernet. In terms of good 'ol market competition, you can select from any number of different vendors on the ethernet side, while you have a limited selection on the InfiniBand side," Bach said. "You get to play one vendor against the other."

One of the biggest challenges in the new network deployment was responding to microbursts of traffic - bursts that in a 1 millisecond window would peak at 5Gbps to 9Gbps on a single data feed. It was an exercise in load balancing to address the issue, Bach said. "You trade off pacing the feeds out, but of course the more pacing you do, the more latency you have. So you have to balance those two off carefully. On the receiver side, obviously we used larger buffers ... to absorb it," he said. "It really forces you to make sure all parts of your network can run at 10 gig wire speed end to end."

While the NYSE has yet to incorporate its data storage traffic into its Ethernet network - it uses the Fibre Channel devices - consolidating that traffic is a project Bach is considering now that the bandwidth is there.

Unlike many of today's large enterprises, which pride themselves on consolidating their server environments through hypervisors, virtualisation has a negative side for the NYSE. Sharing compute nodes among applications increases latency, which effects trade executions.

"Virtualization costs you a penalty of a couple of microseconds. That's something I'd rather give back to the customer," Bach said. "So in the core trading applications, we don't virtualise."



NYSE Euronext boosts network bandwidth

What does your perfect datacentre look like?

What does your perfect datacentre look like?

The financial impact of rising energy bills is a headache for companies toomore ..

Glitch hits Virgin Media email distribution lists

Customers' personal information accidentally shared amongst the email chainmore ..

BBC tenders for £5.4m digital services contract

Digital services roster may be 'opened up' to new entrants at regular intervalsmore ..

Guardian, Washington Post win Pulitzer prize for NSA spying coverage

Both publications won the highly coveted Public Service awardmore ..

What UK business needs to do now with big data

In the final of a three-part series, we look at the potential for companies to become victims in the battle for big datamore ..

Ethical questions around big data

In the second of a three-part series, Pat Brans looks at the ethics of collecting 'big data'more ..

Send to a friend

Email this article to a friend or colleague:

PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

In Depth
How M&A teams can create value by challenging the CEO

How M&A teams can create value by challenging the CEO

A typical “hold” period of nine to 18 months can generate increased sale value more ..

In Depth
What every company needs to do about big data?

What every company needs to do about big data?

In the first of a three part series, Pat Brans explores just how big 'big data' will get? more ..


* *