Successful 100 Gbps testing at AUT a boon for large scale radio astronomy projects

Researchers from Auckland University of Technology (AUT) put the REANNZ network to the test, using a custom 100 Gbps-capable FPGA card to send packets around a 1,200 km loop, passing through more than 700 live devices and travelling through multiple PoP sites around the country.

The test had been designed to transit the REANNZ network in a way that didn’t affect other member traffic, running alongside production traffic with no faults or issues. The testing not only demonstrated the effectiveness of the custom FPGA cards and the software written for the receiving servers, but the reliability of the network, and the efforts that REANNZ engineers go to in order to support our members research goals and overall network access.

The Field Programmable Gate Arrays, or FPGA cards, used to direct the raw data are unique. They are programmable silicon devices which allows specific high-speed digital circuits to be programmed on to the chip, in order to achieve the required functionality.

The test was hugely successful, with the transfer resulting in zero packet loss and sustained speeds of 96.4 Gbps for more than an hour.

The challenge with transferring huge volumes of raw data lies in the distance and the duration that the data has to travel, with an ineffective solution resulting in packet loss or latency issues. The tests AUT conducted were designed to demonstrate that the FPGA card and the custom software developed by the team were capable of achieving sustained 100 Gbps speeds. If possible, the card and the network could then be used to directly transfer raw data at line rate straight into GPUs (graphics processing units), where the data could then be analysed in real time before being sent on to storage.

There was no delay caused at the receiving end for the transfer which can often become a bottleneck, as the messages were sent using the RoCE (Remote direct memory access over Converged Ethernet) protocol, where commodity ethernet networking equipment is used to deliver packets directly to system memory. This prevented packet loss as the receiver could ingest the data at line rate speeds all the while maintaining the integrity of the data.

Successfully building these technologies and deploying these solutions will be a huge contribution to the field of radio astronomy, with numerous collaborative research projects happening around the world, and countries like Australia and South Africa each hosting large scale radio astronomy arrays for the Square Kilometre Array Project.

AUT’s own 30m diameter radio telescope near Warkworth is globally networked through REANNZ. It is connected through a managed optical service that directly links our point of presence in Auckland and the telescope’s site within the Institute for Radio Astronomy and Space Research. New Zealand’s largest radio telescope contributes to many projects, including maintenance of GPS systems and mapping of the Milky Way galaxy.

Scientific instruments like these produce petabytes worth of raw data. Deploying these new technologies on site will enable researchers to send transfers straight into a data centre to be processed in real time, an essential capability that the FPGA card and high-throughput software enable.

REANNZ Weathermap during AUT's FPGA card testing

Where REANNZ comes in:

The process began with lab testing the FPGA card with REANNZ equipment, to simulate transmitting data over a distance. This was used to inform the setup of a network data collection point, the repository where the FPGA card would direct the data.

REANNZ was able to use its unique network, specifically designed to be flexible for research use, by reconfiguring the core to directly mirror the conditions in the field for the radio telescope arrays. REANNZ engineers looped traffic across the REANNZ New Zealand backbone to directly simulate the distances and intermediate conditions that would be encountered once the FPGAs were deployed. 

Initial tests from the FPGA card through a switch to the REANNZ server transferred 468 TB of incrementing integer data, at a throughput of 95 Gbps in a period of 11 hours.  Importantly, this occurred with zero packet loss. A second test also ran flawlessly for 22 hours, transferring 936 TB of data. Reliably preventing packet loss over long distance across high bandwidth connections is a key component of specialised science enabling infrastructures like the REANNZ network.

Proving that data could be sent across the network over a sustained period, the next stage was to test the transfer over a large distance. Working with REANNZ network engineers, the team were able to test the long distance 100 Gbps capability of the FPGA card and the REANNZ core network. One branch of the 100 G network was dedicated to the testing, to run a continuous stream of data from Auckland to Wellington and back again, resulting in a 1,200 km round trip. The other branch of the network remained open and accessible for the use of REANNZ members.

This testing led to the largest sustained transfer speeds yet seen on the REANNZ backbone by a research group - with sustained speeds in excess of 96 Gbps over a nearly 1,200 km distance.

“The people at REANNZ are great to collaborate with, they continue to be enthusiastic in helping with our research to demonstrate that long distance, high rate transport of data is possible without packet loss. I had prepared for an hour of panicked troubleshooting in our testing window, but Daniel almost instantly reconfigured the REANNZ network to route our traffic via Wellington and the REANNZ network performed so flawlessly that there was nothing to do apart from set the test running and go to lunch.

We are tremendously pleased with and proud of the results we obtained. Now we have the ears of our international collaborators, and the confidence to push New Zealand technology into mega-science projects around the globe. Having a world class research network with the knowledgeable people to run it is an absolute asset to Aotearoa.”

Dr William Kamp - AUT