Radio Astronomy research creates huge amounts of data – and, until relatively recently, for the team at AUT University’s Institute from Radio Astronomy and Space Research (ISASR), that created a significant challenge.
The Institute is conducting ongoing work using Quasars (huge, extremely radio-bright objects far out in space) as fixed reference points in the sky, allowing them to map out distances and points on the Earth’s surface with extreme precision.
It’s a project that’s critical to the accuracy of navigation systems around the world, including those used by millions on a daily basis, from their smartphones.
Collected via a pair of antennae at the Warkworth Radio Astronomical Observatory, the massive amounts of data received daily (hundreds of terabytes) must be processed, and the signals correlated, before it can be of any use.
Overseas, this is often done with the help of a High-Performance Computing (HPC) cluster. These are extremely expensive, high-maintenance set ups – and the team at AUT knew that if they were to invest the time and money required to build one, it was likely the hardware would be out of date before it was even up and running.
The reality of sending data back and forth across the Tasman, to the closest existing HPC cluster in Western Australia, was also unfeasible – with the undersea cable ill-equipped to carry such high levels of data at an efficient speed.
AUT had already been working alongside both REANNZ and Catalyst for a number of years, across a variety of projects.
Following extensive discussions between REANNZ and AUT, the idea was born of finding a way to leverage the network bandwidth, coupled with the capabilities of a domestic commercial Cloud Service provider, to conduct the correlation analysis on New Zealand shores.
With AUT having completed some internal testing to prove the viability of using a cloud infrastructure for this purpose, Catalyst was brought on board.
The final solution, the Long White Cloud Correlator (LWCC), uses a dedicated high-speed REANNZ link to transfer data from the ISASR’s Warkworth base, to Catalyst’s data centres around the country – then back again, once analysis is complete.
The nature of the cloud storage gives AUT extreme flexibility in building and tearing down a correlator environment quickly and easily, removing the cost of physical ownership and hardware.
We wanted to figure out how we could make use of the international bandwidth available to us from REANNZ, to take NZ from an astronomy data producer to a consumer and processor. Doing this has allowed New Zealand to move up the food chain!Stuart Weston, Radio Astronomer at AUT
The Role of the REANNZ Network
By keeping the data within New Zealand, we have eliminated the possibility of any latency as the result of geographical distances.
And where traditional commercial networks might fall down transporting such large volumes of data, the REANNZ network bandwidth (thanks to our 20Gbps link up and down the country) means the data transport can be completed reliably, and in close to real-time.
This is only possible as the result of extensive fine-tuning of the network, to ensure the lowest loss and latency, and specifically to support large data transfers, the likes of which are taking place between AUT and Catalyst.
The Long White Cloud Correlator wouldn’t be the clever solution it is without REANNZ’s dedicated high speed link – the network was the missing piece of the puzzle.Bruno Lago, Catalyst IT’s General Manager - Cloud
Apart from demonstrating the value and importance of collaboration between researchers, the private sector, and organisations like REANNZ, the LWCC has played a significant role in making radio astronomy research practical and possible for Kiwi scientists.
Without the capabilities required to process and transfer information at such high speeds, the data would quickly lose its value and relevance - and our ability to contribute to the international body of knowledge on radio astronomy would be extinguished.
The LWCC, and REANNZ’ part in it, has made New Zealand a world player in an area of science and innovation that is so critical to daily life.
REANNZ were absolutely crucial on this project - from the initial conversation, to the critical role of providing the infrastructure, to move the large volumes of raw data for processing. The cooperation and willingness to make things happen quickly and easily has been fantastic – once we’d had that initial meeting, decisions were made and acted upon very quickly.Stuart Weston, Radio Astronomer at AUT