Climate modellers transfer 11 terabytes in 24 hours

Dr Johnny Williams

When NIWA climate scientist Dr Jonny Williams sent a massive 11 terabytes of data to research counterparts in the UK, he didn’t give the size of the transfer a second thought. The REANNZ advanced network seamlessly delivered the entire dataset within 24 hours, and without scrambling or dropping data.

11 terabytes is a lot of data in anyone’s language. It’s roughly equivalent to 5.5 million hours of standard definition film, 3.4 million photographs or 2.2 million songs.

The mammoth transfer of numerical climate data is part of NIWA’s ongoing work with the UK Met Office and Korea Meteorological Administration in joint research for the sixth UN Intergovernmental Panel on Climate Change Assessment Report (IPCC AR6).

Jonny is working with NIWA colleagues Guang Zeng and Olaf Morgenstern to set up and run several atmospheric simulations. Their simulations are among hundreds from modelling centres around the world that will feature in the IPCC’s next weighty assessment report.

He automated the big 11-terabyte data transfer as around 40 separate transfers that ran over the course of a single day. The simulations themselves relate specifically to how chemical species in the atmosphere such as ozone will change in the future because of emissions from human activity.

Contrary to what you might expect, there are no major challenges involved in a data transfer this big. It’s dead easy, says Jonny. So much so that he didn’t give the size of the transfer a second thought. That’s all down to the quality of the specialist REANNZ network, as well as NeSI's data transfer platform and Globus software used to send and receive data.

REANNZ is New Zealand’s advanced National Research and Education Network (NREN). It works seamlessly with over 120 NRENs to provide a specialist international network architected for the unique needs of data-intensive research traffic. NRENs are designed to deal with big data, so there is negligible congestion and therefore no jitter or packet loss.

Jonny has been using the REANNZ network on an almost daily basis for around the last two years. He says advanced networks like REANNZ are transformational in letting researchers collaborate faster and more effectively. He likens them to the difference between a mobile phone from the early 2000s and a modern smartphone in making researchers’ lives easier and enabling new applications.

“Without REANNZ’s high-speed data transfer network we wouldn't be able to transfer this amount of data this quickly. It's that simple. What took us one day could easily have taken months without REANNZ."

“If you can do something faster, it accelerates your workflow. But sending and receiving large data sets quickly also completely transforms your workflow.”

It’s not just about sending data. Climate modellers like Jonny, Guang and Olaf equally rely on the speed and reliability of advanced networks to receive large amounts of information from other researchers around the globe.

Engineers from REANNZ say in total the full transfer over the advanced network amounted to 25 terabytes over three days.

The next large climate output data transfer from NIWA to the UK is set down for a few weeks' time.

Credit: NIWA staff writer Campbell Gardiner