A storage complication with a Cloud solution - AgResearch, Microsoft Azure and REANNZ

The REANNZ network was ready to take on an unexpected transfer that contained terabytes of data, working efficiently and at incredible speed to effortlessly assist AgResearch with their storage issue.

When an unexpected amount of data was placed in AgResearch’s current storage solution, Dan Sun, an HPC consultant with AgResearch, and the infrastructure team were able to recover quickly and effectively by utilising the REANNZ network and Microsoft Azure.

The storage system that AgResearch had in place was based on Network Attached Storage (NAS) technologies. These devices are flexible with scalable capacity, but they also have limitations. When an individual fileserver is used to capacity it can cause a significant strain on its performance, which is what the team at AgResearch were experiencing.

The amount of data that was placed on the user accessible fileserver pushed its utilisation to over 85%. With no other storage space available on site to offload this additional data, and with the fileserver’s performance degrading, Dan and the infrastructure team at AgResearch had to come up with a solution.

The solution – Cloud storage

They decided to move large datasets that had not been accessed by users for some time to reclaim capacity quickly. The team had to maintain the same level of data protection, that ensured two separate copies of the same data were stored at separate locations. Using the REANNZ network Dan was able to move 2 TB of copy datasets from the AgResearch data center in Christchurch offsite to the Microsoft Azure Blob storage instance in Australia.

Among other uses, Microsoft Azure Blob storage is designed for storing data for backup and restore, disaster recovery, and archiving. This suited AgResearch’s storage situation as they were able to choose from four storage tiers depending on how often they would access the data. Dan and the team were able to configure the Cloud storage to automatically migrate data to archive after being in the Cloud for seven days.

REANNZ Network performance

“We were extremely pleased by the performance of REANNZ’s network when we were uploading data to the Microsoft Azure’s instance in Australia. We were able to upload 2TB of data in just over 37 minutes, which translates to 7 Gbps on average.”

Dan Sun, AgResearch

The REANNZ connection to AgResearch Christchurch is 10 Gbps. This meant that the connection was working at speed, as fast as the connection can go, throughout the majority of the transfer. It then took the team another two hours to remove the dataset on the fileserver that was running out of capacity, meaning that it actually took longer to remove the dataset files than it did to send them across to Australia via the REANNZ network.

In just under three hours the fileserver was stabilised. When the situation was resolved, Dan went on to upload a further 6 TB of data on to the Microsoft Azure Cloud system to reclaim capacity from the same fileserver with ease.

With the accelerated uptake of digital tools and services during the COVID-19 pandemic, there has been significant changes to the ways in which research organisations store and process the data they produce. Via the REANNZ network, members are able to connect directly with Cloud service providers like Microsoft Azure, achieving considerable cost savings and efficiencies in performance.