Queensland Brain Institute

Story Details

Every day at the Queensland Brain Institute, scientists are generating hundreds of Gigabytes of research data as they investigate the more-complex areas of neurology around behavior, cognition, ageing, disease, and mental illness. Some high-speed imaging devices, recording brain activity, can generate 20 GB of data in just a few seconds. And over a year, this all adds up to Terabytes of information that needs storing. Jake Carroll, senior systems programmer at QBI says, “We are preparing ourselves to hit the 1 Petabyte of research data mark at some point in early 2009.”

The majority of the data is far too big to sit locally on a desktop, so it needs to be stored in home directories on a network. But the files have to be easily accessible, because they are often crucial to QBI’s research programs. Says Carroll, “This data is the life blood of the institute. We have to make sure that it is available immediately on request. Scientists can’t wait around for research files from manual data retrieval when they are trying to prepare a scientific paper.”

Besides satisfying the needs of scientists, QBI has to meet the requirements of the Australian government, which says any experimental data has to be retained for seven years. While this doesn’t create any more pressure in terms of speed of access, QBI still has to organize the data so there is a clear pathway to the information whenever it is required.

The main challenge for QBI was that scaling the existing data storage was expensive. The only way to increase capacity was by adding more disks to existing arrays. But this solution was unsustainable because of the up front costs of buying new hardware, the expense of power and cooling, and the limited amount of physical space available. To make matters worse, Carroll discovered silent data corruption in a RAID 5 array. “We had to get even more serious about our storage infrastructure, bearing in mind the increasing costs and the challenges we were having,” remembers the senior systems programmer.

Despite Apple being the incumbent IT provider, QBI rejected a possible XSAN data-storage system in favor of a Hierarchical Storage Management (HSM) solution based on Sun Storage. The advantage of HSM is that it seamlessly moves data between high-cost and low-cost storage devices. For QBI, this meant storing a lot more data on low-cost tape, but still having clear pathways and quick access to stored files.

Working closely with Sun in 2007, Carroll and his team created a three-tiered Open Storage HSM infrastructure, mirrored to a disaster recovery site. The first tier featured Sun’s 6140 CSM200 Fibre Channel disk offering. The second tier consisted of multiple trays of Sun’s 6140 CSM200 SATA disk solution. This was controlled via ZFS. The third tier included dual Sun StorageTek SL500 modular library systems with 800 GB of uncompressed storage per cartridge. A Storage Area Network was built using 4 Gbit/sec Brocade Silkworm 200Es. Sun Fire X4200 M2 AMD dual socket dual core Opteron processors were the main load bearers for traffic between clients and storage.

Sun StorageTek Software Archive Manager-File System (SAM-FS) working with Sun StorageTek QFS technology managed the data between the tiers. Thanks to SAM-QFS, Carroll and his team could transfer files from the lower layer to the network very quickly and to multiple users without any problems. Solaris ZFS, with its simplified storage management methodology, provided a local file system for optimized throughput and the whole infrastructure ran on the Solaris 10 Operating System. Carroll says, “We saw that SAM-FS could archive, restore, and export data at zero-second intervals as well as offer two forms of restoration and snapshot technology in the form of ZFS and SAM-FS metadata”.

By implementing an Open Storage system, QBI created a flexible infrastructure, which can deliver more than 1 PB of data storage capacity at low cost. The storage solution delivers highly scalable storage capacity, at a cost approximately 55% lower than using a pure Tier 1 disk solution in the data center. QBI can deliver many times the storage capacity for less than half of the price, and with no vendor lock-in. Carroll comments, “Our Open Storage infrastructure gives us complete protection and redundancy and saves us important resources. We could have paid AU$100,000 in licenses just for replication software alone. Plus there would have been the additional costs for management software and other modules.”

Compared with the past, total cost of ownership for the new storage infrastructure is significantly reduced. “With Open Storage, we have put ourselves in a nice situation where we can add many Terabytes for just a couple of hundred dollars worth of tape,” says Carroll. In addition, power and cooling costs for the new infrastructure are much cheaper. A largely static tape library uses much less power than lots of spinning storage disks and creates far less heat. “You can imagine how important this kind of saving is to an organization that has a strong commitment to eco-responsibility,” says Carroll.

Because of the self-healing properties in ZFS, silent data corruption is now a thing of the past. The obvious benefit is that it helps QBI comply more easily with the Australian authorities, which demand that research be stored and protected. But over time, the self-healing properties of ZFS will deliver major benefits to QBI scientists, who know that the loss of just a small amount of research data can have a significant impact on the progress of individual programs. Likewise, the comfort from knowing data is protected reduces stress levels among the IT team.

Carroll also explains that the peace of mind their Sun solution offers doesn't end with ZFS: "The great thing about Sun Fire x86-64 bit based servers is that you're getting solid, well-engineered technology that really can deliver enterprise-class performance. Sun Fire servers backing onto an Open Storage platform is like the perfect marriage. My question is: why invest in expensive RAID controllers when you've got incredibly powerful x86 servers that can be the controllers for you." He applauds the Sun StorageTek 6140 Arrays for having the same build quality as the servers. "Sun StorageTek arrays also offer a pay-as-you-grow flexibility that helps protect your IT investments," he says. According to the senior systems programmer, QBI plans to move to OpenSolaris in the coming 12 months. "Solaris 10 is our legacy operating system, but I plan to move to OpenSolaris when we look to develop the storage solution further."

The solid engineering and commitment to service that Carroll has come to expect from Sun has also validated their choice of vendor. As Carroll notes, “Many people will suggest that 'an X86 server is an X86 Server' and there really aren't many product differentiators. I don't believe this to be true. If you look at how closely Sun actually works with both Intel and AMD, it shows a commitment to engineering that I don't personally see elsewhere. All this being said, selling products is one thing. It is the support that comes with it, that makes it a worthwhile purchase. When I call up Sun for an issue or a part, or if something has failed, I can have a replacement part on the tarmac at a local airport within hours, let alone locally in their warehouse. It works like clockwork, for us, at least.

The Open Storage HSM infrastructure has simplified day-to-day management significantly. Instead of four administrators working many hours a week managing storage, one administrator can do all the work in one hour. Carroll and his colleagues now have more time to work on other higher-value strategic work. Says Carroll, “We now have time to work on more client-facing issues, and building the internal wikis and calendars that help our research teams collaborate better and develop their research.”