USC Shoah Foundation Institute for Visual History and Education

Story Details

Anyone who does not immediately associate the phrase "never again" with the tragedies of the Holocaust can get a sobering reminder by viewing the testimonies at the University of Southern California (USC) Shoah Foundation Institute Web site. Since 1994, the Institute has collected, cataloged, and indexed more than 100,000 hours of videotaped interviews, making them available to participating universities around the world via Internet2. The Institute also works within the University and with partners worldwide to advance scholarship and research, to provide resources and online tools for educators, and to disseminate the testimonies in its archive for educational purposes. The Institute will also begin to help document the stories of survivors and witnesses of other atrocities, such as the 1994 Rwandan Tutsi genocide.

The Institute's highest priority is preserving these irreplaceable testimonies. For each interview collected between 1994 and 2000, the Institute digitized the original analog videotape and then made high-quality (270Mb per second) digital videotape copies for safekeeping. It also made a complete copy in digital MPEG1 format, suitable for viewing on a PC. Although nine times smaller than its videotape counterpart, this low-quality data library stored on hard disk is an astounding 135TB.

When it became part of USC in 2006, located in USC's state of the art datacenter, the Institute converted its infrastructure to Sun Microsystems. "Sun helped us migrate to new servers and provided 200 terabytes of storage space for the MPEG1data library," says Sam Gustman, Chief Technology Officer at the Shoah Foundation Institute. Still, the Institute needed a longterm solution for preserving videotapes. "The minute you record on videotape it starts to rot," says Gustman. "Conservatively, you have 20 years before you see data damage." By 2014, the Institute would need two new copies of the videotapes at a total cost of $14 million. And because hard disk deteriorates four times faster than video, the low-quality MPEG1 data library would have to be replaced every five years at a cost of about $250,000 each.

In 2008, Sun helped the Institute design a new strategy for preserving the archive. "For the first time [in history], it is more cost effective to preserve content in files than on videotape," says Gustman. "We're going to use digitizing equipment to make four petabytes of Motion JPEG 2000 files of the original analog videotapes and then store them on tape using Sun systems." The solution is based on the Solaris 10 Operating System and consists of eight Sun Fire X4500 servers, six Sun Fire X4600 M2 servers, two Sun StorageTek 6540 arrays, and the Sun StorageTek SL8500 Modular Library System with a capacity of 8.6 petabytes. The Sun StorageTek T10000B tape drives it uses hold 1TB of data compared to 250 or 500GB available from other vendors. "We can add more interviews to the archive without increasing the [storage] footprint in our space of 500 square feet," says Gustman. With the archive estimated to grow to 20 petabytes, the scalability the solution provides is a huge benefit. Regarding a disaster recovery plan, Gustman says, "It will be far easier and faster to make many copies of the archive that we can store in various places all over the world."

The Institute expects staff productivity to increase significantly. "Before, if we needed to work on a videotape, it took two weeks to get it from a remote location, where all the tapes are stored in an acre and a half of space. Now, a digitally processed video will be available to us within an hour," says Gustman. "That leaves us more time to concentrate on our educational program and drive our mission forward much faster than before."

Best of all, the new solution saved the Institute $6 million and will continue to meet its needs at a very low cost. "We plan to completely replace and recycle existing tapes every three years," says Gustman, "and even doing that, this solution is far less expensive than any other."