State of Michigan, Department of Information Technology

Sun Services and Storage Solution Boosts the Efficiency of Michigan’s Backup and Recovery Processes, Cuts Costs by up to 70%

The State of Michigan’s Department of Information Technology (DIT) provides IT services to the state’s agencies such as the departments of Treasury, State Police, Education, Human Services, Corrections, Secretary of State, and Attorney General. Established in 2001 to reduce IT costs and improve efficiency, the DIT has worked to consolidate IT from all of the state’s agencies into one centrally managed infrastructure. DIT is based in Lansing and has more than 1,700 employees.

Customer Challenges

Decrease the risk of data loss and improve the recoverability of business data
Better policy alignment
Simplify processes
Boost scalability and reliability
Improve service levels

Solution

The solution provides for a combination of onsite and remote services to meet demands for monitoring and management of the backup and recovery processes 24x7x365. In addition, the solution provides for greater capacity, performance, and control through tape storage, media servers, master servers, and restructured backup processes and policies.

Business Results

Boosted backup success rate from less than 60% to 98%
Reduced backup policies by 50%
Cut the time required to add new servers by 80%
Increased number of servers in the environment by 55%
Improved planning and management of backups as a result of enhanced reporting

Products and Services

Story Details

Consolidating 29 datacenters to 3 primary data centers and shifting control of those facilities from multiple agencies to one is no small task. Six years into its consolidation efforts, Michigan’s Department of Information Technology (DIT) was struggling with its backup and recovery architecture. DIT technical services employees could not keep up with the volume of new systems which numbered 1,100 servers in December of 2006. At any one time, about 200 servers waited in the onboarding queue for about 15 days to be backed up. In addition, issues that arose from the plethora of technologies and processes reduced the average backup success rate to below 60%.

In the fall of 2006, after evaluating solutions from seven vendors, DIT decided to work with Sun Microsystems — in part because Sun provided the best combination of onsite and offsite services using a cost model based on price per server. Elaborating on its choice, John Kozitzki, enterprise backup and recovery manager of technical services says, “We had a long-standing relationship with Sun. Sun's solution could leverage the equipment that we already had, perform an assessment to see where we were lacking in infrastructure, and then just build it up.”

" The return on investment by using Sun services isn’t just monetary. It’s confidence level. It’s ensuring that we can provide a service that our customers need. We have made a big turnaround here at the state in regards to enterprise backup and recovery, and Sun services has really helped us in that turnaround. "

— John Kozitzki, Enterprise Backup and Recovery Manager, Technical Services, Michigan Department of Information Technology

Between January and May of 2007, consultants from Sun Professional Services and Sun Managed Services underwent extensive security checks. A secure VPN connection was also established between the state’s datacenters and the Sun remote operations center. Sun Professional Services then assessed the existing storage environment and presented its recommendations, which provided for a multitiered backup and recovery solution.

Between October 2007 and March 2008, Sun consultants worked with DIT to increase the scalability and flexibility of the backup infrastructure. Sun consultants added or replaced master and media servers that run Veritas NetBackup — a combination of Sun Fire T2000 servers, Sun Fire T5220 servers, and Sun Fire V445 servers with the Solaris 10 Operating System. Capacity was added to an existing Sun StorageTek SL8500 modular library system. Two clustered T2000 servers that run the Solaris 10 OS, Sun StorageTek Automated Cartridge System Library Software, and Solaris Cluster software provide for greater redundancy and high availability for control of backups. In addition, the team standardized backup and recovery processes and created policies that addressed data categories, backup priority, workload balancing, and reporting. To further boost reliability, all of DIT’s Sun systems are protected by SunSpectrum Enterprise Service, and Sun Managed Services provides 24x7x365 offsite monitoring of the backup solution. Four consultants staffed through Sun local partner Dewpoint provide daily onsite support during business hours, maintenance windows, and emergencies.

An important strategic decision that increased the speed of backups and expanded the window for tape based backups was to divert backups less than 150GB in size to a disk-based Virtual Tape Library (VTL) system. This provided the benefit of allowing larger storage allocated servers to use the physical tape drives more efficiently and at earlier times in their backup windows, ensuring that these longer backups complete within their specified time frames. VTLs allow for the alleviation of backup volume issues by increasing backup speeds, reducing recovery times, and eliminating the physical constraints of an environment having a limited number of tape drives.

Implementing feedback from its customers and a recent Sun assessment, DIT recently added eight T10000 tape drives and additional servers — all Sun T5220s. The state now has 8 master servers, 20 media servers, 28 Sun StorageTek T9940B tape drives, and 36 Sun StorageTek T10000 tape drives. Collectively, the state's architecture has grown to include approximately 1,700 servers and 2.4 PB of data. Future goals include the deployment of two StorageTek SL3000 tape libraries and T10000Bs for redundancy and reliability.

As a result of its new enterprise storage, backup, and recovery solution — built and maintained through a partnership between state employees and Sun consultants — Michigan received a prestigious NASCIO award in recognition for outstanding achievement in 2008. One key reason for the award is that the state demonstrated that its server-backup services are repeatable and fully managed, 7x24x365.