10 Steps to Building Your Energy Efficient Datacenter

Making the datacenter greener can be a hard sell. Most people are concerned it will cut into profits or performance. Say, for example, that an organization cuts over 4100 metric tons of carbon emissions from its facilities. Most decision makers will find this laudable — but wonder whether the impact upon the enterprise is as significant as the benefit to the environment. Plus, the need for organizations to respond quickly to new challenges often obscures green initiatives — just ask the IT and real estate team at Sun.

When Sun decided to consolidate more than 200,000 square feet of datacenter space in the San Francisco Bay area into a new facility in Santa Clara, California, environmental concerns were just one of several considerations. Time was critical, with a goal of completing the move in just 12 months. Reducing energy costs and floor space was also paramount, and the team knew that if it could meet those objectives, environmental benefits would follow. To meet space and budget requirements, the new environment would need to support the same amount of users (and more) in less space.

At the end of its aggressive 12-month schedule, the team succeeded in compressing 202,000 square feet of datacenter space into just 76,000 square feet — and reduced power consumption by over 60 percent in the first phase alone. "A simplified approach that combined new energy efficient hardware and innovative power and cooling solutions made this possible," says Dean Nelson, director of Global Lab & Datacenter Design Services (GDS) for Sun. Replacing hardware made the space and utility reduction targets possible, while also enabling Sun to avoid more than $9 million in construction and infrastructure costs.

Nelson and his team sat down with Sun Inner Circle to share Sun's best practices and outline the top 10 ways other datacenters can achieve similar results. "There's nothing mysterious about how we cut costs in the datacenter," he says. "We used commercially available products throughout the project that are accessible to any organization.

"We think our list of 10 tips will be helpful for not just reducing datacenter costs, but also in assuring IT managers that you can in fact do more with less," Nelson says. "At the end of the project, we actually increased compute performance by over 450 percent even after we cut the server count almost in half."

1. Check the Vintage of the Systems in Your Server Racks
2. Identify and Replace Other Energy Hogs and Space Wasters
3. Work on Immediate and Long-Term Replacement Plans
4. Make Sure IT Management Works with Its Facilities Counterparts
5. Take Power Outages Out of the Picture by Updating in Parallel
6. Upgrade to Similar Architectures to Avoid Migration Issues
7. Take a Look at Virtualization and Container Technologies
8. Turn the Meter On at the Rack Level
9. Put the Chill on Hotspots Instead of the Entire Environment
10. Rethink Cooling and Cut Construction Costs

1. Check the Vintage of the Systems in Your Server Racks
Unlike fine wine, computer hardware rarely ages gracefully. According to Nelson, "Rooting out old systems is often the easiest way to make a datacenter more efficient. Old hardware almost always consumes more space and power than new systems. Plus, older systems usually are more difficult to cool efficiently."

Nelson and his team replaced a variety of older servers with nearly half the amount of servers by using UltraSPARC T1, UltraSPARC IV+ and multicore AMD Opteron processor-based systems. In addition to refreshing its server and storage technologies, Sun decommissioned over 5000 server, switch, and storage devices.

2. Identify and Replace Other Energy Hogs and Space Wasters
Nelson cautions that not every piece of hardware responsible for wasting energy and space necessarily dates to the dot-com era. "Most datacenters are impenetrable jungles of disparate systems that create excessive power and cooling demands while wasting space," he says. "But you can often gain a minimum of 2:1 ratio of consolidation by replacing unwieldy systems with smaller and more powerful systems."

The team identified over 2100 servers and 700 storage devices that were inefficient — and then began replacing these systems. For example, the team replaced 88 Sun Fire V880 systems with 58 Sun Fire T1000 and T2000 systems, which reduced both power consumption and floor space required. In addition to helping cut space requirements, these new servers achieved more than 450 percent increase in compute performance with almost half the number of boxes. Combined with new storage systems, the datacenter now uses over 60 percent less power, which amounts to an estimated savings of more than $1 million a year.

3. Work on Immediate and Long-Term Replacement Plans
Finding and replacing the most obviously inefficient systems should be augmented with a long-term replacement plan that allows for future growth.

"Our experience leads us to believe that replacing systems every three years frees up resources," says Nelson. "With the energy savings and rebate possibilities, these refresh programs can be extremely cost-effective. Additionally, metering today's power and cooling requirements gives organizations the ability to map out next steps intelligently and in line with the needs of the entire enterprise."

4. Make Sure IT Management Works with Its Facilities Counterparts
At most organizations, the CIO doesn't pay the power bill, regardless of how much electricity is being consumed by the datacenter. Instead, the COO writes the check to the utility company, and this often fuels a debate over the importance of compute power over cost savings. It's an unnecessary argument — and counterproductive to everyone's interests in the enterprise.

Nelson says that close cooperation between the CIO and the COO can result in increased compute power and cost savings. Companies need to ensure that there is collaboration between the IT and facilities organizations early in the process to ensure an understanding of the datacenter plans and how energy efficient computing will help address both cost and compute power needs.

As an example, Nelson points to how his team routinely replaced two or more V880 servers with a single Sun Fire T2000 system for more compute power and less operational expense.

"For example, a V880 server costs $1,300 in energy costs annually and takes 18U of space," he says. "Compare this with a T2000, which uses approximately $200 worth of electricity in a year and takes 2U of space. Then add rebates from utility companies like PG&E which offers $700-$1,000 per server and the equipment costs become negligible."

5. Take Power Outages Out of the Picture by Updating in Parallel
Upgrading systems always opens the risk of system failure — which is not how most organizations imagine lowering their power bills. That's why Nelson stresses the importance of planning on ways to reduce the possibility of outages during hardware updates. Yet as he notes, IT staff will often add equipment before identifying the systems that will become obsolete. This often results in unexpected brownouts in the datacenter as old and new systems vie for power.

Sun avoided this situation by staging all of the new equipment beforehand. When the hardware was ready and applications tested, there was a quick cut-over to redirect traffic to the new hardware. The new equipment took over and the old equipment was shutdown. The result? No outages.

6. Upgrade to Similar Architectures to Avoid Migration Issues
It can be tempting to simultaneously overhaul operating systems and applications while also upgrading to more efficient hardware systems.

Nelson cautions against this approach. "If you have a SPARC-based system running a certain version of Solaris, think about replacing the hardware with a more efficient SPARC-based system so that applications are migrated to the same platform," he says. "Start by compressing and reducing hardware, which alone can cut utility costs in half. This gives you a runway to properly plan further compression with other technologies such as virtualization and upgraded operating systems."

7. Take a Look at Virtualization and Container Technologies
Sun attributes much of the project's server reduction to use of Solaris Containers for virtualization. This virtualization approach isolates applications and services by using flexible, software-defined boundaries.

According to Nelson, Solaris Containers can allow you to achieve even greater compression in both servers and storage. On average, Sun achieved 2:1 server replacement and 3:1 storage replacement ratios, but in some cases, Solaris Containers have allowed server replacement ratios of up to 20:1. "With virtualization, we're approaching the point where the cost savings in space and energy are coming close to equaling the initial purchase price of the equipment," he says.

8. Turn the Meter On at the Rack Level
Legacy measurement of watts per square foot in a datacenter may show that the room is running fine on average, but in reality you have hot spots throughout that are damaging equipment. Most datacenters measure load at the perimeter of the datacenter, which, predictably, makes things unpredictable. It is important to take metering on step further. It needs to be measured at the rack level (watts per rack) to truly enable energy efficiency.

Sun measures power consumption at the rack level with IP addressable power strips that report energy usage to allow more exact capacity planning. Similarly, the servers and storage arrays in each rack come with built-in monitoring capabilities that show heat production. This further reduces power consumption by pinpointing attention on certain areas in a datacenter instead of using the traditional, scattershot approach of cranking up the fans when a particular area in the datacenter starts to run hot.

9. Put the Chill on Hotspots Instead of the Entire Environment
Today's rack density makes cooling spaces with traditional perimeter solutions, which push air throughout the datacenter, an expensive proposition. But by using two approaches that closely couple cooling with the heat sources in each rack, Sun was able to reduce cooling costs by up to 50 percent.

These solutions include Emerson Liebert XD cooling systems designed for high-density environments. Locating these systems above server racks reduces the distance cool air travels to address hotspots, while also localizing the neutralization of the exhaust heat at the source. Meanwhile, cooling systems from American Power Conversion Corp. placed between alternating rows of racks capture hot air emitted from the backs of servers, which is then transferred to cooling units located next to the racks.

By using both kinds of closely-coupled cooling approaches, Sun has been able to eliminate the need to indiscriminately pump more air into the entire datacenter when certain areas start to run hot.

10. Rethink Cooling and Cut Construction Costs
As Nelson notes, most IT managers invariably will say "raised floor" and "datacenter" in the same breath. Yet, he says, traditional datacenter design needs to be rethought in light of new cooling technologies.

"We showed that it simply wasn't necessary to build a raised floor environment," he says. "As the density per rack increases, the raised floors start to contribute to the problem because they cannot get the cold air to spots that need it. By using new cooling technologies that localized the cooling, we avoided an estimated $3 million in construction costs needed to build raised floors for all of our datacenters."

With the new datacenter now saving money on a daily basis, Nelson and his team are preparing to introduce these 10 steps to other Sun datacenters. And while the cost savings are tangible, so are the environmental savings. The project has cut over 4100 metric tons of carbon from Sun's global carbon footprint and combined with the operational benefits, a greener datacenter is no longer a hard sell.