Building Energy-Efficient Datacenters

Hello Sun Inner Circle readers. It’s Bob Worrall back with another installment in a series of letters that I publish in this venue to share my insights into the IT industry and latest developments here at Sun. Last month, I shared some of the larger lessons I learned during my first 100 days on the job as Sun CIO. This month, I’m going to discuss an issue that’s increasingly a hot topic (literally!) in today’s datacenters — energy efficiency and eco-responsibility.

If my readers will forgive me, more often than not, CIOs don’t have a clue about datacenter energy expenses. Typically, power bills are sent directly to the real estate team or the CFO. However, with the rising costs of energy worldwide, many CEOs are beginning to hold the CIO responsible for datacenter energy costs. That simple accounting change represents a monumental shift within today’s organizations — and one that I increasingly get questioned about by executives curious how Sun is managing higher energy costs.

To answer some of the queries, I invited Mark Monroe to join me for a Q&A. Mark has been at Sun for more than 13 years, during which time he has spent some time running Sun datacenters. Most recently, Mark went to work for Dave Douglas, Sun vice president of Eco-Responsibility, and now, as director of Sustainable Computing, Mark spends his time encouraging datacenter energy efficiency and determining what portion of Sun’s energy portfolio comes from green power. He understands how Sun is turning down the heat in its datacenters in reduce energy expenses and encourage eco-responsibility.

Worrall: Mark, you’ve been part of the eco-responsibility movement for a while. Why has the issue become such a hot topic for datacenter operators?

Monroe: It boils down to simple economics. Researchers at Lawrence Berkeley Labs have done extensive studies on datacenter efficiency and energy consumption. LBL estimated that 3 percent of the entire energy consumption of the United States — including automobiles, buildings, houses, and everything else that consumes power — derives from datacenters. Furthermore, power consumption rates are following Moore’s Law: As processors become more powerful, energy consumption grows at the same rate as processing power increases. Based on that, LBL concluded that the power consumption rates double every two to four years. Plus, the study that cited the 3 percent figure was done in 2004, and today in 2007, it’s conceivable that datacenters are guzzling upwards of 5 percent to 6 percent of energy in the U.S.

Worrall: Aside from growing energy consumption, what other factors are bringing eco-responsibility into the consciousness of IT professionals?

Monroe: The second piece of the puzzle is that the cost of operating computational resources is beginning to exceed the cost of acquiring the assets. When those two expenses are plotted on a chart (see figure 1) they cross (depending on a couple of factors) sometime between 2004 and 2015. In other words, we might have already passed a key tipping point related to the source of datacenter expenses. As this happens, the focus for CIOs and datacenter operators will naturally shift to the topic of energy.

Worrall: How much energy are we losing or wasting to power our datacenters?

Monroe: Sun studied that, too. We actually looked at the power used by datacenters — starting from coal and ending with bits on a screen. Typically, about 9 percent of the total energy vanishes as a result of line loss before the energy ever hits the datacenter. Once inside the datacenter, there is a metric called the Power Use Efficiency (PUE) that determines how much power comes into the meter versus how much power the IT equipment consumes.

The PUE at Sun’s Broomfield, Colorado, datacenter is somewhere between 2 and 2.4, so every watt consumed by a server requires 2 to 2½ watts of electrical power from the utility. There is a lot of loss — in the UPS, the switching gear, the power supplies of the machines, the wires that run through the floors, and the cooling. The good news — reducing power consumption at the machine is like getting a two-for-one coupon.

Worrall: What are key metrics that datacenter designers should focus on to help reduce power-related expenses?

Monroe: As the old adage says, measurement is key to management. Enterprises have to be able to measure the energy consumption of their datacenters, For instance, Sun can tell what portion of the electric bill is coming the datacenter versus the rest of the building. Plus, the company puts sub-meters at many different points in the datacenter, including before the UPS, after the UPS, and before the power distribution units — Sun even performs including rack-level monitoring. In the future, rack-level monitoring could be done remotely and fed into the overall server management system, so operators can use power as a variable to shift workloads during peak times in order to avoid issues like brownouts.

Worrall: What other metrics should people be keeping track of?

Monroe: Keeping track of systems age has been an eye opening experience at Sun. Newer technology is so much more energy efficient. Newer systems take the advantages of advances in power supplies and other technologies, and Sun has found that knowing the age of the infrastructure and systems — as well as having a plan for upgrading — can makes a big difference on the energy bill.

Worrall: I’d imagine that when you start measuring the age of the infrastructure, at some point it simply becomes cheaper to buy new equipment?

Monroe: Replacing older equipment is important, but so is doing so in a way that makes the most sense. Christian Belady, a distinguished engineer at HP, does a presentation that raises an important issue: If you look at the power of computing over time, why is it that a word processing program needs to run five times faster than it did three years ago? When the issue is examined from that point of view, many enterprises may begin to replace equipment with machines of the same compute power to take advantage of decreased electrical consumption.

Worrall: DC power in the datacenter has been mentioned as a way to lower energy costs. What are your thoughts about that?

Monroe: We did a study with Lawrence Berkeley Labs in conjunction with a number of industry players, including HP, IBM, Emerson Network Power, and an additional 15 to 20 organizations. We built a DC datacenter in our Newark, California facility as a demonstration and to measure the efficiency gain from not having to convert the power from AC to DC and back again. Typically there is about a 2 to 8 percent loss when the AC power that comes into the datacenter is converted to DC power to charge the UPS batteries — and then it is turned back into AC power and into DC power at three different places in the datacenter. So, we built a DC datacenter to see what would happen if we just get rid all of those conversions.

Worrall: And what did you discover?

Monroe: Well, it’s a bit complicated. We had to get high enough DC voltages in the datacenter to make it efficient to use. Telephone companies run on 48 volt DC power, but that’s really inefficient for an IT datacenter. So we decided to run the datacenter at 380 volts of DC power. Now, DC power is more dangerous than AC power, so there are safety issues, and we had a hard time finding electricians to hook up the 380 volt DC power. On top of that, there isn’t a 380 volt DC power plug. In other words, one of the reasons that DC power is not catching on more quickly is that it is tricky to do given the real world limitations. That said, there were some gains. We measured between 5 and 20 percent efficiency gains over a typical low voltage AC power datacenter.

Worrall: It sounds a bit unrealistic today given the real world challenges. Are there any alternatives to DC power?

Monroe: American Power Conversion (APC) has done a great job of demonstrating that high voltage AC power supply — 440 to 480 volt AC standard that is common in many parts of the world — can perform almost as efficiently as high voltage DC. In my opinion, DC power makes sense if a company is building a petabyte-scale datacenter from the ground up, and it can afford extensive customization. Then the 20 percent efficiency gains make a big difference in terms of cost savings. But, for a typical enterprise, DC power probably makes less sense than high voltage AC.

Worrall: I have heard a lot in the press about Google and Microsoft building new datacenters out in the Columbia River. What do you think of that idea?

Monroe: It’s a great eco-responsibility move on their part. There is a huge dam there — The Dalles hydroelectric dam — that Google and Microsoft hope to exploit for cleaner, cheaper energy. And, while the cost of commercial power in Washington and Oregon is two to two-and-a-half-times cheaper than in California, electricity is even less expensive in Nevada or Utah, where it is almost three times cheaper than in California.

I think the reason that Google moved to The Dalles is to use clean energy, as well as cost savings. Nevada, Utah, Wyoming, Montana, West Virginia, and Kentucky have the cheapest power in the country, but they rely on coal for 96 percent of their power production. While the economic motivation might be stronger or weaker at different companies, eco-responsibility should be an important factor, too, and it clearly was in the decision process for these two companies.

Worrall: Has Sun evaluated any eco-friendly solutions to the challenges posed by power consumption?

Monroe: We have. We’re working hard to get some solar photovoltaic arrays on the roofs of our buildings in Colorado. We’ve also looked at using some of our backup power capacity on a more regular basis. For instance, we have considered taking the diesel generators at the Broomfield datacenter and running them on a regular basis to do what’s called peak shaving, which could have a big impact on our electric bill. It turns out that if Sun converts those diesel generators to bio-diesel, it’s five times cleaner than the coal powers the company buys in Colorado. Of course, the first reaction Sun gets from the facilities staff is that the company can’t run the back-up generators unless it’s an emergency. Well, from my point of view, today’s energy costs constitute an emergency.

Worrall: What are some of the other out-of-the box ideas you have had to reduce power-related expenses?

Monroe: One idea is to make the CIO pay the electric bill as part of his or her budget. The datacenter is the biggest consumer of power in an enterprise. In fact, I’ve actually broken out the power consumption of the Broomfield datacenter compared to the rest of the building, and I found that each of the buildings on the Broomfield campus without the datacenter consume about 490 kilowatt-hours on a daily basis. The datacenter at Broomfield consumes about 1900 Kilowatt-hours per day. You as CIO don’t see any of that electrical expense, and so aren’t motivated to save any of it. So, one solution to reducing power consumption could be a simple accounting change — make the CIO responsible for datacenter energy costs.

Worrall: When organizations are designing datacenters, are there any process enhancements that can increase energy-related efficiencies?

Monroe: There is an idea known as retro-commissioning. Actually, it’s an idea that started in the green building industry with the Leadership in Energy and Environmental Design (LEED) benchmark. The idea is to treat a building similar to the way a ship is commissioned: Before someone breaks the champagne bottle and declares it seaworthy, it’s a wise idea to hire somebody to go through all the systems and make sure that everything is working the way it is supposed to operate. Studies have shown that this is a tremendously cost-effective thing to do. It costs something like 27 cents per square foot to perform the retro-commissioning and the payback is generally around 15 percent of the total energy consumption of the building. Furthermore, the investment usually pays for itself in about seven months.

Worrall: Sounds like a sound idea. Can you give an example of what such a process might uncover?

Monroe: Yes. I hear all the time is people are running the datacenter air conditioners to cool the space between 54 and 56 degrees, while the systems are actually specified to run at 72 degrees. The datacenter operators do that because that’s the only way that they have of dealing with hotspots — turning the whole room down to make up for over-heated systems. Instead, doesn’t it make more sense to be more thoughtful about the systems are placed? Instead of cooling the whole room, the staff should concentrate on eliminating the hotspots, and then raise the temperature back. Simply by moving servers around or performing spot cooling, enterprises might be able to save a ton of money.

Worrall: Great stuff. Finally, it would seem that some of Sun’s energy-efficient servers, like CoolThreads servers with multithreaded UltraSPARC T1 processors might also offer the potential for power savings.

Monroe: Here at Sun, Dean Nelson showed me some data that overall they have been able to reduce their systems footprint by 87 percent, while achieving a 75 percent reduction in power consumption reduction by using some of the newer Sun servers. At the same time, he has added 300 percent of the previous compute capacity. So that’s more than triple the compute power and one quarter of the electrical power.

Bob Worrall
CIO, Sun Microsystems, Inc.
cio@sun.com