Green Computing

Build a Green Datacenter

Dave Ohara


At a Glance:

  • Energy monitoring systems
  • Essential traits of a green datacenter
  • Effects of H.R. 5646
  • Best practices for getting started

Green is the hot topic these days, and the concept is having an impact on the way people think about datacenters. Companies around the world are announcing ways to save energy and reduce

costs by buying new hardware and services. Yet, there is little guidance on how you can take action to control energy costs. In the past, electricity has been treated as an overhead expense, like the cost of space. But with rising power costs and issues regarding reliability, supply, and capacity, electricity requires its own specific strategy.

Projects regarding performance optimization and cost reduction are a part of everyday best practices in nearly every area of business. So why not treat energy costs in the same way?

As IT pros, many of us make decisions about the configuration and setup of servers, the specifications on the equipment our organizations purchase, and the requirements for datacenter upgrades and construction. We even provide early design input during application development. When it comes to these projects, we obviously have a golden opportunity to be green and influence the energy efficiency of any datacenter.

The first part of any strategy is to know your current energy usage. You need to know where your energy is used and by what specific equipment, as well as what usage is efficient and what is wasteful in the datacenter. Unfortunately, it's rare to find power-consumption metering in place that can break down usage to a level where people can see the results of their actions. Most organizations typically only see a monthly power bill that rolls up consumption into an overall bottom line. This offers little incentive for saving energy since individuals never see the impact of their decisions, and there is no way for them to prove that their changes have actually saved energy.

One of the first issues people confront when considering a green datacenter initiative is whether they have executive support. For the purpose of the article, I am going to assume the answer is "not yet." Executive support requires a serious commitment that provides resources and budget for your initiative. And while there is a lot of talk about green datacenters, the reality is that there is still often a lack of serious support at the executive level. If you did already have such executive support, you would probably be running a green datacenter right now.

Still, even assuming you are not getting the support you need, there is a great deal you can do to push your green datacenter initiative forward. So how do you determine effective actions to take in achieving your goals? Fortunately, energy efficiency is not a new concept and there is a lot that IT pros can learn from other industries.

Summing Up Energy Consumption

One of the biggest challenges I find when writing on this topic is making sense of the wide variance of numbers provided by different sources. Some studies say that datacenters account for between 1.2 to 2.0 percent of the electricity consumed in the United States. By some estimates, if you were to view datacenters as an industry unto themselves, U.S. datacenters would be approaching the top five industries in terms of energy use. Meanwhile, electricity can account for up to 15 to 20 percent of the operating cost in a U.S. enterprise datacenter.

A study by Lawrence Berkeley National Laboratories for the American Council for an Energy-Efficient Economy determined that datacenters can be as high as 40 times more energy-intensive than conventional office buildings. The study goes into detail about what can be done to improve energy efficiency from a facilities perspective (see A good starting point is to determine to what extent your datacenter uses the best practices set out in this study.

No doubt, there are a lot of perspectives, a lot of studies, and a lot of numbers regarding power consumption. One theme that remains constant, however, is that there is a lot of waste. Just think about how many servers have been purchased with the intent of getting the most bang for the buck—this means buying as much CPU power, memory, storage, and redundant power supplies as possible while staying within a budget. A lot of those servers have been running at an average of 10 to 15 percent utilization for the past 3 to 5 years. While people often focus on CPUs, other components do play an important role here—memory, disks, and network interfaces are all significant power draws and, when combined, exceed the power consumption of the CPU.

Meanwhile, excess hardware capacity—in everything from processors to redundant power supplies—can lead to significant waste. Given the risk-averse culture in most IT departments, excess capacity is the norm. The safest decision is to have excess hardware capacity, even though it may not necessarily be the best financial decision. But while energy is a key factor, it is typically overlooked when calculating lifecycle costs. Unfortunately, there are no incentives to buy efficient solutions, as there is no cost benefit to decision makers after deployment.

There's a median to be reached, however. The pendulum swings the other way when executives come to realize how little of the datacenter hardware capacity is actually being used, and they look to server consolidation as a way to reduce costs. It is much easier to size servers appropriately up front than to attempt to consolidate after servers have already been deployed.

As the IT industry reaches a new level of maturity, optimization with cost reduction becomes a priority. Energy consumption is one indicator that executives can use to measure the effectiveness of decisions made regarding the lifecycle of IT hardware. AMD, Intel, and server and component OEMs are all making references to the performance-per-watt of their platforms. The performance-per-watt numbers can be used to calculate an operating cost by using the inverse formula: watts per performance unit. With the range of people involved in developing IT solutions—from facilities, application development, architects, operations, and business units—performance-per-watt answers the question of "What am I getting in return," and it lets you see how solutions compare in terms of energy efficiency.

Why is it a common practice to ignore energy costs and overlook decisions that affect operating and facilities costs? How many datacenters have you seen that are at capacity due to power and cooling constraints? There was a time when having enough physical space for the datacenter was the problem. Today, however, the problem is whether you have enough capacity to power and cool all the equipment—and how soon until you run out of this capacity.

The Importance of Energy Monitoring Systems

Precise energy consumption can be difficult to measure. As I mentioned earlier, most companies roll up energy costs as a whole rather than breaking down power consumption among specific units. This method of treating power as general overhead creates the perception that energy costs are beyond the control of the business unit. This, however, is not true.

A datacenter can use tens of times more power per square foot than standard office space, and therefore even minor efficiency improvements in datacenters can account for substantial savings. Not surprisingly, cost savings is the primary reason for datacenters implementing energy savings programs. The problem is that the lowest cost method is usually in place to charge for energy consumption and the monthly facilities costs for datacenter operations are rolled up. Costs are then allocated to groups based on their occupied space and/or by port connection—this approach gives a far less than accurate breakdown of the true costs to run a piece of datacenter equipment.

It's rare for an organization to have a metering solution in place that measures the actual power usage by individual datacenter devices—largely due to the fact that installing an energy metering and reporting system requires a significant upfront investment. However, investing in such a system can result in cost savings further down the line.

Of course, a power monitoring system on its own isn't enough—you'll need a plan for how you will use collected data to better manage energy consumption and ultimately see a return on your investment. Consider the Canadian company Abitibi Consolidated, the world's largest producer of newspaper and paper products. In 2001, when the Ontario electric power market was to be deregulated, Abitibi Consolidated was exposed to fluctuating market pricing and availability. To optimize its energy purchases, the company implemented a monitoring and metering system. The new system paid for itself within a few months, thanks to a 1 percent savings the company realized simply by buying their energy more efficiently.

How much energy do you need to save to justify an energy monitoring upgrade in your datacenter? Bill Laing, General Manager of Windows Server® 2008 development at Microsoft, said in a recent interview that the new Windows Server release with default energy savings enabled could reduce consumption by up to 20 percent, a significant savings. (Figure 1 shows some of the Power Options available in Windows Server 2008 beta 3.) Note, though, that the savings cannot be calculated as a direct reduction in the cost of power consumed. There are a number of other factors involved, such as the cost of building a new datacenter and the number of other systems currently supporting less efficient configurations.

Figure 1 Power Options in Windows Server 2008 beta 3

Figure 1** Power Options in Windows Server 2008 beta 3 **(Click the image for a larger view)

Right now, you could select the Server Balanced Processor Power and Performance option in Windows Server 2003 SP1. But without a metering system in place, you won't be able to quantify the savings, or any other factors involved in making and supporting this change.

It's also worth noting that for many datacenters, power consumption typically fluctuates only 4 percent during any 24-hour period, indicating that power consumption is not load dependent. With recent innovations by hardware vendors, this outcome is changing for new equipment and the amount of power that is being consumed decreases under lighter loads.

Defining the Green Datacenter

I could go on for pages listing the criteria or traits that make a datacenter green. Of course, I don't have room to cover everything in this article, but I want to highlight some key characteristics that you should keep in mind if you are serious about implementing a green datacenter. Even if you are only in the planning stages right now, you can measure against this list to determine where your datacenter currently stands:

  1. Meters are used to break down energy usage to the level of components (such as a 2U server, a 4U server, a switch, a SAN, and a UPS) and which business units are charged for the power being used by those components.
  2. Energy usage is continuously monitored to determine peak and low energy demands.
  3. Energy capacities are monitored on a total datacenter level all the way down to circuits to make sure all circuits are within acceptable limits.
  4. The energy savings plan is documented and rewarded.
  5. The energy savings plan is reviewed regularly and corrective action is taken to address failures.
  6. Determining how costs are charged back to business units is used to shape behavior, encouraging energy savings among independent business units. This point must be driven at the executive level.
  7. CPU throttling is enabled on the servers, and the performance lab measures the range of power consumed under a variety of loads.
  8. Thermal profiling is used to identify hot spots and overcooling.
  9. IT performance engineering includes energy efficiency measurements.
  10. Feedback of live data is available to individual organizations, allowing them to react appropriately. (California ISO goes as far as to provide real-time data to the public on its Web site, as shown in Figure 2.)

Figure 2 The California ISO Web site provides a live snapshot that includes current demand and current level of conservation being requested of its customers

Figure 2** The California ISO Web site provides a live snapshot that includes current demand and current level of conservation being requested of its customers **(Click the image for a larger view)

The more of these characteristics you have in place, the closer you are to successfully running an advanced energy monitoring system. I've included a scorecard (see Figure 3) with eight questions you can ask yourself to see the current level of your datacenter: basic, intermediate, or advanced.

Figure 3 How green is your datacenter?

Level Do you ...
Basic Review your monthly electricity bill?
Check your electrical equipment on a regular basis for capacity planning?
Intermediate Have automated metering and reporting of load and consumption for electrical circuits?
Review equipment energy efficiency when purchasing?
Advanced Monitor energy consumption per rack, device, and circuit?
Analyze historical data monitoring and reporting?
Use real-time reporting of energy consumption for departments to understand impacts of changes and deployments?
Use data to negotiate favorable power rates with the utility company?

Knowing Is Half the Battle

Another problem you'll face is standardization throughout the organization. Making a change such as I've described in this article—one that changes the paradigm by which energy is measured, and one that involves participation by business units throughout the organization—involves alignment between facilities groups across multiple datacenters working with a broader set of IT groups. Such organizational issues are the top blockers to putting an energy savings program in place.

It's important to make sure you have a single person or team responsible for understanding power consumption across datacenters. This center of knowledge (which most companies lack when it comes to power consumption) is essential. It's also important that your tools collect data from the disparate facilities in a uniform manner and report the results back to each business unit in a consistent way (Figure 4 shows a sample dashboard for reporting data).

Figure 4 Sample global data monitoring dashboard (courtesy of OSIsoft)

Figure 4** Sample global data monitoring dashboard (courtesy of OSIsoft) **(Click the image for a larger view)

Without the right instruments, you won't know whether you are really operating efficiently—nor will the individual business units know whether their efforts are producing results. If you really want to change human behavior, you need to give users the information in real time with historical data trends so they can visualize the results of their actions. You can't effectively reward groups for achievement in energy efficiency if each business unit is on a different system, using a different set of tools, and different measurements for reporting their success.

Industry Forces

Don't feel alone in your efforts to make your company greener. Leaders in other markets—from companies like Boeing and Toyota to government agencies such as the State of California—are making serious investments in making their operations and products more energy efficient.

As for the world of IT, in 2006 the U.S. House of Representatives passed a bill—H.R. 5646, now public law 109-431—calling upon the Environmental Protection Agency (EPA) to investigate energy costs and electricity consumption by servers and datacenters and the potential effects of moving to more energy efficient systems. The study, which was carried out by ENERGY STAR, an EPA program, projected near-term growth in energy use of U.S. computer servers and datacenters, assessed potential cost and energy savings related to computer server and datacenter energy efficiency improvements, and assessed potential incentives and voluntary programs for promoting energy-efficient computer servers and datacenters. The final results of the study were presented to Congress in August 2007. You can find more information about this study at

New building standards for green buildings are also being developed. These standards call for such criteria as hourly sub-metering capabilities for circuits depending on purpose and energy consumption, building meters that communicate energy consumption data to a meter data management system, and meter data management systems that store data and create user reports showing calculated hourly, daily, monthly, and annual energy consumption for each meter. Many of these criteria are required for compliance systems if you sign up for energy savings programs where you are required to prove your changes have worked (for instance, renewable energy credits, carbon credits, and energy reduction programs).

Some users have asked for specification testing, but it is difficult to represent the spectrum of uses. Attempts are being made to standardize tests, but these tests—like most other performance tests—can be used to optimize the "perceived" results, which do not represent actual usage. Consider miles per gallon (MPG), for example. The EPA ratings estimate the miles per gallon a typical driver should get under typical city and highway conditions. However, most drivers and driving environments aren't typical and the factors that affect fuel efficiency can vary significantly. This same guidance applies to energy savings in the datacenter. The only accurate results for your organization will come from measuring energy savings in your own environment.

Where Are the Savings?

There are many factors to consider when determining savings. Do you know the cost of adding more power in a given area? Do you have a strategy to minimize power constraints? Can you solve the problem by locating your datacenters somewhere that power is cheaper? Or should you focus on improving operations in your existing facilities?

For some companies, the best way to reduce costs may be to relocate the datacenters to areas with cheaper electricity sources. Server virtualization offers another approach to reducing energy consumption. In fact, some power companies, (PG&E, SoCal Edison, and SDG&E, to name a few) are offering financial incentives to customers that undertake server virtualization projects that result in a reduction of server equipment.

But there are simpler, less drastic steps you can take. Most IT departments currently have no idea how much power their systems consume, and this means there is enormous opportunity for potential savings. A first step could be to look at the energy consumption associated with individual devices and then see if there are any changes you could make on a per-device basis. Many small changes can add up to substantial savings.

For comparison, think of the typical household where the owner has no idea about how much energy each item in their house uses—they just pay a total monthly bill. You can get a simple inexpensive device, such as a Kill A Watt, that measures the energy consumption of individual devices. How much do you spend to keep your microwave powered when you're not using it? How much power is consumed when leaving your phone charger plugged in? Taking this same approach, you can add power meters to racks and collect data from various devices. Management tools can be used to collect data directly from some servers with energy consumption SNMP traps.

Getting Started

Figure 5 provides an outline of best practices that you should use to get your plan underway. The important thing to understand at this stage is that you're not limited to choosing from the two extremes. You can opt for a more balanced approach, which is probably more realistic. Start a grass-roots effort, identifying those individuals within your company who are passionate about energy savings. Start measuring some devices within a business unit to provide visibility. Getting some momentum and showing quantifiable results will help you get the backing you need to expand the company's investment in going green.

Figure 5 Keys to putting an energy savings plan in place

Collect data manually to determine the state of your current systems. Determine how much power is consumed by devices and applications, and break down the costs.
Create a closed loop feedback system for measuring the effectiveness of changes made.
Be sure you monitor electricity consumption in the right places and to the appropriate level (by device, IT organization, business unit, application, and so on).
Create a fund for energy savings projects to provide capital funding.
Find out who within your organization can help drive energy efficiency and get them involved.

Who are the people who will drive energy efficiency in your company? Are they the top-level executives, middle management, or individuals who simply care about the impact your company has on the environment. There are passionate individuals at every company. One of your first decisions is whether to try the grass-roots method and involve these people from the start.

Ultimately, for whichever direction you choose, planning an energy efficiency program for your datacenter will require collaboration across groups in IT. Until recently, the typical approach to planning IT solutions has been to ignore power costs early on during the design phase, focusing on the hardware and software being purchased, along with the labor and hosting costs of the solution. When power is buried in the overhead cost of running solutions in a datacenter, energy efficiency is a low priority. Exposing the actual power being consumed by solutions is the first critical step in changing the behavior of your organization.

Dave Ohara has 26 years of experience in technology, working for such companies as Microsoft, Apple, and Hewlett-Packard. Working in product development, marketing, technical evangelism, infrastructure architecture, process engineering, reliability engineering, and distribution logistics, Dave is now working with multiple companies implementing green initiatives.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.