U.S. Department of Energy - Energy Efficiency and Renewable Energy

Weatherization & Intergovernmental Program

Conservation Update: A Vision for Energy Efficiency at Data Centers

This article was featured in the September-October 2008 edition of the State Energy Program's bimonthly newsletter, Conservation Update.

by Allan Chen, Lawrence Berkeley National Laboratory

Photo of a man in a white shirt adjusting controls on a wall of computer modules in the NREL data center.

A 2006 report to Congress said that data centers used about 1.5% of total U.S. electricity consumption, worth about $4.5 billion. Without taking measures to improve energy efficiency, data center energy use could double by 2011.
Credit: Pat Corkery

Data centers are truly the backbone of the global economy. Although these facilities are unseen by most of us, their uninterrupted performance affects everyone who participates in the economy, from the individual using an Internet search engine, to companies doing business with one another across the globe.

So indispensable are data centers in this information-hungry world that their numbers have been growing at a breakneck pace—and along with it their energy use. A special report to Congress released in August 2007 noted that data center energy use in the U.S. doubled between 2001 and 2006.

Report to Congress on Server and Data Center Energy Efficiency

Prepared at the request of Congress by the U.S. Environmental Protection Agency (EPA), much of the report's technical content was contributed by researchers from the Department of Energy's Lawrence Berkeley National Laboratory (Berkeley Lab). Congress's request for the report reflected the widespread recognition in the information technology industry and the larger business community that the rapid growth of data center energy use and costs had become a significant national challenge.

Pie chart showing the percent of electricity consumption by component for typical data servers: site infrastructure: 50%; network equipment, 5%; storage, 5%; high-end servers, 2%; mid-range servers, 4%; volume servers, 34%.

Source: Report to Congress on Server and Data Center Energy Efficiency, Public Law 109-431, U.S. Environmental Protection Agency and ENERGY STAR® Program, August 2, 2007

The report contained a number of recommendations for industry as well as federal and state governments on how to encourage the development of more efficient servers and data centers, including public-private partnerships, financial incentives, recognition programs, and cooperative efforts with utilities.

As the information technology (IT) industry began mobilizing for solutions, it drew on experts from around the nation, including Berkeley Lab's Environmental Energy Technologies Division (EETD), where a research program was already under way to study data center energy use and develop energy-efficient solutions and related support to the ENERGY STAR program for IT equipment.

Data center research at Berkeley Lab is a multi-track program involving numerous government and private sponsors, and research partners. Its goal is to develop solutions to cost-effectively reduce data center energy use, and to work with the private sector to get those solutions into the marketplace and common use.

The program has produced results and practical information, including guides for energy-efficient design and operational practice. (See the Data Centers page on the Berkeley Lab's High-Performance Buildings for High-Tech Industries Web site.)

Funding agencies include the U.S. Department of Energy Industrial Technology Program and Federal Energy Management Program, the EPA, the California Energy Commission's Public Interest Energy Research program, the New York State Energy Research and Development Authority, Pacific Gas & Electric, the General Services Administration, and the University of California.

Doubling of energy use in five years

Chart showing energy use from 2000 to 2006 using historical data, and from 2006 to 2011 using projections under different scenarios. The following scenarios and corresponding energy use projections are illustrated: Historical trends scenario -- rises from 60 billion kilowatt-hours per year in 2006 to more than 120 billion kilowatt-hours per year by 2011; Current efficiency trends scenario – rises to nearly 110 billion kilowatt-hours per year by 2011; Improved operation scenario – rises to more than 80 billion kilowatt-hours per year by 2011; Best practices scenario – decreases to around 45 billion kilowatt-hours per year by 2007 and stays close to that rate through 2011; State-of-the-art scenario – decreases to about 33 billion kilowatt-hours per year by 2011.

Source: Report to Congress on Server and Data Center Energy Efficiency, Public Law 109-431, U.S. Environmental Protection Agency and ENERGY STAR Program, August 2, 2007

In 2006, according to the report to Congress, data centers used about 1.5% of total U.S. electricity consumption, worth about $4.5 billion. This is as much electricity as used by 5.8 million average U.S. households—5% of the housing stock. Supplying this electricity requires about 15 base load power plants. The report estimated that without taking measures to improve energy efficiency, data center energy use would be on track to nearly double again by 2011, requiring the equivalent of 10 more plants.

Berkeley Lab's research program takes a number of approaches to addressing the data center challenge.

Understanding the dimensions of the problem. Researchers measured energy use of real data centers (more than 25 case studies) and developed benchmarking procedures—necessary to get a clear understanding of how much energy different centers were using, and allowing for comparison of different facilities that might have very different characteristics, such as floor space, number of servers, building HVAC systems, redundancy, etc.

Technical advice for better data center operation. The benchmarking research provided basic knowledge that was used to develop best practices guidance for data center design and operation, as well as a self-benchmarking procedure that data center operators can use to measure the performance of their center, and identify measures to improve efficiency. This guide focuses on how to improve the building systems to improve energy efficiency in the data center.

Equipment studies. Studies of the variation in the energy use of IT equipment as a function of computing load provided technical input to the efforts of the EPA to develop performance metrics and measurement protocols for server benchmarking. Researchers also undertook studies of power supply efficiency in IT equipment.

Data center design. Improving the design of data centers can provide energy efficiency improvements beyond applying best operational practices. Berkeley Lab and its partners conducted a unique project to study the energy savings possible by using a direct current (DC) power, architecture-based data center. This project, which received substantial industry and media attention, tested a data center power distribution scheme that eliminated some of the power conversions between the grid and the end uses in server equipment. Sun Microsystems provided one of its Silicon Valley facilities for the test. Intel and Sun Microsystems provided prototypical servers with power supplies modified to accept DC and eliminate a power conversion in the server. More than 25 other firms participated by providing equipment and services. (See the DC Power for Data Centers of the Future Web page on Berkeley Lab's High-Performance Buildings for High-Tech Industries Web site.)

Uninterruptible power supplies. The Electric Power Research Institute and Ecos Consulting, program contractors, studied efficiency of uninterruptible power supplies (UPS) and how improvements in this technology could help raise overall data center efficiency.

Practicing what we teach

A significant goal of the program is to practice energy efficient design and operation in Berkeley Lab's own facilities. The laboratory is the home of the National Energy Research Scientific Computing (NERSC) facility, and other research and business computing facilities.

EETD researchers are working with NERSC and Berkeley Lab facilities staff to design a new supercomputer facility for Berkeley Lab. The Computational Research and Theory facility will handle state-of-the-art computing while being a model of energy efficiency. EETD researchers are also working with University of California (UC) staff to bring efficient technology to the UC campuses.

Finally, researchers in EETD and the Lab's Computing Research Division are also working with the computer industry on research to improve the efficiency of computing technology and digital networks. Many computers have energy-saving features that are disabled because these personal computers (PCs) need to be active to stay connected with the Internet. The researchers are developing new technologies and industry standards to allow PC energy-saving features to work while remaining Internet-connected.

Support for EPA's ENERGY STAR programs

EETD researchers are helping to develop criteria for an ENERGY STAR specification for enterprise servers, as well as criteria for ENERGY STAR for Buildings to use in rating and labeling data centers.

Information to guide data center energy efficiency efforts

All of the research produced by the data center program is available at the Web site addresses listed in this article. Data center designers, equipment manufacturers, and public officials can all find useful information in these reports.

There are also a number of national efforts underway to propagate technological solutions. Berkeley Lab is assisting the Department of Energy Office of Energy Efficiency and Renewable Energy under the Save Energy Now program by developing web-based assessment tools, awareness training for data center professionals, best practices information, and demonstration projects. In addition, a national research agenda is being developed. Berkeley Lab is assisting DOE's Federal Energy Management Program in providing Save Energy Now tools, training, and technical assistance to federal data center managers.

Research is continuing in such areas as optimizing power distribution, advanced cooling technology, and improved IT hardware and software. EETD continues to work with IT manufacturers to demonstrate advanced technology for data centers.

Rapid mobilization is paying off—there are energy-efficient solutions for data centers out there, and thanks to this broad partnership of researchers, private industry, and public agencies, those solutions are making their way into the marketplace.

The following are links that provide access to supporting information.

Direct Current-Powered Data Centers—A State-Funded Research and Development Demonstration

Photo of an aerial view of a large white room with two rows of tall black box-like structures that house computer equipment in the LBNL data center.

Computing clusters at Berkeley Lab's National Energy Research Scientific Computing Center. Energy efficiency researchers work with scientists at this center to develop energy efficient technologies and operational practices.
Credit: LBNL

One little-used approach to reducing the energy use of data centers is to eliminate power conversion losses by using direct current (DC) instead of alternating current (AC) power throughout the data center. Researchers in Berkeley Lab's Environmental Energy Technologies Division (EETD) proposed this technology demonstration and the California Energy Commission's Public Interest Energy Research (PIER) program sponsored the work. Data centers are a matter of great interest in the state because of the presence of large data center-using corporations such as Google and Yahoo, and because the manufacture of information technology for data centers is such a significant part of California's economy.

Berkeley Lab researchers teamed with Silicon Valley giants, including Sun Microsystems and Intel, and more than 25 other firms to demonstrate technologies that could save billions of dollars a year in the energy costs of operating data centers, as well as improve reliability and lengthen equipment life. The demonstration took place during the summer of 2006 at a test facility at Sun Microsystems in Newark, California.

The Berkeley Lab team, which consisted of project leader William Tschudi, Steve Greenberg, and Evan Mills, conceived the project and provided oversight for the demonstration's planning and design, which was executed by private-sector firms ECOS Consulting and EPRI Solutions under a contract with Berkeley Lab. The numerous partner companies provided technical advice, equipment, and staff to set up the demonstration facility.

As mentioned before, DC power offers an opportunity for improving energy efficiency. In the typical data center, the power distribution system provides 480-volt AC power through redundant uninterruptible power supplies (UPSs) to a transformer, which then steps it down to a lower AC voltage, such as 208-volt, which feeds racks of servers. Then, individual power supplies (typically also redundant) within each server convert this power into a voltage appropriate for further conversions for the various internal components. The power conversion steps are often inefficient, generating substantial heat, which the room's air conditioning system must remove at great expense.

However, substituting DC power in data centers as a replacement for conventional AC power has not made significant inroads into many data centers because the technology is unfamiliar to many facilities engineers. There is reluctance within the industry to switch to new technologies without field experience showing that the switch could be done safely and would have operational and economic benefits, without causing unanticipated problems. This was why the California Energy Commission stepped in with funding for the research—the existence of a thoroughly tested DC powered data center could provide a case study for IT manufacturers to develop new technology, and data center managers with a new approach to reducing the energy consumption of their centers.

Some servers on the market can run on DC power—typically at 48 volts DC, which is the standard in the telecommunications industry. The Berkeley Lab team's demonstration utilized a higher DC voltage and showed how a DC-powered data center could skip multiple conversions in the power distribution chain:

  1. From AC to DC and back to AC in the UPS system
  2. From higher voltage AC (480V) to lower voltage AC (208V)
  3. From AC to DC in the server—from 208V AC to 380V DC.

In this way, one conversion from 480V AC was made producing 380V DC power, which was then fed directly into circuits that were already designed for that voltage inside the servers. (AC power supplies in servers today convert to 380V DC.)

The demonstration met all of its objectives successfully. It showed that DC-powered servers, in the same form factor as AC-powered units, exist in the marketplace and could be used in data centers, and they provide the same level of computing performance and functionality as AC-powered units.

The project reported a measured energy efficiency improvement of 5 to 7 percent over the measured AC powered distribution systems, without optimizing any of the equipment. This figure does not include additional gains in efficiency that would be possible by reducing the cooling load. With optimization, DC-powered data centers could be 20% more efficient than traditional data centers.

The project team held a number of open houses throughout the period of operation of the DC data center demonstration. More than 200 people from corporations and government agencies toured the facility, and media coverage helped disseminate the results of the research.

For more information, use the following links:

Cooperation Between Federal Government and Industry Creates New Solutions

Save Energy Now logo

The U.S. Department of Energy's Industrial Technologies Program (ITP), through its Save Energy Now program, is working with the private sector and other government agencies to find new technological solutions, and offer a variety of tools to help data center operators reduce their energy consumption.

The Save Energy Now program offers the Data Center Energy Profiler (DC Pro) which is an online software tool designed by Berkeley Lab in collaboration with industry experts. DC Pro helps data centers worldwide to quickly gauge their energy use and identify potential steps for saving energy and money. Entering information about a data center into DC Pro results in an energy profile and provides an overview of energy use, savings potential, and a list of specific actions to realize savings.

Other tools offered by the Save Energy Now program help operators with energy saving and serve to advance the state of the art. Through the program, experts perform pilot energy savings assessments to refine software tools, and implement short-term strategies to save energy. Another program thrust is in training data center designers in energy management best practices and tools.

Program managers are also working to develop other information resources, including Save Energy Now case studies based on pilot energy assessments, a best practices training curriculum (developed in collaboration with ASHRAE and Green Grid), and guidelines for "Best-in-Class" data centers validated through technology demonstrations. By 2011, the Save Energy Now program aims to have trained up to 500 qualified specialists to assist companies in designing energy efficient data centers.

In addition, the EPA's ENERGY STAR® for Buildings program is developing an energy performance rating for data center infrastructure, to be added to the ENERGY STAR Portfolio Manager tool. The ENERGY STAR Program intends to recognize the top quartile performers by awarding an ENERGY STAR label.

In addition to its cooperation with ITP's Save Energy Now Program, the EPA's ENERGY STAR program is developing an ENERGY STAR specification for enterprise servers, which will allow this equipment to earn the ENERGY STAR label. Products with this designation are among the most energy efficient products on the market. EPA is also looking at opportunities to develop ENERGY STAR specifications for other IT equipment, possibly including data storage and network equipment.

The Save Energy Now Program and the EPA are working with researchers at the Berkeley Lab and industry representatives to develop consensus on energy efficiency metrics and benchmarking protocols for data centers. Currently, no consistent, universally agreed upon metrics exist to define the energy performance of a data center. This hampers the industry's efforts to improve performance and develop new technology.

With the EPA, ITP is managing a joint national data center energy efficiency information program, whose purpose is to coordinate activities of the ITP Save Energy Now initiative, the Federal Energy Management Program (FEMP), and the EPA ENERGY STAR program.

Industry groups participating in this effort include 7 x 24 Exchange, AFCOM, American Society of Heating, Refrigeration and Air Conditioning Engineers (ASHRAE), Critical Facilities Roundtable, Information Technology Industry Council (ITIC), Silicon Valley Leadership Group, The Green Grid Association, and The Uptime Institute.

FEMP is also working to ensure that best practices are implemented at federal data centers. FEMP program managers are adopting pilot Best-in-Class guidelines for federal datacenters, and when the industry develops Best-in-Class standards, these will be applied at new federal data centers.

For more information, access the following links:

About the Author

Allan Chen, Lawrence Berkeley National Laboratory

Allan Chen serves as Leader of the Communications Office in the Environmental Energy Technologies Division at Lawrence Berkeley National Laboratory, in Berkeley, California.

He oversees a staff of five, including a Web master, systems administrator, and graphic designers. The office provides communications services to the Environmental Energy Technologies Division of Lawrence Berkeley National Laboratory, which conducts research on energy efficient technology and environmental impacts of energy generation. He also serves as editor of its newsletter, EETD News, and media relations specialist for this division.

Mr. Chen has a B.S. and an M.S., both in geophysics, from Stanford University, and has worked for Smithsonian and Discover magazines as a writer and editor, and for Intel Corporation as a communications manager.