Printable Version

Success Stories

EERE programs have a history of doing evaluation studies. This page highlights selected examples of evaluation successes in EERE.

Peer Reviews

The hydrogen program used the findings from a 2003 peer review to make decisions to discontinue project research for 5% of their project portfolio, in part based on performance assessment scores and review comments from external independent experts.

Project Number	Project, Performing Organization	Avg. Score	Continued	Discontinued	Project Completed	Summary Comment
10	Low Cost H2 Production Platform, Praxair	2.95	V			Emphasize collaboration.
11	Defect-free Thin Film Membranes for H2 Separation & Isolation, SNL	2.87	V
12	Maximizing Photosynthetic Efficiencies and H2 Production in Microalgal Cultures, UC Berkeley	3.33	V			Focus on program RD&D goals for 2005.
13	Reformer Model Development for Hydrogen Production, JPL	2.27		V		Model analysis in this area is no longer a program requirement.
14	Photoelectrochemical H2 Production, University of Hawaii	3.30	V			Emphasize further development of multi-junction photoelectrodes to meet program RD&D goals for 2005.
15	Photoelectrochemical Water Splitting, NREL	3.23	V			Focus on candidate lighting materials.
16	Encapsulated Metal Hydride for H2 Separation, SRTC	2.83			V
17	Economic Comparison of Renewable Sources for Vehicular Hydrogen in 2040, DTI	2.90			V
18	Biomass-Derived H2 from a Thermally Ballasted Gasifier, Iowa State University	2.70	V
20	Evaluation of Protected Metal Hydride Slurries in a H2 Mini-Grid, TIAX	3.20			V
22	Novel Compression and Fueling Apparatus to Meet Hydrogen Vehicle Range Requirements, Air Products & Chemicals Inc.	3.20	V
30	Techno-Economic Analysis of H2 Production by Gasification of Biomass, GTI	2.60			V	Project completed.
31	Supercritical Water Partial Oxidation, GA	2.57		V		Unlikely that cost barrier can be overcome.
32	Development of Efficient and Robust Algal Hydrogen Production Systems, ORNL	3.47	V			Focus on designing new DNA sequence coding for proton channel.
34	Water-Gas Shift Membrane Reactor Studies, University of Pittsburgh	2.90	V			Emphasize feasibility of high-temp water-gas shift under realistic operating conditions.
38	Low Cost, High Efficiency Reversible FC Systems, Technology Management, Inc.	2.80		V		High electrical input requirement prevents overcoming energy efficiency barrier.
39	High-Efficiency Steam Electrolyzer, LLNL	2.37		V		Carbon deposition at anode is a recurring problem.

This is an example where the recommendations of a panel of independent experts were taken into consideration to inform program decision making. [Source: FY2003 Hydrogen Program Merit Review & Peer Evaluation Report]

General Evaluation Studies

Performing program evaluations can have significant benefits for the program involved, as the following examples from past EERE experiences illustrate:

Market baseline assessment example:

• In addition to estimating the inventory of motor systems in the U.S., an EERE motor system market baseline assessment study determined that 71 percent of the total potential savings is system improvements. The study also found that the baseline level of knowledge and implementation of systematic approaches to motor systems energy efficiency was low. This was valuable information to help the program modify its design to better target identified opportunities and barriers in the motor systems market.

• The hydrogen program recently completed a 2004 hydrogen baseline knowledge assessment survey. Four populations targeted by the program for a hydrogen education campaign were surveyed. The survey data provided a baseline of current technical knowledge and opinion about hydrogen and fuel cells in the target populations. It found that "State and local official government scored much higher on the technical questions than other target groups."

Number of responses and average score for each of the four surveyed populations

Population	Average percent correct on technical questions
General public	32.8
Students	32.2
State & local government	67.5
Large-scale end-users	43.1

Also, "technical understanding appears to influence opinions about safety. For the General Public, Student, and Large-Scale End User Surveys, respondents with above-average scores on the eleven technical questions are more likely to have an opinion about hydrogen technology safety, and for those respondents who have an opinion, their opinion is more likely to be positive." "State and local officials also scored much higher on the technical questions." This baseline data is being used by the program to design an effective education program for the target audience, and as the basis for gauging progress in 2007 and 2010.

Process evaluation example:

• Several EERE evaluation studies included recommendations for the programs to develop activities & simple tools that will support end users' decision-making at various stages of project implementation: general plant profiling, screening for technology opportunities, and implementation of projects. These evaluations were conducted for the Industrial Assessment Center (IAC) Program, the Industry Best Practices (BP) Program, Building America, Rebuild America, FEMP, and the former Motor Challenge Program).

• The 2001 FEMP Evaluation study provided recommendations for the program to develop a strategy to target the sizable audience of interested non-participants, and to develop marketing strategies that emphasize and take advantage of customer referrals and networking. The program has since acted on these recommendations.

• Six past EERE evaluation studies contained a recommendation to develop program-tracking databases to routinely collect the types of data needed for improved day-to-day program coordination and management, and for future evaluation activities. Below are examples of two:

Motor Challenge Program Evaluation: "Program record keeping must be enhanced to enable managers and implementation staff to better characterize establishments quickly as to their function (end-user, vendor, utility, or trade association), industry, and size. This will aid in program marketing, client relations management, and evaluation."

Best Practices Program Evaluation: "Submit training attendee lists, track materials acquired via the Web, identify training sessions that also include [estimates of potential energy savings in specific facilities], assist with characterizing BP clients, and maintain consistent records."

All of these programs receiving this recommendation have taken steps to improve their program record keeping systems.

Outcomes/ impact evaluation example:

• It was determined from a statistical survey (the Energy Star Household Survey), stratified by publicity 'promotion' areas, that substantial fractions of consumers recognize and understand the Energy Star label. 41% households recognize ENERGY STAR label. A larger proportion of households in high- than in low-publicity areas recognize the ENERGY STAR label.

• An evaluation of the Industrial Assessment Centers (IAC) program quantified the substantial direct and indirect energy savings achieved through all relevant pathways. The study also found there were unaccountable benefits, identifying additional IAC energy savings of up to 25%.

• According to the 2001 FEMP Evaluations, FEMP ESPCs have influenced the awareness, decision, and implementation behaviors of program participants. "Since involvement with FEMP ESPCs,

- No participants remain unaware of performance contracts (unaware stage) and only 10 percent indicate that they have just become aware of performance contracts (awareness stage).

- FEMP ESPCs have moved about 41 percent of participants at least through the first two stages of the adoption cycle.

- 56 percent of participants are in either the implementation or confirmation stage, compared to 25 percent before hearing about FEMP ESPCs. An additional 31 percent of participants moved into either the implementation or confirmation stage."

Movement of FEMP ESPC participants and nonparticipants through the adoption cycle

Stage of adoption	Percent of participants (N=101)			Percent of aware nonparticipants (N=117)			(7) Percent of unaware nonparticipants (N=188)
	(1) Before hearing about FEMP	(2) Since involvement with FEMP	(3) Movement from (-) / to (+) stage	(4) Before hearing about FEMP	(5) Since hearing about FEMP	(6) Movement from (-) / to (+) stage
Unaware	24	0	-24	21	0	-21	63
Aware	27	10	-17	31	40	+9	24
Persuasion	12	7	-5	9	10	+1	5
Decision – no	10	7	-3	14	16	+2	1
Decision – yes	3	21	+18	8	9	+1	1
Implementation	7	24	+17	4	6	+2	4
Confirmation	18	32	+14	14	18	+4	2

 

• In 2005, FEMP completed the FEMP Peer Review of its program elements. Upon successful completion of the peer review OMB increased FEMP's PART score from 63.7% to 72.14%. This changed FEMP's overall PART performance rating from "Adequate" to "Moderately Effective."