Program Evaluation: Examples of Results

EERE offices have a history of doing evaluation studies. This page highlights selected examples of evaluation results from EERE reviews and studies. Eleven examples are provided for the following three types of evaluations.

  • In-progress peer review example
    • Example 1: Hydrogen program peer review
  • Baseline assessment example
    • Example 2: Hydrogen program completed a 2004 hydrogen baseline knowledge assessment
  • Outcomes/ impact evaluation examples
    • Example 3: Energy savings impact
    • Example 4: Fuel savings impact
    • Example 5: Economic return on investment
    • Example 6: R&D knowledge diffusion impact
    • Example 7: R&D knowledge diffusion impact
    • Example 8: Consumer awareness impact
    • Example 9: Direct and indirect energy savings
    • Example 10: Impact on stages of diffusion behavior
    • Example 11: R&D acceleration effect

In-Progress Peer Review Example

  • Example 1: Hydrogen program peer review. This is an example where the recommendations of a panel of independent experts were taken into consideration to inform program decision making. The hydrogen program used the findings from a 2003 peer review to make decisions to discontinue project research for 5% of their project portfolio, in part based on performance assessment scores and review comments from external independent experts. Below is a selected section of a review results table extracted from the FY2003 Hydrogen Program Peer Evaluation Report.

    Programs are able to manage public resources efficiently and effectively whenever poor-performing projects are discontinued or others are modified for improvement, based on peer review findings.

Review Results Table

 

Project Number

Project, Performing Organization

Avg. Score

Continued

Discontinued

Project Completed

Summary Comment

10 Low Cost H2 Production Platform, Praxair 2.95 V     Emphasize collaboration.
11 Defect-free Thin Film Membranes for H2 Separation & Isolation, SNL 2.87 V      
12 Maximizing Photosynthetic Efficiencies and H2 Production in Microalgal Cultures, UC Berkeley 3.33 V     Focus on program RD&D goals for 2005.
13 Reformer Model Development for Hydrogen Production, JPL 2.27   V   Model analysis in this area is no longer a program requirement.
14 Photoelectrochemical H2 Production, University of Hawaii 3.30 V     Emphasize further development of multi-junction photoelectrodes to meet program RD&D goals for 2005.
15 Photoelectrochemical Water Splitting, NREL 3.23 V     Focus on candidate lighting materials.
16 Encapsulated Metal Hydride for H2 Separation, SRTC 2.83     V  
17 Economic Comparison of Renewable Sources for Vehicular Hydrogen in 2040, DTI 2.90     V  
18 Biomass-Derived H2 from a Thermally Ballasted Gasifier, Iowa State University 2.70 V      
20 Evaluation of Protected Metal Hydride Slurries in a H2 Mini-Grid, TIAX 3.20     V  
22 Novel Compression and Fueling Apparatus to Meet Hydrogen Vehicle Range Requirements, Air Products & Chemicals Inc. 3.20 V      
30 Techno-Economic Analysis of H2 Production by Gasification of Biomass, GTI 2.60     V Project completed.
31 Supercritical Water Partial Oxidation, GA 2.57   V   Unlikely that cost barrier can be overcome.
32 Development of Efficient and Robust Algal Hydrogen Production Systems, ORNL 3.47 V     Focus on designing new DNA sequence coding for proton channel.
34 Water-Gas Shift Membrane Reactor Studies, University of Pittsburgh 2.90 V     Emphasize feasibility of high-temp water-gas shift under realistic operating conditions.
38 Low Cost, High Efficiency Reversible FC Systems, Technology Management, Inc. 2.80   V   High electrical input requirement prevents overcoming energy efficiency barrier.
39 High-Efficiency Steam Electrolyzer, LLNL 2.37   V   Carbon deposition at anode is a recurring problem.

Back to Top

Baseline Assessment Example

  • Example 2: Hydrogen program completed a 2004 hydrogen baseline knowledge assessment. The hydrogen program completed a 2004 hydrogen baseline knowledge assessment survey. Four populations targeted by the program for a hydrogen education campaign were surveyed. The survey data provided a baseline of current technical knowledge and opinion about hydrogen and fuel cells in the target populations. It found that "State and local official government scored much higher on the technical questions than other target groups."

Number of responses and average score for each of the four surveyed populations

Population

Average percent correct on technical questions

General public 32.8
Students 32.2
State & local government 67.5
Large-scale end-users 43.1

Also, "technical understanding appears to influence opinions about safety. For the General Public, Student, and Large-Scale End User Surveys, respondents with above-average scores on the 11 technical questions are more likely to have an opinion about hydrogen technology safety, and for those respondents who have an opinion, their opinion is more likely to be positive." The survey also found that "State and local officials also scored much higher on the technical questions." This baseline data was used by the program to design an education program for the target audiences.

Back to Top

Outcomes/ Impact Evaluation Examples

  • Example 3: Energy savings impact. In a retrospective economic benefit-cost study for the Wind Program, the energy benefits (i.e., wind power generation) were calculated as part of the process for determining the economic impacts. In the wind study, it was determined there was a six year acceleration effect, as determined from interviews with experts. Six years was the mean value of the distribution of responses. This acceleration led to wind energy supplying 139.8 billion kWh that displaced electricity otherwise supplied by a fossil-fuel generation mix (where the mix is determined from the weighed fuel use of energy generation from fossil fuel mix of the States with the largest share of wind energy power):
    • 61.5 billion kWh (i.e., 44% of total from wind) supplied by coal-fired generation
    • 76.9 billion kWh (i.e., 55% of total from wind) supplied by natural gas-fired generation
    • 1.4 billion kWh (i.e., 1% of total from wind) supplied by petroleum-fired generation

    The energy benefits estimate provided the data input needed to perform the economic benefits calculations. It also produced the inputs to the subsequent environmental emissions analysis for that benefit-cost study
  • Example 4: Fuel savings impact. In an Advanced Combustion Engine (ACE) R&D Benefit-Cost Study, a statistical approach was adopted for the calculation of the fuel savings associated with miles per gallon (MPG) fuel economy improvements, where MPG improvements were linked to a 4.5% DOE-supported R&D improvement in Brake Thermal Efficiency (BTE) – shown in the figure below (an expert-derived counterfactual BTE). The change in MPG was statistically estimated (∂MPG / ∂BTE = 0.153) and the reduction in MPG absent the ACE R&D research was calculated. The calculated reduction in MPG was translated to reduced fuel consumption, where 17.6 billion gallons of diesel fuel was saved over the period 1995 to 2007. This information then provided the energy benefits estimate needed to perform the economic benefits estimates for the net benefit-cost analysis (see figure below). Environmental and energy security impacts were calculated from the energy savings results, as well.
A graphic showing the trend in BTE over time with counterfactual. Along the left side of the graphic is brake thermal efficiency from 0.3 up to 0.46. Along the bottom are the years 1960 through 2010. BTE Absent ACE R&D Research on Diagnostics and Modeling runs from 0.34 in 1960, rising steadily to 1994, then taking some drops and rises, and ending at about 0.4 in 2007. The BTE is 0.41 in 1993, rises then dips and ends at about 0.42 in 2007.
  • Example 5: Economic return on investment. In the same study mentioned in the previous example, after the full analysis was completed to determine attributed economic return on investment results, the 17.6 billion gallons of diesel fuel saved was found to correspond to net economic benefits of $69 billion ($2008, undiscounted). The internal rate of return was 63%. (See table and figure below.)
Two graphics: one showing VTP combustion for program or subprogram, and the other showing Vehicles Technologies Advanced Combustion Engine Research Subprogram Net Economic Benefits (undiscounted). For the graphic showing VTP combustion: Total program budget is $5.8 billion, program budget for technology portfolios examined in ROI studies is $0.93 billion, total attributed benefits is $70 billion, net benefits is $69 billion, NPV of net benefits at 3 percent of discount rate is $42.6 billion, NPV of net benefits at 7 percent of discount rate is $23.1 billion, benefit-cost ratio at 3 percent discount rate is 66 to 1, benefit-cost ratio at 7 percent discount rate is 53 to 1, and internal rate of return is 63 percent. For the Vehicles Technologies Advanced Combustion Engine Research Subprogram Net Economic Benefits (undiscounted) chart on the right, the net benefits in billions range from -$10 to $80 along the left, and the years along the bottom run from 1976 to 2008. The number begins rising at 1994 and ends up at about $70 billion in 2008.
  • Example 6: R&D knowledge diffusion impact. Bibliometrics analysis has been used to test and quantify the strength of linkages of program-attributed patents to downstream research and commercial applications. Bibliometrics analysis provides a quantification of knowledge diffusion benefits, as measured through various metrics, such as patent citation rates, citation rank among leading research organizations in number of patents linked to earlier funded patents, citation indices, among others. Bibliometrics analysis also serves to document the links between R&D and commercial applications in the market to reinforce the attribution elements of impact assessment for R&D evaluation studies.

    For example, the figure below shows the influence of EERE's solar PV research reaches beyond U.S. leading PV producers to extend to leading companies in the semiconductor industry, as well.
A graphic showing organizations from all industry areas with the largest number of patent families from all technologies linked to earlier DOE-attributed PV patents. The number of linked families runs along the bottom, from 0 to 700. Along the left side of the graphic is the list of organizations from highest number of linked families to lowest: Micron Technology, Semiconductor Energy Lab, Canon, Applied Materials, IBM, Sharp, General Electric, Toshiba, Advanced Micro Devices, Hitachi, Texas Instruments, Motorola, Samsung, Sanyo, NEC, Panasonic.
  • Example 7: R&D knowledge diffusion impact. Another example is for the Geothermal Technologies Office. In comparing the Influence of DOE-funded geothermal research with that of other organizations on subsequent developments in geothermal energy, Chevron and DOE are at the head of the figure below. Of the 1,016 total geothermal patent families* identified, 253 (24.9%) are linked to Chevron's earlier geothermal patents. Meanwhile, 209 geothermal patent families (20.6%) are linked to earlier DOE-attributed geothermal patents.
A graphic showing organizations whose geothermal patent families are linked to the most subsequent geothermal patent families. The list from the top is Chevron with the highest number, then in descending order of number of linked patent families: U.S. Dept. of Energy, Magma Power, Dow Chemical, Occidental Petroleum, Ormat, Bechtel, Deuterium, Unisys, Kyoto Central, Nalco, Env. Impact Research, Envirotech, Geothermal Investment, Pacific Gas & Electric, Southern Pacific Land, United Technologies, Geosystems Inc.
  • Example 8: Consumer awareness impact. It was determined from a statistical survey (the ENERGY STAR Household Survey), stratified by publicity 'promotion' areas, that substantial fractions of consumers recognize and understand the Energy Star label. Forty-one percent (41%) of households recognize the ENERGY STAR label. A larger proportion of households in high- than in low-publicity areas recognize the ENERGY STAR label.
  • Example 9: Direct and indirect energy savings. An evaluation of the Industrial Assessment Centers (IAC) program quantified the substantial direct and indirect energy savings achieved through all relevant pathways. The study also found there were unaccountable benefits, identifying additional IAC energy savings of up to 25%.
Vertical bar chart showing the following percent energy savings achieved through relevant pathways:  17.4% by Spin Off; 3.8% by Internal Replication; 3.7% by External Replication; 17.9% by Delayed Implementation; -18.0% by Decommissioned or Unimplemented, and 25.0% Total.

 

  • Example 10: Impact on stages of diffusion behavior. According to the 2001 FEMP Evaluations, FEMP ESPCs have influenced the awareness, decision, and implementation behaviors of program participants. "Since involvement with FEMP ESPCs:
    • No participants remain unaware of performance contracts (unaware stage) and only 10 percent indicate that they have just become aware of performance contracts (awareness stage);
    • FEMP ESPCs have moved about 41 percent of participants at least through the first two stages of the adoption cycle; and
    • 56 percent of participants are in either the implementation or confirmation stage, compared to 25 percent before hearing about FEMP ESPCs. An additional 31 percent of participants moved into either the implementation or confirmation stage."
Movement of FEMP ESPC participants and nonparticipants through the adoption cycle

Stage of adoption

Percent of participants (N=101)

Percent of aware nonparticipants (N=117)

(7) Percent of unaware nonparticipants (N=188)

(1) Before hearing about FEMP

(2) Since involvement with FEMP

(3) Movement from (-) / to (+) stage

(4) Before hearing about FEMP

(5) Since hearing about FEMP

(6) Movement from (-) / to (+) stage

Unaware 24 0 -24 21 0 -21 63
Aware 27 10 -17 31 40 +9 24
Persuasion 12 7 -5 9 10 +1 5
Decision – no 10 7 -3 14 16 +2 1
Decision – yes 3 21 +18 8 9 +1 1
Implementation 7 24 +17 4 6 +2 4
Confirmation 18 32 +14 14 18 +4 2
  • Example 11: R&D acceleration effect. Since 1975, investments made by the DOE, along with direct DOE-industry partnerships, have made significant contributions to the development of PV technology and markets. These investments were made largely under four major program efforts―Flat-Plate Solar Array Project (FSA) (1975–1985), PV Manufacturing Technology Project (1991–2008), Thin-Film PV Partnerships (1994–2008), and Measurement, Characterization, and Reliability R&D (1975–present). According to a retrospective benefit-cost evaluation, DOE investments of cost sharing, technical expertise, and technology infrastructure greatly contributed to declining production costs―without these investments, module cost per watt would have been approximately $5.27 in 2008 instead of the actual $1.92 (2008$). See the Figure below. Similarly, DOE investments have accelerated the introduction of warranties (currently 25 years) into the marketplace (not shown in the Figure).
A graphic showing annual and counterfactual PV module production cost per watt curves. The counterfactual cost curve without FSA, PVMaT, or TFP has decreased from $53.28 in 1976 gradually over time to $5.27 in 2008. The counterfactual cost curve without PVMaT or TFP begins at 1992 on the graph, where it is $6.93, then it drops to $2.95 in 2008. The actual weighted average cost curve, which is the line on the bottom of the three, goes from $53.28 and drops over time to eventually be $1.92 in 2008.

Source: Friedman et al. (2005); EIA (2008); IEA (2009); Authors' (Alan C. O'Connor, Ross J. Loomis, and Fern M. Braun) calculations.

Back to Top