Program Evaluation: Examples of Results
EERE offices have a history of doing evaluation studies. This page highlights selected examples of evaluation results from EERE reviews and studies. Eleven examples are provided for the following three types of evaluations.
- In-progress peer review example
- Example 1: Hydrogen program peer review
- Baseline assessment example
- Example 2: Hydrogen program completed a 2004 hydrogen baseline knowledge assessment
- Outcomes/ impact evaluation examples
- Example 3: Energy savings impact
- Example 4: Fuel savings impact
- Example 5: Economic return on investment
- Example 6: R&D knowledge diffusion impact
- Example 7: R&D knowledge diffusion impact
- Example 8: Consumer awareness impact
- Example 9: Direct and indirect energy savings
- Example 10: Impact on stages of diffusion behavior
- Example 11: R&D acceleration effect
In-Progress Peer Review Example
- Example 1: Hydrogen program peer review. This is an example where the recommendations of a panel of independent experts were taken into consideration to inform program decision making. The hydrogen program used the findings from a 2003 peer review to make decisions to discontinue project research for 5% of their project portfolio, in part based on performance assessment scores and review comments from external independent experts. Below is a selected section of a review results table extracted from the FY2003 Hydrogen Program Peer Evaluation Report.
Programs are able to manage public resources efficiently and effectively whenever poor-performing projects are discontinued or others are modified for improvement, based on peer review findings.
Review Results Table
Project, Performing Organization
|10||Low Cost H2 Production Platform, Praxair||2.95||V||Emphasize collaboration.|
|11||Defect-free Thin Film Membranes for H2 Separation & Isolation, SNL||2.87||V|
|12||Maximizing Photosynthetic Efficiencies and H2 Production in Microalgal Cultures, UC Berkeley||3.33||V||Focus on program RD&D goals for 2005.|
|13||Reformer Model Development for Hydrogen Production, JPL||2.27||V||Model analysis in this area is no longer a program requirement.|
|14||Photoelectrochemical H2 Production, University of Hawaii||3.30||V||Emphasize further development of multi-junction photoelectrodes to meet program RD&D goals for 2005.|
|15||Photoelectrochemical Water Splitting, NREL||3.23||V||Focus on candidate lighting materials.|
|16||Encapsulated Metal Hydride for H2 Separation, SRTC||2.83||V|
|17||Economic Comparison of Renewable Sources for Vehicular Hydrogen in 2040, DTI||2.90||V|
|18||Biomass-Derived H2 from a Thermally Ballasted Gasifier, Iowa State University||2.70||V|
|20||Evaluation of Protected Metal Hydride Slurries in a H2 Mini-Grid, TIAX||3.20||V|
|22||Novel Compression and Fueling Apparatus to Meet Hydrogen Vehicle Range Requirements, Air Products & Chemicals Inc.||3.20||V|
|30||Techno-Economic Analysis of H2 Production by Gasification of Biomass, GTI||2.60||V||Project completed.|
|31||Supercritical Water Partial Oxidation, GA||2.57||V||Unlikely that cost barrier can be overcome.|
|32||Development of Efficient and Robust Algal Hydrogen Production Systems, ORNL||3.47||V||Focus on designing new DNA sequence coding for proton channel.|
|34||Water-Gas Shift Membrane Reactor Studies, University of Pittsburgh||2.90||V||Emphasize feasibility of high-temp water-gas shift under realistic operating conditions.|
|38||Low Cost, High Efficiency Reversible FC Systems, Technology Management, Inc.||2.80||V||High electrical input requirement prevents overcoming energy efficiency barrier.|
|39||High-Efficiency Steam Electrolyzer, LLNL||2.37||V||Carbon deposition at anode is a recurring problem.|
Baseline Assessment Example
- Example 2: Hydrogen program completed a 2004 hydrogen baseline knowledge assessment. The hydrogen program completed a 2004 hydrogen baseline knowledge assessment survey. Four populations targeted by the program for a hydrogen education campaign were surveyed. The survey data provided a baseline of current technical knowledge and opinion about hydrogen and fuel cells in the target populations. It found that "State and local official government scored much higher on the technical questions than other target groups."
Number of responses and average score for each of the four surveyed populations
Average percent correct on technical questions
|State & local government||67.5|
Also, "technical understanding appears to influence opinions about safety. For the General Public, Student, and Large-Scale End User Surveys, respondents with above-average scores on the 11 technical questions are more likely to have an opinion about hydrogen technology safety, and for those respondents who have an opinion, their opinion is more likely to be positive." The survey also found that "State and local officials also scored much higher on the technical questions." This baseline data was used by the program to design an education program for the target audiences.
Outcomes/ Impact Evaluation Examples
- Example 3: Energy savings impact. In a retrospective economic benefit-cost study for the Wind Program, the energy benefits (i.e., wind power generation) were calculated as part of the process for determining the economic impacts. In the wind study, it was determined there was a six year acceleration effect, as determined from interviews with experts. Six years was the mean value of the distribution of responses. This acceleration led to wind energy supplying 139.8 billion kWh that displaced electricity otherwise supplied by a fossil-fuel generation mix (where the mix is determined from the weighed fuel use of energy generation from fossil fuel mix of the States with the largest share of wind energy power):
- 61.5 billion kWh (i.e., 44% of total from wind) supplied by coal-fired generation
- 76.9 billion kWh (i.e., 55% of total from wind) supplied by natural gas-fired generation
- 1.4 billion kWh (i.e., 1% of total from wind) supplied by petroleum-fired generation
The energy benefits estimate provided the data input needed to perform the economic benefits calculations. It also produced the inputs to the subsequent environmental emissions analysis for that benefit-cost study
- Example 4: Fuel savings impact. In an Advanced Combustion Engine (ACE) R&D Benefit-Cost Study, a statistical approach was adopted for the calculation of the fuel savings associated with miles per gallon (MPG) fuel economy improvements, where MPG improvements were linked to a 4.5% DOE-supported R&D improvement in Brake Thermal Efficiency (BTE) – shown in the figure below (an expert-derived counterfactual BTE). The change in MPG was statistically estimated (∂MPG / ∂BTE = 0.153) and the reduction in MPG absent the ACE R&D research was calculated. The calculated reduction in MPG was translated to reduced fuel consumption, where 17.6 billion gallons of diesel fuel was saved over the period 1995 to 2007. This information then provided the energy benefits estimate needed to perform the economic benefits estimates for the net benefit-cost analysis (see figure below). Environmental and energy security impacts were calculated from the energy savings results, as well.
- Example 5: Economic return on investment. In the same study mentioned in the previous example, after the full analysis was completed to determine attributed economic return on investment results, the 17.6 billion gallons of diesel fuel saved was found to correspond to net economic benefits of $69 billion ($2008, undiscounted). The internal rate of return was 63%. (See table and figure below.)
- Example 6: R&D knowledge diffusion impact. Bibliometrics analysis has been used to test and quantify the strength of linkages of program-attributed patents to downstream research and commercial applications. Bibliometrics analysis provides a quantification of knowledge diffusion benefits, as measured through various metrics, such as patent citation rates, citation rank among leading research organizations in number of patents linked to earlier funded patents, citation indices, among others. Bibliometrics analysis also serves to document the links between R&D and commercial applications in the market to reinforce the attribution elements of impact assessment for R&D evaluation studies.
For example, the figure below shows the influence of EERE's solar PV research reaches beyond U.S. leading PV producers to extend to leading companies in the semiconductor industry, as well.
- Example 7: R&D knowledge diffusion impact. Another example is for the Geothermal Technologies Office. In comparing the Influence of DOE-funded geothermal research with that of other organizations on subsequent developments in geothermal energy, Chevron and DOE are at the head of the figure below. Of the 1,016 total geothermal patent families* identified, 253 (24.9%) are linked to Chevron's earlier geothermal patents. Meanwhile, 209 geothermal patent families (20.6%) are linked to earlier DOE-attributed geothermal patents.
- Example 8: Consumer awareness impact. It was determined from a statistical survey (the ENERGY STAR Household Survey), stratified by publicity 'promotion' areas, that substantial fractions of consumers recognize and understand the Energy Star label. Forty-one percent (41%) of households recognize the ENERGY STAR label. A larger proportion of households in high- than in low-publicity areas recognize the ENERGY STAR label.
- Example 9: Direct and indirect energy savings. An evaluation of the Industrial Assessment Centers (IAC) program quantified the substantial direct and indirect energy savings achieved through all relevant pathways. The study also found there were unaccountable benefits, identifying additional IAC energy savings of up to 25%.
- Example 10: Impact on stages of diffusion behavior. According to the 2001 FEMP Evaluations, FEMP ESPCs have influenced the awareness, decision, and implementation behaviors of program participants. "Since involvement with FEMP ESPCs:
- No participants remain unaware of performance contracts (unaware stage) and only 10 percent indicate that they have just become aware of performance contracts (awareness stage);
- FEMP ESPCs have moved about 41 percent of participants at least through the first two stages of the adoption cycle; and
- 56 percent of participants are in either the implementation or confirmation stage, compared to 25 percent before hearing about FEMP ESPCs. An additional 31 percent of participants moved into either the implementation or confirmation stage."
Stage of adoption
Percent of participants (N=101)
Percent of aware nonparticipants (N=117)
(7) Percent of unaware nonparticipants (N=188)
(1) Before hearing about FEMP
(2) Since involvement with FEMP
(3) Movement from (-) / to (+) stage
(4) Before hearing about FEMP
(5) Since hearing about FEMP
(6) Movement from (-) / to (+) stage
|Decision – no||10||7||-3||14||16||+2||1|
|Decision – yes||3||21||+18||8||9||+1||1|
- Example 11: R&D acceleration effect. Since 1975, investments made by the DOE, along with direct DOE-industry partnerships, have made significant contributions to the development of PV technology and markets. These investments were made largely under four major program efforts―Flat-Plate Solar Array Project (FSA) (1975–1985), PV Manufacturing Technology Project (1991–2008), Thin-Film PV Partnerships (1994–2008), and Measurement, Characterization, and Reliability R&D (1975–present). According to a retrospective benefit-cost evaluation, DOE investments of cost sharing, technical expertise, and technology infrastructure greatly contributed to declining production costs―without these investments, module cost per watt would have been approximately $5.27 in 2008 instead of the actual $1.92 (2008$). See the Figure below. Similarly, DOE investments have accelerated the introduction of warranties (currently 25 years) into the marketplace (not shown in the Figure).
Source: Friedman et al. (2005); EIA (2008); IEA (2009); Authors' (Alan C. O'Connor, Ross J. Loomis, and Fern M. Braun) calculations.