Program Evaluation: Types of Evaluations
The areas within the Office of Energy Efficiency and Renewable Energy (EERE) use several types of evaluation to quantify impacts, assess progress, and promote improvement. Several of these are listed below.
- Merit review of proposals, conducted by experts, are used to evaluate project proposals through the use of merit review criteria.
- In-progress peer reviews, conducted by independent external experts, are reviews of program and subprogram portfolios to assess the following:
- quality, productivity, and accomplishments;
- relevance of program success to EERE strategic and programmatic goals; and
- Stage Gate or go/no-go reviews are conducted by experts using a set of defined technology and business case evaluation criteria to determine whether or not to advance an R&D activity forward to another stage of development.
- Impact evaluations are conducted by independent external experts to quantify program-attributed outcomes, such as energy saved, MW capacity, emissions reduction (e.g., CO2), economic performance (e.g., return on investment), knowledge diffusion (e.g., measured via various patent citation metrics), and market effects (e.g., changes in the scale-up of supply and distribution chains, consumer awareness, and adoption decision practices).
- Technical Program Reviews and Advisory Committee Reviews are conducted by EERE senior management, technical teams, or advisory committees.
In-Progress Peer Reviews
In-progress peer reviews are a distinct form of process evaluation. They seek to answer questions of why and how well an activity is being implemented. EERE defines in-progress peer review as:
"A rigorous, formal, and documented evaluation process using objective criteria and qualified and independent reviewers to make a judgment of the technical/ scientific/business merit, the actual or anticipated results, and the productivity and management effectiveness of programs and/or projects." [EERE Peer Review Guide, 2004]
The EERE Guide and this Web page focus only on in-progress peer reviews (sometimes referred to simply as "peer review").
Downloads of EERE peer review reports can be found at EERE Evaluation Plans and Reports.
Learn more about planning and conducting peer reviews.
Stage Gate or Go/No-Go Reviews
Stage Gate reviews are a form of process evaluations that focus on technology and market readiness. They determine when a technology or activity is ready to move to its next stage of development.
Stage gate reviews involve use of a series of "Gates" to review projects before moving forward, typically using external experts to assess the technical and business cases. Each stage of the process is designed to reduce levels of uncertainty and risk.
Similarly "Down-Select" reviews winnow the R&D paths pursued and "Off-Ramp" reviews determine if the DOE job is done and the technology can be graduated or terminated.
Downloads of completed Stage Gate or go/no-go reviews can be found at EERE Evaluation Plans and Reports.
There are four types of impact evaluations.
- Impact assessments determine the impacts, such as energy saved, resources/costs saved, changes in MW capacity, CO2 and other emissions reduced, oil displacement, and economic performance.
- Knowledge diffusion assessments track knowledge formation and transfer, for example, as measured by patent citation rates, patents of other organizations that link back to earlier EERE-attributed patents, and identify highly-cited patents.
- Market effects assessments measure EERE influence on aspects of supply chain and market infrastructure or structure that accelerate development and adoption of technologies. These include, for example, (1) newly formed and effective R&D and business alliances, (2) increases in production capacity and effectiveness throughout the supply chain (e.g., numbers and profiles of manufacturers, installers, material suppliers, product distributors), and (3) changes in user behaviors.
- Benefit-cost assessments determine the "realized" economic benefits and costs of R&D and technology deployment programs using discounted cash flow analysis.
Downloads of completed impact evaluations can be found at EERE Evaluation Plans and Reports.
Learn more about planning and conducting general program evaluations.
Learn more about planning and conducting benefit-cost evaluations of R&D programs.
Technical Program Reviews and Advisory Committee Reviews
EERE has a long history of performing reviews of its programs. These include technical and strategic EERE program portfolio reviews by EERE Senior Management and reviews by advisory committees and technical teams.
Strategic Program Review
An example of an EERE-corporate level technical program review is the EERE 2002 Strategic Program Review (SPR).
The main focus of an SPR is to review each program's key goals and objectives and multi-year technical and portfolio plans, as well as identify issues requiring EERE Management attention.
For example, in an SPR, each program might be asked to present:
- A compelling portfolio of R&D activities, given status of applicable markets, technology state-of-the-art, and past program performance;
- A coherent organization of program activities and management functions with an outline of decision-process and analytical activities;
- A technology research and development plan that is executable and enumerated in cross-linked performance parameters;
- Cost, schedule, and technical performance metrics; and
- A plan for program implementation that includes a cogent work breakdown structure, managerial structure, and credible suppliers or key personnel.
As a result of an SPR, EERE Office leadership are able to identify key management and research portfolio issues along with a range of possible solutions.