U.S. Department of Energy - Energy Efficiency and Renewable Energy
Planning, Budget, and Analysis – Program Evaluation
General Program Evaluations
General program evaluation studies are management tools that answer critical questions about program improvement and accountability. They are generally longer-term quantitative field studies, in contrast to peer reviews, and are more often done for deployment programs. They help managers determine if timely adjustments are needed in program design or implementation to improve the rate, or quality, of achievement relative to the committed resources and to quantify program impact.
Types
There are five types of general evaluation studies.
Outcome and impact evaluations and cost-benefit studies are focused on retrospective. Market needs/assessments and process evaluations primarily have a retrospective focus, although planning activities are sometimes evaluated.
See EERE Evaluation Plans & Reports for available EERE general evaluation studies by type. A few studies listed included more than one type of evaluations within the project scope.
It is commonly believed that general program evaluations are only performed by deployment programs, but they are also applicable to technology development programs. See Applicability for Research and Technology Development Programs for how general program evaluations apply to research and technology development programs.
Market Needs/Assessments Evaluations
This type of evaluation focuses on understanding customer needs and identification of key market segments and how they can be best addressed by the program in question. Findings allow managers to decide how well the program serves key markets and clients and how it might be changed to better serve the intended customers and other key market actors.
Market needs/assessment evaluations may have objectives related to a specific program or to a more general program area. When performed for a specific program, market needs evaluations identify areas of customer need that are consistent with the program's objectives and with DOE/EERE's mission but which are not being met by the current program. When performed for a general program area, e.g., buildings, market needs evaluations identify areas of customer need that are consistent with DOE/EERE's mission but which are not being met by a program in the general area. In this situation, a market needs evaluation may identify the need for a new program.
Market needs/assessment evaluations may also have other functions. They may be conducted for the purpose of developing a good understanding of how a particular target market functions so that a new program targeting a need of the market is based on a good theory of who or what to target and why. Market assessments allow the program to establish a market baseline, and more fully understand who are the key market actors and how they interact.
Back To Top
Process or Implementation Evaluations
This type of evaluation examines the efficiency and effectiveness of program implementation processes. The results of the evaluation allow program managers to decide how to improve program operations.
Process evaluations include descriptions of the program's operations and focus on their effectiveness. Some examples of these operations include: internal administration; promotional practices; program delivery; data management; and customer satisfaction with the program. Process evaluations may also identify newly perceived threats to, or opportunities for increased, program effectiveness.
Back To Top
Outcome Evaluations
This type of evaluation can quantify program savings and also measure progress with market transformation. Findings show how well the program is achieving its objectives in a specific time frame. This helps program managers decide whether to continue the program and at what level of effort.
Outcome evaluations measure the outcomes of the program's activities and outputs (as these were identified in the program's logic model or other method of specifying program effects). They make no effort to determine whether the outcome was actually caused by the program or by external influences and, probably, would have occurred even without the program. Therefore, if feasible, it is usually good practice to expand the outcome evaluation into an impact evaluation.
Back To Top
Impact Evaluations
Impact evaluations take outcome evaluations one step further by estimating the proportion of the outcomes that are attributable to the program rather than to other influences. As with Outcome Evaluation, these findings help program managers decide whether to continue the program and at what level of effort, but these decisions can carry greater weight because they are based on findings of causality.
Because impact evaluations estimate the proportion of the outcomes that is attributable to the program, they often require more complex methods of data collection and analysis. Their findings of program effect are usually cost-benefit or cost-effectiveness evaluations. When used for this purpose, it is important that all credible indirect outcomes (secondary effects of outcomes that result directly from program outputs) be included in the impact evaluation.
Back To Top
Cost-Benefit or Cost-Effectiveness Evaluation
These evaluations show the relationship of benefits achieved by program activities and outputs to the costs incurred to achieve those benefits. The findings allow program managers to judge the cost-effectiveness of entire programs and component activities and, based on that, to decide whether to retain, revise, or eliminate the efforts in question.
Cost-benefit evaluations may be performed using benefits to society or to individual customers of the program. Benefits usually consist of economic benefits expressed in dollars; however, benefits such as increased comfort, reduced environmental emissions, reduced waste products, and increased national security can also be used. In these cases, interpreting the results requires qualitative judgment.
Back To Top
These various types of general evaluation are useful for research and technology development programs as well as for deployment programs. Market and needs assessment might be done as part of stage gate analysis and planning a technology development program. Although peer review is one of the primary form of evaluation used by EERE R&D programs, such reviews cannot assess program impact. Outcome, impact, and cost-benefit analysis for technology development have to be done with general program studies, since no set of peers would be able to assess these without data from such a study. Note that "Impact" is NOT one of the EERE core peer review criteria for that reason.
While peer review is well understood and will provide credible results if the process is done well, it looks primarily at quality at the project level, and given the right panel expertise, at relevance and management of programs. However, peer review cannot assess impact or provide quantitative measures or examine questions looking more generally at the process of knowledge formation and transfer.
There are a Variety of Evaluation Methods that can be used to assess R&D programs. They include,
- Peer Review
- Bibliometric Method - Counts & Citations
- Bibliometric Methods - Data Mining
- Hot-spot Patent Analysis Method
- Network Analysis Method
- Composite Performance Rating System (CPRS) based on indicator metrics
- Case Study Method - Descriptive
- Survey Method
- Benchmarking Method
- Technology Commercialization Tracking
- Case Study Method - Economic
- Economic Cluster Method
- Econometric Methods
- Historical Tracing (often including citation analysis)
- Spillover Analysis using combination of methods
Newer methods of evaluation of research and technology development programs are being developed to answer important questions, such as network analysis to identify spillovers, a cost index method to estimate societal benefits, and a Composite Performance Rating System for projects (the U.S. Department of Commerce Advanced Technology Program). The U.S. Department of Energy, Office of Science is developing new methods including network analysis to show knowledge creation and value chain, data mining to describe portfolio and show emerging fields, and technology hot spots using patent data to show influence.
Download the Overview of Evaluation Methods for R&D Programs (PDF 3.0 MB) that provides a quick reference guide to research evaluation methods. Download Adobe Reader.
Learn more about planning, conducting, and using general program evaluations.
|