Printable Version

Evaluation Strategy

One way to have useful and cost effective evaluations is to strategically plan evaluation activity so it supports management processes such as planning, budgeting, program implementation, and benefits estimation. In this way evaluations generate information to feed into continuous improvement. At the same time, the evaluative information can be used to answer the external mail such as the ongoing call for performance data by appropriators and stakeholders.

Strategically planned evaluations are also an important part of program performance assessment. For example, information provided by peer reviews and general program evaluation studies provide explanatory information that complements data gathered from the routine performance monitoring. General program evaluation studies can investigate market impacts more than technology-focused technical review and analysis can.

Taking a strategic view of evaluation means that you need to 1) establish your program's information needs to, for example, support decision making, and 2) develop an overall strategy that lays out a plan for collecting, analyzing, and using evaluative data.

First, establish your program's information needs and questions

Since the idea is to have all evaluative information working together, you might start by taking stock of your planning, budgeting, program implementation and performance monitoring, and benefits estimation needs and identifying the kinds of information necessary to support these management processes and related decision making. [If you haven't already done so, it would be good to review the information on types of evaluations and other parts of "why and when to evaluate" on this site.] The table below is one way to take stock of information gaps. In the table are questions that you address as program managers. List your sub programs in the columns and go across, placing a check where you could use more information. You may see patterns, such as a lack of information on energy savings generated to date by program activities.

Indicates this area of the program needs this information

Management Activity	Example Information Needs	Program Area 1	Program Area 2	Program Area 3
Planning and Budgeting	Technical and market baseline What delivery mechanisms work better than others?
Implementing and Monitoring	Satisfaction of customers with the program Why we did or didn't reach our performance targets?
Assessing Benefits/Impact	Quantified outcomes resulting from program outputs What has been actual impact on energy savings or clean energy advances attributable to the program to date?

Indicates this area of the program needs this information

Second, develop an overall evaluation strategy

Ideally a program office should have a program-wide evaluation strategy that addresses each program element, with a schedule of planned performance monitoring and evaluation activities and the resources set aside for them. A mix of approaches, not just peer review, is needed to meet information requirements even for technology programs.

Once you have established your program's information needs and questions, what should be your evaluation strategy? Here are some areas to consider.

1. Target evaluation activities to your information gaps

• Concentrate evaluation activities on your program strategy. Developing program logic models can be very useful here. These help capture in a concise way goals and how these goals will be achieved and therefore highlight key performance measures and evaluation questions.
• Prioritize needs, including considerations of who needs to know what, when, and in what level of detail and format.

2. Plan for high quality, cost effective evaluations

• A multi-year strategy is needed because evaluation studies often take 6 months or more to complete, and a schedule helps ensure that all program elements are assessed about every 3 years and information is timely for decision points.
• Avoid the trap of unnecessarily limiting or constraining the scope of an evaluation study to meet a compressed time schedule. In some cases, results from an evaluation may become available every other budget cycle, if an in-depth scope of investigation is called for.
• Put in place standard procedures for gathering as much data as possible as a routine part of program record-keeping
• Combine efforts where you can, for example do customer satisfaction surveys for all program customers at one time.
• Have a quality assurance process in place that ensures data quality and consistency (especially when evaluation data comes from multiple sources).

3. Establish evaluation resources, motivation, and utilization

• Budget resources need to be set aside for evaluation, including an amount for ad hoc studies that cannot be predicted. In Federal agencies, evaluation activity comprises 1 to up to 10 percent of a program's budget.
• Rework management processes and rewards to ensure evaluative information is an integral part of decision-making.
• Summarize your strategy in the Multi-year Program Plan (MYPP). A good MYPP should include a section that describes a systematic plan for carrying out program-wide performance monitoring and program evaluation activities. Although details would be documented in a separate evaluation plan document, the MYPP should include an overview of the following:
• A schedule and description of planned performance assessment activities, including how data on routine performance measures will be validated and analyzed.
• For the major evaluation studies, the decisions informed, and questions addressed along with criteria for answering the questions.
• Brief description of how data is to be collected and analyzed, such as use of peer review or statistical survey.
• Description of how the independence and quality of the evaluation process will be assured.