U.S. Department of Energy - Energy Efficiency and Renewable Energy

Energy Analysis

Program Evaluation: Purpose and Costs

Steps one through three will help you establish the purpose of and determine which resources are available for your evaluation (learn more about the other steps in general program evaluations):

Step 1: Decide the Evaluation Objectives

Evaluation objectives are determined by careful consideration of the possible decisions to which the evaluation's results will contribute. Such decisions include: whether modifications to the program can improve the efficiency with which it achieves its objectives; whether, on the strength of achieving its objectives, the program should be continued in its present form, given increased resources, have its resources reduced, or be terminated. Evaluations also provide information for high-level decisions about whether the program's contribution to society relative to other programs warrants its continued funding.

  • Determine the types of decisions about the program to which the evaluation will contribute
  • Establish objectives for the evaluation that will inform the decisions being made about the program
  • Decide the type(s) of general program evaluation needed to satisfy the objectives.

Programs use evaluations to help decide on future actions

Ask what decisions about the program will be made as a result of the evaluation, who will make them, and when. Examples of the types of decisions made are:

  • Continue the program as is
  • Expand the program, consolidate components, or replicate components found to be most cost-effective
  • Reallocate funding within the program; add funding to the program; reduce program funding
  • Streamline, refine, redesign the program (e.g., to meet a pressing resource constraint)
  • Set more realistic objectives
  • Discontinue ineffective delivery components
  • Discontinue the program.

Specify the evaluation objectives

The decisions listed above may not be based exclusively on the findings from a general program evaluation. The evaluation's objectives will specify what the evaluation must contribute in the way of findings to the decision process. Write the statements for the objectives.

Ask what kinds of information, in general, are needed to satisfy the objectives

 

Decide the type of evaluation needed to develop these types of information

Depending on the objectives and amount of resources available, a program may decide to perform more than one type of evaluation in the same time frame.

Back To Top

Step 2: Determine Cost and Resources Available

Resources consist of budget, staff, and time.

The resources that are available for evaluation will help determine the content of the general program evaluation. Deciding on the resources that will be committed to a general program evaluation involves simultaneous consideration of (1) the importance of the program decisions to which the evaluation will contribute, (2) the resources needed to satisfy the evaluation's objectives, and (3) the resources that the program can afford. The importance (to the program) of the information that the evaluation will develop should strongly influence the resources committed. The more important information needs will require more defensible evaluation methods, and more defensible methods usually cost more.

Defensibility is the ability of the evaluation results to stand up to scientific criticism. Such criticism involves the validity of the evaluation's indicators as measures of performance; the accuracy of its measurements on the indicators; and the appropriateness of its research design and method of analysis for satisfying its objectives.

Some state, electric, and gas utility organizations have used a percent-of-annual-program-cost rule of thumb to set the cost for general program evaluations. These have ranged from 1% to 10%. However, such rules of thumb assume a number of factors that are not always made clear.

Broadly speaking, three factors weigh heavily on cost. These are:

  • The type of evaluation
  • The scope of the information requirement, e.g., number of questions and size of sample
  • The defensibility required of the information results, e.g.

This table shows illustrative ranges of cost for general program evaluations for these and other factors. The actual costs may differ from those shown: a larger evaluation scope will cost more, while a smaller scope will cost less. Measurement methods with higher accuracy, e.g., metering, can increase cost. Most of the factors that affect cost are discussed in the steps for defining questions and methods.

Type of Evaluation with Illustrative Scope Defensibility Lower Defensibility Higher Other Factors

Needs or Market Assessment
Illustrative Scope:
opportunities to serve a wider customer market

$50,000 - $70,000 $70,000 - $150,000 (can be more)
  • Difficulty in identifying and contacting the target population
  • Choice of survey methodology (e.g., on-site, mail)
  • Number of customer segments
  • Need for OMB clearance
Process Evaluation
Illustrative Scope:
customer satisfaction measurement
$20,000 - $40,000 $30,000 - $100,000
  • Difficulty in identifying and contacting the target population
  • Choice of survey methodology (e.g., telephone, mail, Web)
  • Need for OMB clearance
Outcome Evaluation
Illustrative Scope:
quantification of 5-8 direct and indirect outcomes (gross impacts)
$50,000 - $100,000 $70,000 - $200,000
  • Number and complexity of outcomes
  • Source of information (e.g., survey vs. program records)
  • Analytical methodology used to estimate outcomes
  • Need for OMB clearance
Impact Evaluation
Illustrative Scope:
quantification of 5-8 direct and indirect outcomes attributable to program (net impacts)
$150,000 - $200,000 $200,000 - $500,000
  • Number and complexity of outcomes (scope)
  • Source of information (e.g., participant and non-participant surveys)
  • Research design used to control for outside influences (e.g., control group vs. participant self-report of program influence)
  • Methodology used to estimate net outcomes
Cost-benefit Evaluation
Illustrative Scope:
Quantification of energy, cost, and environmental benefits and costs from one or more elements of the program
$40,000- $80,000 $60,000 - $100,000
  • The benefits are usually based on quantified impacts.
  • The costs of the program that produced the impacts used for the benefits (Recognizing partner and customer costs as well).
  • Extent of confounding influences between expenditures and impacts.

Back To Top

Step 3: Overview of Management Steps

To determine the Timeline, schedule and key milestones for conducting the general program evaluation, work backwards from the end of the evaluation process.

  • Determine when the information from the evaluation is required.
  • Allow time for quality assurance review of the results.
  • Estimate the time it will take to perform the evaluation. If the evaluation is likely to require a survey to collect data, allow time for the OMB to approve the survey.
  • Determine when the evaluation must begin in order to deliver its information when it is required.
  • Account for the administration time required to hire an evaluation expert.

Designing the Evaluation

The program manager does not need to make all of the decisions required for the evaluation design. An evaluation contractor will make the more technical decisions; however, the evaluation contractor will take its cues for these decisions from the specifications in the request for technical services. The program manager will need to have a basic understanding of the methods used in designing and performing evaluations in order to give those cues. The program manager will also need such an understanding to judge which evaluation contractor will give the best evaluation for the money. To help, this section provides a brief introduction to the steps in designing an evaluation.

As you proceed through this section, you will need to keep in mind many factors because the parts of an evaluation research design are interrelated. Ultimately, you will need a written management document -the Evaluation Plan- that lays out the plan for performing the evaluation. When working with an evaluation contractor require that the contractor develop such an Evaluation Plan before the actual evaluation gets underway.

Back To Top