Program Evaluation: Why, What, and When to Evaluate
Program evaluations are systematic studies conducted periodically or on an ad hoc basis to assess program performance. As tools to support good management practice, they help managers determine if timely adjustments are needed in program design to improve the rate or quality of achievement relative to the committed resources. Evaluations also help the EERE offices quantify program impacts.
Why Perform Evaluations?
Program management is linked to measuring and demonstrating results. Thus there is strong emphasis on good management practice. Good management practice includes the critical action by the program manager to decide, preferably on his/her own initiative, to perform evaluations; followed by use of the findings/results. Ideally programs should gather evaluation information as a matter of routine management practice because evaluation is a tool that can help programs achieve management excellence. Evaluation does this by serving two critical purposes – program improvement and accountability. Many evaluations will be designed to serve both of these purposes.
Improvement: Evaluations help managers assess how well their programs are working by estimating the extent to which desired outcomes are being achieved and by identifying whether improvements are needed to increase effectiveness with respect to objectives. Evaluation activity helps program managers proactively optimize their programs' performance. Evaluations provide actionable information on program activities that are not being performed as intended, outputs that are not as effective as they were expected to be, customer needs or expectations that are not being met, and outcomes that are below projections. Managers can use this information to request additional resources or to modify the program design to improve its effectiveness. A program that undertakes evaluation activities to obtain timely information during the planning, budgeting, implementation, and other management cycles to make continuous improvements will achieve greater success than one that does not.
Accountability: Evaluations help program managers demonstrate internal and external accountability for the use of public resources. This includes demonstrating fiscally responsible management, establishing evidence that goals are being met or services are being delivered as promised, and quantifying "realized" impacts of the program. Accountability is, in part, motivated by a need to address internal and external evaluation requirements and expectations. A program that can account for its public investments by demonstrating progress and results is more likely to be viewed as successful by the appropriators and stakeholders.
Programs that perform evaluation activities out of a desire to promote continuous improvement are motivated by the need for timely information to inform decisions and will automatically be able to answer calls for "public accountability" by Congress and stakeholders.
What and When to Evaluate?
What and when to evaluate program performance depends on several interrelated considerations, such as:
- Stage of the program in its life cycle (creation of program, program underway, or at or near program closeout),
- Decisions to be informed by evaluative information during critical planning, budgeting, implementation, benefits reporting and communication cycles, and
- Any need to meet program evaluation requirements and expectations.
Ideally, an overall program-wide evaluation strategy and plan should be developed to map out planned evaluation activity over a multi-year time frame, so information produced:
- is timely, without being so frequent that not enough time passes to act on recommendations from the evaluation,
- in sync with the program's management and decision processes, and
- complements other program data collection activities.
What to Evaluate
There is no hard and fast rule on what programs or program areas should be evaluated. Here are some circumstances that may suggest your program is in need an evaluation:
- For a new program that is underway or the creation of a new program:
- Desire to understand if and how a new program or pilot works because you want to expand or replicate it later if determined to be successful
- New initiative with lots of uncertainties that need to be better understood to achieve the initiative's full potential
- A new initiative that is a major budgetary priority, or addresses an important problem
- For an existing program that is underway:
- Program has never been formally evaluated, or has persisted year after year without adequate evidence that it is producing results
- Program activities that cover large portions of the program's budget or are highly visible to major stakeholders for other reasons
- A need to re-evaluate an existing program where significant changes were made
- A program (or subprogram activity) that appears to be in trouble (e.g., progress or participation lower than expected), or are experiencing better than expected success
- A program having high potential risks, or a complex design
- A program (or subprogram activity) where there is an interest in adapting it elsewhere
- A program that is a major budgetary priority, or addresses an important problem
- A program that has advanced to a stage where consumer and markets benefits are accruing and economic return on investment can be determined, such as:
- One that has potential for large benefits, or appears to be cost-effective, and the return on investment is currently not known;
- One that was previously evaluated in an earlier period but had shown no significant realized economic return on investment, in part because accrued value (e.g., in the market) had not been established for its commercialized technologies during the period of the earlier study;
- One previously evaluated but requires an update (e.g., two to five years later); or
- One that had performed benefit-cost calculations using an alternative approach not consistent with the EERE approach to retrospective benefit-cost assessment.
- Congress directs a study be done
- For a program at or near closeout:
- Program that is phasing out or has closed out, where lessons learned and best practices could be identified and applied to the development of a future successor initiative
When to Evaluate
Deciding on when to evaluate program performance takes into account the interrelated considerations described above (i.e., stage in program life cycle, timing of program's decisions and evaluation requirements). Generally speaking, selected program activities or the entire program should have some evaluation performed over a time period ranging from annually to every two or three years.
Currently, EERE's internal requirements for when to conduct evaluations include the following:
- Merit reviews of project proposals are performed annually.
- EERE requires in-progress peer reviews of project portfolios or entire programs be performed on average every two years.
- Stage Gate reviews are performed on an as needed basis, determined by when a project portfolio is expected to reach a critical gate in the stage gating process.
See Recommended Type of Program Evaluation by Program Life Cycle Stage on Program Life Cycle.