U.S. Department of Energy - Energy Efficiency and Renewable Energy

Energy Analysis

Program Evaluation: Monitor Implementation

Once you have chosen the evaluation contractor, the next step is to work with the contractor to get the evaluation plan finalized and to provide the lists and other assistance the contractor needs. Your input is essential if the evaluation plan is to reflect what you know and what you need to know about your program. Also this provides a work plan that is an important tool for managing the evaluation. Helping as help is needed and monitoring the implementation of the work plan ensures the work stays on track, and has an added benefit. Many of the lessons learned from evaluation come informally as the evaluation unfolds.

Steps 10 and 11 will help you monitor the implementation of your evaluation (learn more about the other steps in general program evaluations):

Step 10: Develop an Evaluation Plan and Conduct the Evaluation Study

The Evaluation Plan

The statement of work for the evaluation should specify that the evaluation contractor develop and deliver a written Evaluation Plan. This plan will be used to manage the evaluation and will usually be an expansion of the contractor's proposal, modified as agreed between the program manager and the contractor. The contractor will develop it as soon as the project is awarded, and the Plan should be approved before the contractor begins to implement other study activities (unless the program manager approves exceptions so that the contractor can begin collecting time-sensitive data).

The Plan should include:

  • The logic model
  • The questions stipulated
  • The developed research design, data-collection plan, and analysis plan
  • The developed Quality Assurance procedures
  • The deliverables that will be produced during the project
  • The basic content of the evaluation report
  • The schedule for performing

The program manager will approve the document, or set of documents, that constitutes the Evaluation Plan and have it disseminated to all who will implement and review the evaluation. It is advisable to have the Plan reviewed by outside experts. The contractor should not begin to implement the evaluation study until the program manager gives final approval to the Plan.

It should read - Four appendices of the EERE Guide provide additional information relevant to the development of Evaluation Plans. Appendix 10 contains a model Evaluation Plan outline. Appendix 9 describes lessons learned for improving the quality of evaluation studies. Appendix 8 illustrates an outline of a typical evaluation report. Appendix 11 contains the American Evaluation Association's guidance on ethical principles for evaluators. These principles should be reflected in the Evaluation Plan's content.

Conduct the Evaluation

After the program manager approves the Evaluation Plan the evaluation contractor will begin to implement the evaluation study. Most of the effort to implement the evaluation falls on the contractor's staff; however, there are several tasks the program manager may have to undertake during the evaluation to ensure that the study remains on schedule and achieves its objectives. One of these is monitoring the evaluation contractor's work. The next step describes the program manager's responsibility for monitoring the evaluation contractor's work during the evaluation. Others include providing support to the evaluation contractor that will enable it to perform one or more of the evaluation tasks. Such support may include:

  • Supplying program participant lists
  • Supplying copies of tracking systems
  • Preparing letters of introduction to participants from whom the contractor will request data
  • Copies of written program procedures
  • Copies of prior or related general program evaluations
  • Assistance in obtaining OMB clearance for a survey, if applicable
  • Access to interview program staff
  • Timely review of delivered documents and reports. The evaluation report will become the basis of future action; therefore, it is advisable that the program manager allows time to give careful attention to it.

Back to Top

Step 11: Monitoring

These are some features of an evaluation project that should be monitored to ensure the evaluation delivers useful and defensible results:

  • Monitor the evaluation team's performance:

    • Require and hold periodic progress-review meetings. Establish and keep a regular schedule for these. They help the contractor as much as they help the program manager ensure that the project activities are being performed correctly.
    • Written monthly progress reports are useful monitoring tools.

     

  • Monitor contractor invoices relative to work progress. The rate of expenditure on the project can provide an early warning sign of problems to come.

  • Review all milestone products.

  • Meet all of the government's milestones for deliverables or support that have been promised to the contractor. Because program staff tends to give evaluations lower priority than program operations, the contractor's progress can be delayed by inattention to promised assistance.

  • Verify that the elements of the Quality Assurance Plan are followed.

  • If something is not as you expected, conduct a special progress review meeting to confront the issue and develop a resolution before it becomes a serious problem.

  • Verify that the draft evaluation report reasonably satisfies the agreed outline.

    • The major findings must answer the priority questions selected for data collection and analysis.
    • The method descriptions should be clearly presented in a manner that is understandable by a technical peer-review audience.

     

Back to Top