Program Evaluation: Post-Review Activities
Activities after the peer review include finalizing the program response, action plans and tracking, and evaluating the process.
Finalizing Program Response
It is important that the appropriate experts/staff/managers are consulted on the review findings and have time to discuss and respond before the report is provided to senior management. Review leaders and program managers should attempt to respond to findings and recommendations and develop initial action plans within two weeks of receiving the reviewers' report, while the discussion is fresh in everyone's mind. This should be done in a clear and transparent process. The response could be developed at a post-review meeting after both program management and the project teams have had sufficient time to review the peer review results and then discuss the overall outcomes of the review and actions for the coming year.
Part of the consideration of peer review findings can be analysis of what has persisted and what trends are present between the present review and past reviews. In many cases it could also include an examination of reviews across related programs, subprograms or similar sets of activities.
The program manager should prepare a memo or other written record that responds to the peer review comments and specifies acceptance or, when appropriate, rebuttal and non-acceptance of these findings. Key issues would be identified as well as a timeline for implementing actions taken in response to specific review comments or recommendations. Action planning is a key step that is often lost in the press of daily business, but change rarely occurs if attention is not given to post-review follow-up. An example of a form for program response to peer review is shown in Appendix M.
The review report and the program response are packaged into one document for distribution to program and senior management and to the principal investigators, project team, and/or other presenters. This enables effective decision-making by all parties involved.
As stated in Section 3.2-Minimum Requirements- "Reports will be promptly communicated to senior management and all persons involved in the review." The report will also be made available to the public (see Section 7.4).
Finally, it is important that reviewers be formally thanked for their efforts by the program and any agreed upon payment provided. As participation as a peer reviewer is often not recognized by their institutions-for example, universities focus on the individual's publications-it is also often useful to formally thank senior management in the institution itself for the work provided by the reviewer.
It is also strongly recommended that the review leader describe to the reviewers how EERE responded to the results of the review in a clear and transparent way, including how the project/program was changed or why the comments were not used. This could be done by providing them with a copy of the review that includes the program response. Or it could be a separate effort. It probably will not include a point-by-point response to recommendations, but will include at least a reasonably detailed description of the internal process of discussion and decisions. This description would include a caveat that addressing every recommendation may not be feasible because implementation may depend on negotiations with contractors, cost implications, etc. This feedback will help ensure that reviewers feel their efforts are well spent and they will be more likely to participate in future reviews.
Action Plans and Tracking
After information needed from the review has been provided to those who need it, the program then needs to track and regularly follow up on actions. Documentation on actions taken and progress and results will likely be requested as part of the next peer review. It is recommended that programs make peer review response tracking a routine part of management practice. Appendix N provides an example of a form for peer review response tracking.
The table below provides an example of what an action plan for peer review response tracking might look like. Notice that actions are specific and a person is assigned responsibility for making sure it is accomplished.
From the Biomass Program Review, 2002
|Action Item||Relevant Reviewer Comment(s)||Action Owner||Progress to Date|
|1||Improve communication among the Program's research areas and work on feedstock and related issues with USDA||Give higher priority to feedstock-related activities, and improve communication regarding these projects with other Biomass Program research activities, as well as with USDA.||[name]||[narrative]|
Evaluating the Peer Review Process
EERE has planned for continuous improvement as it moves to a more systematic and consistent application of peer review. To aid this process improvement, the peer review leader should ensure that an evaluation of the peer review is completed on-site at the end of the review or forwarded to a designated person within a specific period (no more than one week). The recommended post-review evaluation questionnaire is available for download. The peer review leader may wish to supplement the completed post-review evaluation questionnaire with informal but documented discussion about the review process and what worked and did not work. This could be done on-site in an informal meeting called by the review leader with the reviewers and review chairperson and with those who have been reviewed. The following aspects of the peer review, with responses analyzed separately for reviewers, the reviewed, and program staff, are covered by the recommended questionnaire:
For all participants:
- Elements of the process (purpose, reviewer credentials, presentations, frequency)
- Evaluation criteria (realistic for types of purposes, clear, sufficient)
- Greatest strengths and weaknesses of the process
- Comments on how the process can be improved
For reviewers and principal investigators only:
- Adequacy of data provided, opinion of the resulting report
For staff from program office under review:
- Contributions and burden of the review, ideas for reducing the burden
- Costs compared to benefits