Program Evaluation: Conducting the Review

Conducting the peer review involves facilitation, analysis, and reporting.

Facilitating the Review

The function of the chairperson for the review is to be the technical lead, ensuring the information requested is obtained and documented in the review report. Having provided reviewers with written direction prior to review, it is recommended that the review leader or chairperson reinforce guidelines orally at the opening of the review. This will ensure that the reviewers are clear on what is being asked of them and clarify the purpose of the particular peer review. This provides time to settle any outstanding reviewer concerns or questions before the review begins.

Reviewers are instructed to keep all evaluations strictly confidential during and after the review. The specifics of on-site instruction depend on choices made by the review leader, review chairperson, and/or group. However, in general, reviewers could be instructed to:

  • Read and understand the evaluation criteria and peer review procedures,
  • Evaluate each program element (except those that present a conflict of interest or an appearance of a conflict),
  • Prepare preliminary comments on the merits of the project/program in accordance with the peer review evaluation criteria,
  • Be prepared to discuss each project and/or the program at the meeting or assign a rating or ratings that reflect the reviewer's opinion of the merit of the project/program in accordance with the specific evaluation criteria, and
  • Complete the post-review evaluation form.

It is very important that the review chairperson control group dynamics and ensure that all points of view are heard. The review chairperson does not have to be responsible for logistics. It would be helpful to assign a non-technical person (possibly a contractor) to work with the review chairperson to handle these logistics during the review. That person could act as timekeeper, for example, advising the presenters when the allotted time is about to expire. Additionally, this person can ensure that all reviewers have paper copies of the presentations as well as a set of any standardized review forms.

General procedures during the meeting are the following:

  • A program official provides an overview of the program.
  • Program projects are individually presented.
  • Following each presentation or group of presentations, the chairperson begins a question and answer period during which individual reviewers ask questions of presenters (normally principal investigators for R&D programs).
  • When a reviewer requests clarifying information, the chairperson ensures that is provided, if appropriate.
  • At scheduled times on the agenda, the chairperson will request each member to individually prepare quantitative or qualitative ratings and narrative critique for the program and each program element that is the subject of review. This may be preceded by a discussion session with only the reviewers present.
  • Complete the reviewers' draft summary report.

There needs to be a balance struck between getting abstract details and getting an overview of the materials, but there is frequently a tendency to be superficial and sufficiently detailed and rigorous. Probing questions should be encouraged wherever possible. Depending on the level of the review (project or program) the depth and amount of coverage vary. In general, higher cost and/or more complex projects may require more presentation and discussion time.

Open question-and-answer periods following each presentation allow for clarification and for better understanding of what was presented and helps the reviewer thoroughly evaluate the material and its significance. Experience has shown that a good rule of thumb is that the time needed for question-and-answer sessions for a presentation is at least one-half the presentation time, and preferably as long as the time allotted for presentation. Ample time for discussion is likely to create opportunities for increased reviewer-presenter exchange and produce greater rigor in the review process. Reviewer-to-reviewer interactions can also provide very valuable informal exchanges. These might not be appropriate to include in the formal report, but can greatly assist individual researchers or teams in their efforts. Providing time to assist such interactions can be quite useful.

Getting the right presentation person is often a compromise, as the person most skilled at the presentation may not be the person best able to answer technical details. The recommended approach is to get the "best person" who can answer the types of questions likely to be fielded.

Back to Top

Analysis

The reviewers' project and/or program evaluation forms are either collected at the completion of the peer review or are forwarded to a designated contact within a specific time period (no more than one week). It is essential that written comments be received from all peer reviewers and that an official record of these comments is delivered by the review chairperson or designee to the review leader. While it may be true that better judgments are offered if the reviewers have additional time to think about what they have heard, in practice reviewers are busy people and when reports are not completed prior to leaving the review it becomes more difficult to complete the review in a timely manner.

Once the reviewers' assessments are in the hands of the review leader, the next step is to prepare the review results for presentation to the program manager. The responsibility of preparing the peer review report rests with the review leader in consultation with the review chairperson. Objectivity is ensured because the review leader is required to disclose in the peer review report the reviewers' comments as they were provided. It is also appropriate to let reviewers see and comment on any reporting of their opinions not generated by themselves. Tabulating results and compiling comments is an important part of the review process because the information presented to management will aid in decision-making. Summarizing may include preparing a summary statement for each program or project, as well as calculating summary ratings and an evaluation summary sheet, if appropriate.

Calculating review ratings will, of course, depend upon the rating system chosen by the review leader and/or panel. The calculations may vary from simple to complex. The reviewer's rating decisions are collected and entered into a spreadsheet format. The narrative comments are then transcribed and compiled, using the same evaluation form originally provided to the reviewers.

Concurrent with the review leader's preparation of the review summary or after it is complete, the reviewers' assessments are provided to the program manager for response prior to the report being given to senior management. The reviewers' assessments and recommendations for each program element reviewed, as well as the review leader's summary and the program manager's response, become part of the official peer review report and record.

Back to Top

The Peer Review Report

The peer review report provides managers with an independent assessment of the program's productivity, relevance, and management. The report should include the following features when applicable:

  • Program/project identification, description, and budget;
  • A narrative summarizing the salient features of the comments of the individual reviewers and their primary reasons for their judgments;
  • Support of conclusions with specific observations;
  • Summary of reviewers' rating or assessment on each individual criteria as well as the overall assessment;
  • Actionable recommendations aimed at improving program performance, including areas where further study is desirable;
  • As appropriate, comments on the status of recommendations made at prior reviews; and
  • Appendices with the full text of reviewer input.

Back to Top