Program Evaluation: Selection of Reviewers
Peer review results depend greatly on the quality of reviewers and the process by which they are selected. The two most important things for competent and credible peer reviewers are that (1) the selection process is such that peer reviewers are selected primarily on the basis of necessary expertise in the subject area under review, and (2) peer reviewers disclose to agencies any conflict of interest prior to being selected, and do not participate in any portion of the review where they have a conflict of interest.
Rather than recommend a single best practice for selecting reviewers, this guide provides a few options and encourages programs to offer or test additional options and then share the process and the circumstances in which the option was, or was not, successful. It is not acceptable and is not an option, however, to have only internal involvement in the nomination and selection process.
Step 1. Define expertise and perspectives needed. The peer review leader, working with staff, the external steering group, if any, and others, examines the entire program portfolio and selects the slate of projects to be reviewed.
All activities should be identified for the peer review chairperson and panel members, after their selection, and the opportunity provided for them to identify additional projects they believe appropriate to include in the review.
For the projects selected for review, the peer review leader and others then: review the technical content of each project individually and for the set of activities as a whole; identify common technical disciplines among projects and unique aspects of each project; and identify the expertise needed for the review. The need for an appropriately broad and balanced spectrum of expertise and perspectives across the final panel will typically require multiple sources of nominations. A matrix might be drawn up that indicates the various expertise needed for the areas being reviewed across the top, and various perspectives or characteristics desired down the side. Reviewers meeting the qualifying standards for individual reviewers are then chosen because they fill a need for the panel as a whole. For example, reviewers would be selected who represent diversity in backgrounds and experience sufficient so the panel collectively covers the range of expertise required. This is especially important for program level reviews.
- In-depth knowledge of the subject area for which he/she is being selected. It is important that this knowledge has been "demonstrated" in terms of publications, patents, professional awards or positions, or other recognized credentials in the subject area.
- That reviewers have no real or perceived conflicts of interest.
In addition to satisfying the qualifying criteria, it is desirable that reviewers be people who are motivated to speak out, who have knowledge and perspectives that the program may not have, and who will challenge the program to improve.
Also, review panels should be structured to ensure peer review activity is undertaken by at least three experts for each smallest unit being assessed to reduce the risk that an unfounded individual perspective dominates the assessment and recommendations.
- Arranging for several independent, external, and objective groups familiar with the program to nominate candidates. These external groups might include, for example, research institutions (including universities or university associations, such as the National Association of State Universities and Land-grant Colleges, and not-for-profit laboratories), management institutions (including public agencies), professional societies, or Advisory Boards, such as the Biomass Research and Development Technical Advisory Committee.
- Identifying candidate chairpersons and reviewers from experts identified in a bibliometric search of the published literature on the topic, or from their roles in research or management institutions or professional societies.
- Employing a co-nomination approach for identifying and nominating reviewers, where reviewers are selected from those nominated by more than one external expert in the relevant field.
- Selecting the review chairperson, following the procedures below, who then nominates candidate reviewers.
- Contracting with a third party to run the peer review, including selecting the reviewers, following approaches such as the above).
Candidates from previous reviews are a good source for participants in a future review. Generally speaking, it is helpful to have some portion of a review panel carryover from one review to the next. There is a trade-off, however, between continuity to provide institutional memory and turnover to ensure new perspectives. Having exactly the same membership for multiple reviews could give the appearance of a standing committee that has additional requirements under FACA. Best practice suggests that one-third to one-half of a review panel be carried over from one review to the next.
- Reviewing the performance of reviewers in past reviews, noting who did or did not meet selection criteria based on this experience.
- Contacting candidates to determine their general interest and availability; sending them project summary descriptions to further identify interests and possible conflicts; and requesting from them and reviewing self-assessment forms (see Appendix E).
- Obtaining staff and/or public input, as appropriate, to identify candidates that may have known biases or other issues. Considerable care is needed here to prevent gathering of materials or other input that could unfairly or inappropriately characterize an individual and to make sure that the privacy act or other concerns are not raised.
Step 5. Develop initial selection list. The peer review leader, working with the external steering group, the panel chairperson (after selection) and/or others, assembles an initial list of nominees from the above candidates, by using the available information about the candidates to decline candidates where their self-assessment or possible conflicts-of-interest raise substantial concerns, but taking into account the OMB criteria that necessary scientific and technical expertise is primary.
- Arranging for independent, external, unbiased, objective university, professional society, or other groups familiar with the program, as identified above, to select the chairperson and/or the reviewers from the nominees.
- Selecting from the nominees the review chairperson, who then chooses the rest of the reviewers.
- Identifying the chairperson and the reviewers based on a co-nomination process among the candidates, as described above.
- Using an independent, unbiased, objective contractor to select from the nominees either directly, or in collaboration with the steering group, independent, external, unbiased universities, professional societies, or others.
The selection process should be carefully and fully documented to ensure transparency, as other aspects of the peer review process are, and included in the final peer review report.
Learn about other peer review preparation activities.