Program Evaluation: Independence, Conflict of Interest, Openness
This page discusses the aspects of independence, conflict of interest, and public vs. closed sessions, in preparation for successful peer reviews.
The review leader and the program manager should decide at the outset the desired level of independence and the degree of external control for the review. To a large degree, this decision requires a conscious trade-off between two often-competing goals.
The definition of peer review and the EERE core principles and minimum requirements all stress that achieving a high level of independence is crucial to success of the review. The essential component of this is the independence of the reviewers themselves and their ability to maintain objectivity. However, the level of independence in managing and executing the peer review process is also important. These two aspects are related in that the perception of independence may depend on demonstrated independence of the peer review process itself. Process independence depends on the level of control by independent external groups.
If a key goal of the peer review is to achieve the highest possible level of perceived independence, then the review should be turned over to an outside group for management and execution, following appropriate guidance from DOE/EERE on the scope of the review and the criteria for evaluation.
If key goals include high efficiency and effectiveness of the process, more rapid response times in completing the peer review to aid relevance of the review to program decisions, and staff buy-in and implementation of results, then the review should be managed with an appropriate level of direction and support from the EERE office, while still ensuring the independence of the peer reviewers themselves.
The appropriate balance between these goals varies among EERE programs. Some programs require a greater degree of perceived independence due to their funding levels, higher levels of external scrutiny from particular individuals or groups, or other factors. Other programs may be better served with some degree of internal direction to ensure the review process yields timely results for management consideration. An appropriate level of internal direction is one in which the program manager provides substantial guidance in framing the scope and approach of the review, the timelines for conducting the review, and other key issues in order to best meet his/her management needs. However, the peer reviewers are still selected per the process described in Section 5 to ensure independence, and the overall peer review is still guided by the core principles, minimum requirements, and evaluation criteria described in this guide.
Conflict of Interest
Individuals with a conflict of interest in particular areas generally should not participate as reviewers in those areas of the peer review. The concern is that a reviewer's personal affiliations and involvement in particular activities could indicate that they lack the impartiality required for the peer review. Directly identifying possible conflicts of interest is an awkward process in the science and engineering culture, which has largely based its peer review processes, such as for scientific journals, on an implicit honor system. The challenge here is to balance explicit identification of conflicts of interest without implicitly damaging the traditional honor system, while avoiding legalistic transaction overheads. The effort here is intended to provide a mechanism to flag potential conflicts that could raise questions of objectivity in the public arena that a potential reviewer might not be sensitive to or aware of.
Although their involvement in the activities listed below and in Appendix F of the Guide is not necessarily grounds for exclusion from the peer review, it is important that the nominee or reviewer declare these activities to the review leader prior to the review so that conflict of interest can be considered. Upon reading an individual's disclosure of potential conflict of interest, the peer review leader would flag the possible conflict of interest and contact the person to get more detail. Then the review leader would make a determination on whether or not to include that person as a nominee to be a reviewer on parts or all of the review because of real or perceived conflicts of interest.
Affiliations or activities that could potentially lead to conflicts of interest may include the following:
- Work or known future work for parties that could be affected by the individual's judgments on projects or program developments that the individual has been asked to review;
- Any personal benefit the individual might gain (or benefit of their employer, spouse or dependent child) in a direct or predictable way from the developments of the program/ projects they have been asked to review;
- Any previous involvement the individual has had with the program/projects they have been asked to review, such as having participated in a solicitation to the program area that was subsequently not funded, or having a professor, student, or collaborator relationship with the program or its research staff;
- Any financial interest held by the individual (or their employer, spouse, or dependent child) that could be affected by their participation in this review; and
- Any financial relationship the individual has or had with DOE/EERE, such as participation in research grants or cooperative agreements.
All peer reviewers must sign a Conflict-of-Interest form prior to the beginning of the review process. This form is available in Appendix F of the Guide. By signing the form the reviewer is certifying that he/she will not participate in the review of any part of a project/program review that involves a particular matter in which the reviewer has a conflict of interest or where a reasonable person may question the reviewer's impartiality, unless other factors—such as technical expertise—motivate the review chairperson to request the individual's participation.
In addition, during the review the reviewer should agree to disclose any actual or perceived conflicts of interest as soon as the reviewer is aware of the conflict. If the review is underway and a conflict of interest is disclosed, the review leader should probably exclude that individual's opinion from the summary assessment, retaining written comments that may be helpful to the program manager. In any case where a reviewer is used who indicates a possible conflict of interest on the form or at the review, the review leader should take a moment to prepare a memo for the file explaining briefly the nature of the conflict and the benefit to the program of having this person be part of the review should there be a question about this later.
Additionally, every participant (the audience, as well as the reviewers) in a closed session where proprietary information will be discussed or presented must sign a nondisclosure agreement. An example agreement is found in Appendix G, but the actual form to be signed will generally be that of the firm that is protecting its proprietary information. Also, all sensitive data must be marked as such. When the peer review includes an evaluation of proprietary information, having the proper nondisclosure agreements in place protects the information from public disclosure under the Freedom of Information Act (FOIA).
Public vs. Closed Review Sessions
A determination needs to be made early in planning about whether or not the public will be invited to be present or participate in the review sessions. There are positive and negative aspects of having the public present. It is up to the peer review leader in consultation with the program manager and others to decide whether or not to have reviews that are open or closed to the public, or to employ a combination of the two.
Those in favor of having open-to-the-public reviews suggest that having the review meeting open can:
- Help sharpen the questions raised;
- Improve the transparency of the peer review process;
- Help improve or legitimize the technical or management approach;
- Strengthen integration networks for research, deployment delivery, or business management;
- Broaden public learning by providing an opportunity for individuals—and competitors—to hear first hand what others are accomplishing and how they manage their work; and
- Encourage participants to improve performance due to the pressures of presenting publicly to their peers.
Relying primarily on open sessions does not preclude closed session at certain times. For example, reviewers may meet with the key researchers in a closed session for discussion of proprietary data, and the reviewers often meet by themselves at the end of the session for discussion. In many cases, open meetings can also be preserved even when the work has involved proprietary, business sensitive, or confidential information. Open reviews have been uniformly positive.
Those warning against having the public present suggest that this may:
- Bring into the review people with personal motives and biases that detract from the review purpose,
- Inhibit candor on the part of both reviewers and those being reviewed,
- Increase the cost of the review,
- Inhibit the time available for the review panel, and
- Limit consideration of proprietary data (see counter argument above).
It is important that when a closed session is held, that transparency be assured by making publicly available detailed notes or even transcripts—without attributions or proprietary information—from the meeting. This provides a mechanism for external parties to correct inaccurate or misleading information and to improve public understanding of how particular conclusions were drawn.
Learn about other peer review preparation activities.