Program Evaluation: Criteria, Logistics, Cost, and Analysis

There are a few criteria that are often recommended and used by the DOE, OMB, NAS, and others. Although programs may choose to define additional criteria, at a minimum all EERE programs are expected to use the following three criteria (referred to as "core criteria"). The three core criteria are the following:

Quality, Productivity, and Accomplishments. Quality, as used here, is primarily a measure of the inputs: the quality of the technical approach, the quality of the people, and the quality of the facilities and other resources involved. Productivity is a measure of the activity underway and outputs: what has been achieved and what is the value of the program's output compared to costs. This criterion examines if projects and programs are making progress in meeting targets and goals commensurate with funding levels and degrees of risk. Accomplishments are a measure of the outputs: what has been achieved. This criterion allows evaluation of projects at different stages of development—from beginning to end—and determines how well the mission and goals are being achieved.

Relevance. Relevance means that the program or set of activities provide an important actual or potential contribution to the EERE office's and DOE's mission, goals, or strategy, and to society. For most activities, this means that activities address current known or anticipated technical, market, or policy barriers, or business management or communications support challenges. There could be some longer term, high-risk research where specific contributions are not yet well defined or known.

Management. The management criterion examines how well projects and programs are managed. This includes the quality of program portfolio selection and planning (past and future), how well resources are applied and are leveraged (e.g., in public-private partnerships), and the effectiveness of program execution as well as program integration aimed at improving outputs and overall program delivery. The assessment should try to distinguish between what was under program management control and what was beyond it (external influences).

In addition to specific criteria, reviewers should always be asked to provide an overall assessment. This picks up important points that may have been missed in addressing the defined criteria and also provides an overall summary. For example, the overall assessment typically includes the question "Please provide your general overall impressions?" as well as multiple general questions, such as the following examples of R&D-related questions:

  • "Please identify overall strengths and weaknesses."
  • "What areas of research or other work could be ended or modified; what new areas or directions could be added?"
  • "Identify changes that may have occurred in the research context (markets, policy, competing technologies, etc.) that might alter the planned targets or goals?"

Review Questions

Each of the EERE core criteria and additional criteria for which the program seeks review input can be stated as questions for the reviewers in terms specific to the project or program. There may be one question for each criterion that provides the definition of that criterion, or several specific questions may be asked for each. An advantage of asking specific questions is that it makes it easier for the reviewer to do the job requested of him/her. The type of question will vary depending on the level of the review (project, sub-program, or program) and the type of program (research, technology development, deployment, analysis, or business administration). Questions should go as deep and as technical as necessary to meet management information and decision-making needs. Examples of review questions are provided in Tables 4.2 and 4.3 and in Appendix CPDF of the Guide.

Logistics and Cost Considerations

Often the size and scope of the program or project determines the venue for the peer review. Scheduling the event using public facilities (hotels, conference centers), meal planning, and audiovisual requirements, all should be completed well in advance of the actual meeting.

Often an honorarium is paid to reviewers depending on time spent, as well as reimbursement of travel expenses. This is particularly true if the same reviewers are used more than once. Federal employees cannot be reimbursed for their time, while academics and consultants usually are. A typical honorarium is $500 per day.

Typically, meeting logistics are one of the major costs of a peer review for DOE/EERE. The table below shows there is a wide range of costs for peer reviews. Expenditures vary depending on the number of projects reviewed, the number of reviewers, whether the meeting is open to the public, and the length of the review. Ways of controlling the cost of the review meeting include the following:

  • Structuring the agenda carefully so that the agenda is focused and people's time is used efficiently, and
  • Making maximum use of teleconferences, videoconferences, and other electronic media to prepare the review panel. This is particularly helpful when international reviewers are involved.

Examples of Costs in EERE Peer Review Experience (2001-2003)

EERE Peer Review Total Cost Number of Projects Reviewed Cost per Project Reviewed
Program A $250,000 131 $1900
Program B $150,000 40 $3800
Program C $100,000 25 $4000
Program D $70,000 30 $2300
Program E $65,000 25 $2600
Program F $50,000 13 $3800
Program G $28,000 18 $1600

For some programs, utilizing an identified EERE common support contractor for peer review logistics and related activities might reduce the complexities and costs of conducting peer reviews. A single contractor could provide continuity and consistency across time and reviews. The contractor could work with those who have ongoing reviews to maintain institutional memory and provide tutorials to others, in each case tailoring review logistics to meet the needs of individual programs. This support would reduce the cost and burden of review, simplify the tracking of progress, and aid in the sharing of lessons learned. To help ensure fidelity to the core principles of the guidelines presented here, the contract for this work could be re-competed after a set period, say 4-5 years. For other programs, these benefits may be outweighed by the specialized knowledge and experience that other contractors have developed over the years in managing peer reviews for particular programs or topical areas. Also, it may be a good idea to rotate contractors providing logistics support to keep the review process fresh and objective.

In the past, at least one program has defrayed costs by turning the peer review effectively into a large conference and charging attendees (other than the peer reviewers) a conference registration fee. If such an approach is considered, careful attention should be given to allow waivers of registration fees so that stakeholder groups such as nonprofits or others are not inadvertently excluded by the cost of registration.

Data Collection and Analysis

Part of preparation is planning ahead for the data collection and analysis. Things to think about are:

  • Format for the review,
  • Data collected and provided to the reviewers
  • How the reviewers' analysis will be collected

The important point is that the review leader (and review chairperson) should make decisions by "beginning with the end in mind" of what management decisions need to be made in order to picture the desired report, and then develop the data collection and analysis plan and the tools to develop and display it in the desired manner.

Format. A first and fundamental decision to be made by the peer review leader and the review chairperson (once selected) is what format to use for collecting the expert review input. The most common format in EERE is the independent expert panel that has a chairperson, meets in person with program staff and researchers, and generates written review opinions containing the individual review findings and recommendations. A second format to be considered is a group of external reviewers that does not meet in person but has a chairperson that sends out review guidelines and materials to the group, collects the data by mail or email, and writes a summary report of the group responses with a final review by the group. A third format combines these two to hone in on the specific questions to be addressed in a panel session. Materials are sent to reviewers by mail and comments received back. The questions that are addressed in the face-to-face meeting, and the data collected and presented, respond to the areas of concern found in the earlier written review.

Whichever format is used, and whoever writes the report, responses of the individual reviewers and any summary of those opinions must be thoroughly and accurately reflected in a final report because the expert judgment is both the primary data collection and data analysis method.

When the format is a panel of experts, there is the question of whether a consensus will be sought or not. It should be noted that the Federal Advisory Committee Act (FACA) allows agencies to form panels that provide consensus reports only under special circumstances). Thus, the review panel should not be required to arrive at a consensus opinion nor rating unless the review is turned over to an external contractor or the panel is one already formed under FACA. In fact, it is often preferable not to press for a consensus as the discussion of differences of opinion often brings out important facts.

Review Materials. A second decision is what information to collect and provide for the reviewers prior to the review. The data collected must be sufficient for reviewers to judge the set of activities against the standards that have been set by the definition of the criteria and the specific questions. The data includes material that is provided prior to the review and during the review. A balance must be struck between having too much data (because of resources required to collect and review it) and not having enough data. To the extent possible, using materials already developed or planned for other purposes, rather than developing new materials just for the peer review should minimize the burden on researchers. Depending on the type of program, data can include the following:

  • Information sheets describing the program or project mission, goals, and targets and milestones (and for the higher level "total program" reviews, including data on how funding is allocated across key activity areas);
  • Summary project reports, plans, and budgets;
  • Principal investigator or project manager presentations;
  • Lists of publications or patent applications and the results of citation analysis;
  • Customer surveys, available impact studies;
  • Various reports prepared by other external groups such as the IG, GAO, NAS or others; and/or
  • Any additional data and information reviewers may request.

Rating Techniques. A third decision to be made by the peer review leader and chairperson is whether the program wants to have responses and ratings collected in a format that provides quantitative data. Rating techniques are discussed in the section on evaluation tools.

Learn about other peer review preparation activities.