OST improves its peer review program
	The U.S. Department of Energy's Office of Science and Technology has revised the system by which it evaluates the technical merit of proposed and ongoing technology development projects. In November 1996, OST provided a grant to the Institute for Regulatory Sciences to collaborate with the American Society of Mechanical Engineers in conducting technical peer reviews of OST technology projects. ASME is nationally recognized for setting technical standards and has a strong tradition of conducting independent technical peer reviews. The impetus for OST's new approach is rooted in external criticisms of past OST practices. In its 1995 review of the Office of Environmental Management's technology development program, the National Research Council Committee on Environmental Management Technologies found little evidence of a credible OST technical peer review process to guide environmental technology development. The U.S. General Accounting Office concurred in a July 1996 report (GAO/RCED-96-184) that focus areas used different systems to select projects and either didn't evaluate the scientific or technical merit of projects before making funding decisions or didn't use independent experts to conduct merit reviews. According to GAO, "nonlead sites... expressed concern that their proposals are not being treated fairly because their focus area's management has a vested interest in selecting proposals submitted from the lead site." As a highly respected technical society and experienced practitioner of peer reviews, ASME should restore confidence in the credibility and effectiveness of OST's technical peer review program and subsequent project funding and development decisions. The new approach and the gate model As recommended by NRC and explained in the provisional guidance, the new approach to technical peer review is applied to all focus area and crosscutting programs and is closely tied to OST's Technology Decision Process. This process identifies six R&D stages leading to technology implementation and six points or "gates" in the technology development process where funding and development decisions are made (see Initiatives, December 1995). The purpose of the gates process is to ensure early evaluation of projects against technical and nontechnical criteria. Gates 2 and 4 signal the transition to exploratory development and engineering development, respectively. These two gates represent major decision points at which significant increases in project funding and scale up are expected. Every technology OST supports will be independently peer reviewed by ASME at Gate 4. Gate 2 technical peer reviews by ASME are strongly recommended. A better technical peer review system The new approach will improve the credibility and effectiveness of OST peer reviews. By requiring focus area and crosscut program managers to respond in writing to ASME review panel reports, the new peer review system challenges program managers to fully consider peer review results in their project selection and development. This measure will also increase the credibility of OST's review program within the technical community and among review participants by demonstrating that review results are being considered. Under OST's previous approach to technical peer review, review participants often expressed concern that their work would serve no useful purpose. Focus area and crosscut program managers were largely free to accept, ignore, or even suppress technical peer review results. To establish a greater degree of oversight and accountability for project development, OST has begun collecting information on the status (i.e., development stage or gate location) of technical projects it supports. This information will be used in part to signal when a technical peer review is needed or required by OST policy. This is an important step in the continued improvement of OST's peer review program. To ensure the continuous improvement of its technical peer review program, OST will prepare a series of guidance briefs to clarify issues that arise as its review program matures. Possible topics include criteria for determining when projects should undergo technical peer review, how technical peer review can improve OST's credibility with stakeholders, uniform formats for technical presentations during peer reviews, and procedures for cataloging and disseminating peer review reports. As part of its annual peer review program plan, OST will also issue a schedule of anticipated technical peer reviews. The review schedule will become more comprehensive and predictable when OST's technology decision process is fully implemented and the list of OST- supported projects is completed. For more information on OST review activities and plans, call Dr. Michael Heeb, OST-HQ review program manager, at (301) 903-7954. Technical peer review at OST Technical peer review is a widely accepted method for evaluating the scientific or technical significance of proposed or ongoing work, the competence of the researchers, the soundness of the research plan, and the likelihood of success. To be credible and effective, technical peer reviewers must be recognized experts in their field, have no real or perceived conflict of interest with the review subject, and be completely independent from the review sponsor. Implemented correctly, peer review can help OST assemble a portfolio of technologies based on sound scientific principles and designed to meet site remediation needs. The figure below illustrates key features of the new technical peer review program. OST's peer review coordinator, appointed by OST's deputy assistant secretary (step 1), serves as a liaison between focus area and crosscut program managers (step 2) and the ASME peer review committee (step 3). ASME's peer review committee meets twice a year, approves appointments to review panels (step 4), reviews and approves all review panel reports (step 9), and approves and presents the ASME annual report to the deputy assistant secretary (step 10). An executive panel of five is responsible for the day-to-day operations of the peer review committee. ASME membership is required only for the peer review committee's executive panel. Review panels consist of three to five members selected for their technical expertise, professional reputation, and lack of conflict with the review sponsor or review subject. A review panel executes the review (step 5), prepares a report detailing its recommendations and observations, and transmits the report directly to the sponsoring focus area or crosscut program manager (step 6). Review panels terminate upon completion of their specific review task. Focus area or crosscut program managers initiate the peer review process by making written requests for technical reviews to the peer review coordinator. The new approach to peer review also requires focus area or crosscut program managers to provide written responses to review panel reports. The peer review coordinator establishes priorities for peer reviews according to urgency and logistical requirements. The focus area or crosscut programs cover the cost of documentation, meeting facilities, and travel for program personnel. RSI, through its grant from OST, covers reviewer costs for ASME. Current funding levels are sufficient for approximately one major review (e.g., involving multiple technologies and a site visit) and five lesser reviews (e.g., involving one technology or a document review requiring no site visit) annually for each focus area or crosscut program. To date, ASME has conducted four peer reviews of OST technologies.