This action might not be possible to undo. Are you sure you want to continue?
SUMMARY OF NATIONAL SCIENCE BOARD WORKSHOP: IDENTIFYING, REVIEWING AND FUNDING TRANSFORMATIVE RESEARCH Santa Fe Institute Santa Fe, New Mexico September 22 – 23, 2004 Contents I. Introduction II. Overview of Major Workshop Themes 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Defining “potentially transformative research” Accepting increased risk of failure as inherent in transformative research Emphasizing people, not projects Selecting appropriate reviewers and panelists Creating separate panels for potentially transformative research Developing mechanisms to improve peer review of potentially transformative research Examining funding mechanisms for transformative research Preparing program officers to identify and fund transformative research Disseminating definitions and procedures within and outside of NSF Establishing metrics and tracking systems for potentially transformative research I. Introduction The National Science Board (NSB, the Board) is reviewing existing mechanisms and considering new policies to enhance the National Science Foundation’s (NSF) ability to identify, evaluate, and fund innovative, “potentially transformative” research. Potentially transformative research is defined as research that has the capability to revolutionize existing fields, create new sub-fields, or cause paradigm shifts. The Board sponsored a workshop to discuss current strategies employed within NSF and at other organizations and to solicit input from the scientific community on NSF’s ability to identify and fund transformative research. The workshop took place from September 22-23, 2004 at the Santa Fe Institute in Santa Fe, New Mexico. Participants were principally external researchers from a variety of disciplines, institutions, and at different stages in their careers, but also included several NSF staff and NSB Members. The workshop was an informational, exploratory “brainstorming” discussion, with the goal of raising issues, identifying problems and proposing solutions that would enhance NSF’s support of potentially transformative research. No decisions were made in the workshop and no consensus was sought. Rather, individual participants offered their ideas and recommendations. Prior to the workshop, all invitees were encouraged to submit short position papers. The Board Office also worked with its contractors to develop a white paper to serve as a basic overview of the current funding and identification techniques used by NSF and some non-NSF entities, and to catalyze discussion at the workshop itself. This paper resulted from the review of reports, documents, and websites, interviews with staff of NSF and other government agencies, private foundations and institutes, as well as other interested parties, and initial discussions with the Board’s ad hoc Task Group on HighRisk Research. The white paper identified five key variables in the funding of potentially transformative research: • • • • • The process by which potential grantees are identified The process by which proposals are reviewed Funding mechanisms The amount and duration of funding Administrative structures for identification, funding, and management of transformative research
Revised nf 031805
Initially, the workshop discussion were divided into four general topics: perceptions from within NSF; perceptions of NSF by the wider scientific community; potential issues with NSF’s current mechanisms for identifying, evaluating and funding potentially transformative research; and suggestions for potentially addressing selected issues. Based on issues raised in the position papers, presentations, and discussions among workshop attendees, ten major themes emerged that require attention in the effort to enhance NSF’s ability to identify, evaluate, and fund potentially transformative research. These themes are discussed below. A compilation of the position papers submitted by various workshop participants is included as an appendix. II. Overview of Major Workshop Themes 1. Defining “potentially transformative research” NSF does not have a common definition for “transformative” or “high-risk” research. Potentially transformative research is difficult to define, with a specific definition being context or discipline specific. At the workshop, a breakout group that focused on definitions, criteria and evaluations agreed upon an operative definition for transformative research: “Potentially transformative research produces changes in the way we do science or changes concepts or views on science within the context of a field.” The breakout group came to this definition in an effort to avoid creating definition that “do harm” by discouraging submission of proposals. Investigators building on existing knowledge and not transforming a field might be alienated. However in the ensuing discussion there was considerable dissent about this definition. Some suggested that this definition was too inclusive and that a more exclusive definition was needed, such as the one offered in the Board Office white paper. 1 In this paper, potentially transformative research is defined as: “research that has the capability to revolutionize existing fields, create new subfields, or cause paradigm shifts in thought”. It is important for NSF to develop and adopt a common definition of potentially transformative research for several purposes: • Providing a clear definition of what NSF considers “potentially transformative research” would enable review panels across the Foundation to be more alert and receptive to such proposals. Articulating a definition may also help change the culture and create an environment that encourages risk-taking. Appropriate language for transformative research proposals needs to be incorporated into NSF public guidance for submission of proposals, in the context of the two NSF peer-review criteria (intellectual merit and broader impacts). However, there was some concern that asking PIs in the request for proposals (RFPs) to describe how their research is potentially transformative might cause PIs to exaggerate the originality of their proposals. A clear definition will help NSF to communicate the importance of potentially transformative research to Congress and the public, enhancing their understanding of the importance of risk-taking. It will also allow NSF to develop a tracking mechanism to gather data on potentially transformative research proposals and metrics for success in identifying and funding transformative research.
2. Accepting increased risk of failure as inherent in transformative research Potentially transformative ideas and research are inherently more risky than incremental research based on current paradigms. The ideas may be wrong and the research unproductive. As presently practiced, the peer review process focuses on minimizing risks, both intellectual and technical, creating an atmosphere in which radical ideas fare poorly. Both the desire to maximize the return on the investment of public funds and the social structure of peer review panels militate against the selection of risky, but potentially transformative proposals. This also makes proposing new ideas and approaches risky for principal investigators (PIs). Researchers also face academic constraints that make them risk-averse. Academic advancement depends of getting grants and publishing papers. Workshop participant Dr. Ellen Goldberg (Santa Fe
Overview of Approaches for Identifying, Reviewing, and Supporting Transformative Research. Working Paper of the National Science Board. 2
Revised nf 031805
Institute) suggested that support for transformative research could be encouraged by organizing institutes or departments with a critical mass of like-minded individuals who think differently about taking risks. Dr. Erich Jarvis (Duke University) reworded a well-known proverb: it takes a village to raise a transformative scientist. There clearly needs to be support from senior mentors to help researchers grow into transformers. Dr. Gerald Pollack (University of Washington) outlined some of the problems he perceived in the current competitive peer review system. The peer review system itself constrains potentially transformative research because panels are populated by individuals who speak for the current paradigms and have a stake in maintaining the status quo. He articulated the view that groundbreaking ideas might fare poorly for the simple reason that they challenge entrenched paradigms. He further suggested that promising transformative paradigms need to be bootstrapped into the mainstream so that they can gain visibility, discussion and consideration. NSF could contribute by encouraging individuals to submit proposals for workshops on controversial topics. There are particular risks for young researchers who are anxious to get research done and establish a reputation. Dr. Jarvis suggested that the academic system constrains young researchers by encouraging them to get tenure first and to put off taking risks until later in their careers. He suggested that many young researchers do not send potentially transformative research proposals to NSF for fear of ridicule from their more senior colleagues. It was his experience that “ambitious” proposals are often rejected with an injunction to submit a less ambitious proposal. When successful, such proposals are scaled down and the budgets cut. Several workshop participants suggested that some NSF program officers are reluctant to take risks. Workshop participants stressed the importance of the discretionary authority vested in NSF program officers, division directors and assistant directors and articulated the need to fill these positions with risk takers. Finally, workshop participants acknowledged that while NSF should give higher priority to potentially transformative proposals, this should not be at the expense of other types of proposals. NSF needs to balance and diversify its risk portfolio. Questions that need to be addressed include determining the fraction of the NSF budget that should be allocated to potentially transformative research, the of type and level of risk appropriate to the transformative segment, and the amount, duration and funding rate of transformative proposals relative to other types of NSF grants. Participants also felt that Congress and the public did not understand or appreciate the importance of supporting potentially transformative research as a critical source of the ideas and technical developments that power our economy. 3. Emphasizing people, not projects Placing an increased emphasis on the investigator rather than on the research project itself may improve NSF’s ability to support transformative research by identifying individuals with the instincts and ingenuity to develop ground-breaking ideas or approaches. Emphasizing the quality of the investigator’s previous contributions, as well as his/her track record of creativity, will help distinguish among investigators whose novel ideas are likely to be more sound from those less likely to be sound. It was noted that the quality of an individual’s intellect can be assessed from his/her track record even as early as at the post-doctoral level. Workshop participants suggested that the track record of investigators should be evaluated separately from the approach of the research project. A suggestion was made that transformative scientists be empowered to pursue their own ideas by shortening the length of proposals and emphasizing the rationale and significance of the proposed work, rather than the detailed methodology and extensiveness of preliminary data, as is the current practice. It was pointed out that transformative research is almost always performed by individuals. Small projects minimize administrative and managerial tasks and give more time for project work, while larger projects that involve many PIs can lead to an excessive administrative burden. Researchers need to use their time for the research itself. Moreover, too great a focus on milestones leads to incremental research.
Revised nf 031805
4. Selecting appropriate reviewers and panelists Ideally, potentially transformative proposals should be reviewed by panels that understand, appreciate, and support risk-taking. This is difficult, if not impossible, to achieve, in part because of the human dynamics that occur in groups seeking consensus. Nonetheless, using reviewers who have had experience with multi-disciplinary or risky research, or from related or complementary disciplines may produce more enlightened reviews. Participants took to calling such individuals “transformers” or “luminaries.” They might also include prominent researchers recently retired, or others who have less of a stake in maintaining the status quo. It was suggested that it was important to examine incentives for researchers who had themselves done transformative research to participate in review panels. It was suggested that NSF could markedly improve its ability to identify and involve such luminaries in what might be a special review process for potentially transformative research. Investigators themselves need to be more active in suggesting open-minded and appropriate reviewers. The proposer could also specify the expertise needed to address the proposal. 5. Creating separate panels for potentially transformative research Supplementing current NSF mechanisms for evaluating grants with a separate panel for reviewing or co-reviewing proposals that are identified as potentially transformative may improve their success rates. It was suggested that proposals could reach such a special panel by one of three routes: on the recommendation of a program officer, of one or more panelists, or of an outside reviewer. It was suggested that NSF could develop a network of “luminaries” to run its transformative panel. The panel could be an NSF-wide activity run out of the Director’s office, which would make the transformative research awards. It was suggested that the 5% of funds now earmarked for Small Grants for Exploratory Research (SGERs) be pooled in a special fund at the Director’s level for this purpose. Although NSF could also request more money from Congress, internal reallocation would underscore the importance of potentially transformative research. Since program officers would compete for these funds, this mechanism, while adding some complexity, would also emphasize the importance of transformative research. It was suggested that NSF create a special program with its own review process for “transformative” proposals. However, if investigators themselves could decide to submit to this program, proposal numbers could overwhelm it. For example, NIH Director’s Pioneer Award program received approximately 1300 applications for 5-10 awards. One participant observed that sending a proposal to multiple panels can kill it, hence there is a need for multidisciplinary to review multidisciplinary research. Creating a separate NSF “transformative” panel might send the message that NSF does not fund potentially transformative research through its usual mechanisms. However, by creating a special program for potentially transformative research, NSF would send a clear message that to the scientific community about the value of potentially transformative ideas to NSF. Setting aside specific funds for these purposes could also underscore NSF’s intention to support potentially transformative research. 6. Developing mechanisms to improve peer review of potentially transformative research Proposals that challenge existing paradigms are often misunderstood by review panels. Giving investigators the opportunity to clarify misunderstandings during review will help ensure that novel ideas and approaches are properly judged. Participants discussed mechanisms that give transformers a fairer hearing. These include: 1. A rebuttal process that would allow proposers to respond to reviews prior to panel meetings and/or in real time during panel meetings would make the review process less prone to misunderstanding innovative ideas. Many problems that reviewers have with a proposal are related to clarity; a timely rebuttal process could avoid these problems. 2. A journal-style review, where the proposal is reviewed and comments sent back to the author, would allow the investigator to respond to comments or questions before the panel has ranked the proposals. 3. Investigators could be allowed to include a potentially transformative piece within a proposal. Such a piece could be funded separately without supplanting the main proposal.
Revised nf 031805
4. A two-step approach to peer review could be used, in which the proposal’s intellectual merit and potential impacts are first considered, and then costs are taken into account. This could prevent reviewers from looking more favorably at certain grants with a lower cost because they appear to be “lower risk”. 7. Examining funding mechanisms for transformative research NSF has many current mechanisms that are intended to promote potentially transformative research, including SGERs, FIBRs, Creativity Extensions, and Accomplishment-Based Renewals. 2 The use of these mechanisms should be expanded. There was a sense that program officers currently did not utilize the existing mechanisms to their full potential, and that there should be a push to do so. NSF’s Division of Chemistry has implemented a new approach to supporting the development of new centers. Its Chemical Bonding Centers (CBC’s) initiative grew out of a 2003 NSF workshop that discussed new models for establishing new centers, organized by George Whitesides and Ralph Nuzzo. 3 The first round of competition for the CBCs was based on ideas from this workshop: a “big” chemically based challenge with broad technical interest and public interest that required a long time frame. Critical factors that were identified included strong leadership, a range of skills and approaches, use of cyberinfrastructure, a critical mass of researchers and resources, agility to change, and program autonomy combined with substantial NSF input. The CBC competition involved a three-step process that was designed to minimize the burdens of writing and reviewing, and to produce a reasonable funding rate. Innovative, potentially transformative research, however, cannot be confined to short-term demonstrations with minimal budgets. It takes time to acquire novel facilities, and accumulate individuals with varied backgrounds and technical standards. Yet, it is important to be able to quickly respond to research opportunities at the forefront of research. It was recommended that NSF explicitly and broadly encourage the submission of unsolicited proposals with novel ideas at any time, independent of program announcements. Added flexibility would also be gained if investigators could request additional money in the middle of a project instead of at the end. 8. Preparing program officers to identify and fund transformative research As an alternative to the “special program” approach, participants discussed greater decentralization of decision-making. Encouraging program officers to utilize existing funding mechanisms for potentially transformative research, granting program officers the authority to follow their instincts in funding grants that are high-risk but potentially high-impact, and offering incentives for identifying potentially transformative proposals may improve the evaluation process and success rates. The LIGO Laboratory at the California Institute of Technology was highlighted as a project made possible by the presence of Program Officers who were able to recognize this project as being potentially transformative and their willingness to take a risk. NSF does have models of funding potential transformative research, such as that done at the Santa Fe Institute or through the National Center for Ecological Analysis and Synthesis (NCEAS). These models have been driven largely by external individuals, but required the support of NSF program officers and higher administrators. However, it was recognized that although NSF strives to attract the best people from the community to serve as NSF program officers, not all are equally adventurous. Creating incentives, such as competition for “transformative” funds from a central pool, as well as explicit training in recognizing potentially transformative proposals, may improve NSF’s success rate in supporting transformative research. 9. Disseminating definitions and procedures within and outside of NSF Participants noted that there was a need to identify and propagate successful approaches among the NSF Directorates. In addition, although a letter addressing Foundation activities is periodically circulated around the Foundation, NSF is somewhat opaque to the scientific community and there is a need to address the problem of communication with the scientific community.
For an overview of NSF structures for supporting potentially transformative research, see the Board Office white paper Overview of Approaches for Identifying, Reviewing, and Supporting Transformative Research, Section 2. 3 Nuzzo, Ralph G. and Whitesides, George M. (co-Chairman). (2003) Report of a Workshop on New Mechanisms for Support of High-Risk and Unconventional Research in Chemistry. National Science Foundation, Arlington, VA. 5
Revised nf 031805
It was suggested that NSF’s website could be made both more informative and easier to navigate. There is a larger number of special programs and initiative descriptions are not well understood by the scientific community. The language used in program descriptions could be clarified. Some participants felt that there was a need to simplify: much time is wasted figuring out which programs are relevant and crafting proposals to fit precisely into the NSF program requirements. Although NSF feels like it has an open and transparent system, this is not the perception from the outside. NSF’s Committee of Visitors is mechanism that is meant to be a conduit to the scientific community, but it isn’t very effective. The extent to which NSF welcomes unsolicited proposals was unclear to many workshop participants. Defining potentially transformative research in the NSF guidance document, and outlining the Foundation’s mechanisms for supporting this type of research may increase the submission of proposals. Transformative research takes time and money, and requires a commitment that is often longer than the typical 2-3 year grant. It was pointed out that five-year grants and Creativity Extensions or Accomplishment Based Renewals are effective ways to increase flexibility, but they are not widely known outside the Foundation. Moreover, they may not be effective in the context of transformative research, which may take more than 5 years to bear fruit. 10. Establish metrics and tracking systems for potentially transformative research NSF needs to examine, evaluate and share its practices internally. Metrics need to be developed for assessing past performance in identifying and funding transformative ideas, as well as tracking such research in the future. For example, a retrospective analysis of transformative ideas in diverse fields is necessary to set a baseline. How many of the most original research concepts and methods of the past few decades were funded by NSF? What was the level and duration of NSF funding? How often did proposals deemed transformative lead to dead ends? How often did NSF fund proposals that either reviewers or panelists saw as potentially transformative, but risky? Applicants themselves are not currently asked if their proposal is high-risk or potentially transformative. However, some sub-units of NSF have randomly begun to collect data on “high risk” proposals, and there was a sense that NSF-wide mechanisms for this type of data acquisition should be established. In an example provide by NSF, review panelists were asked to pay attention to proposals that they considered to be high-risk but high-payoff, as they review the proposals. One NSF staff member tracked proposals that she considered high-risk, high-payoff. Her data suggested that the number of transformative proposals had jumped recently, raising questions about whether the definition had changed. Her data suggest that the fraction of proposals deemed transformative remains small and that the funding rate for such proposals is about the same as that for more conventional proposals. However, it was pointed out in subsequent discussion that transformative research cannot always be identified a priori. Transformative research can, in some cases, only be recognized in retrospect. It was suggested that PIs be asked to self-identify their work as potentially transformative; however, even the PI may not see the downstream impacts of the work. Although there is no objective way, at least at present, to identify potentially transformative research, better correlates may emerge from the process of tracking. It remains more productive to identify and take risks on the ideas of creative people, rather than to identify potentially transformative projects. Three characteristics of a transformer were suggested to be: motivation/drive, strong communications skills and creativity.