You are on page 1of 21

CORE QUESTIONS and REPORT TEMPLATE for FY 2005 NSF COMMITTEE OF VISITOR (COV) REVIEWS

Guidance to NSF Staff: This document includes the FY 2005 set of Core Questions and the COV Report Template for use by NSF staff when preparing and conducting COVs during FY 2005. Specific guidance for NSF staff describing the COV review process is described in Subchapter 300Committee of Visitors Reviews (NSF Manual 1, Section VIII) that can be obtained at http://www.inside.nsf.gov/od/gpra/. NSF relies on the judgment of external experts to maintain high standards of program management, to provide advice for continuous improvement of NSF performance, and to ensure openness to the research and education community served by the Foundation. Committee of Visitor (COV) reviews provide NSF with external expert judgments in two areas: (1) assessments of the quality and integrity of program operations and program-level technical and managerial matters pertaining to proposal decisions; and (2) comments on how the outputs and outcomes generated by awardees have contributed to the attainment of NSF’s mission and strategic outcome goals. Many of the Core Questions are derived from NSF performance goals and apply to the portfolio of activities represented in the program(s) under review. The program(s) under review may include several subactivities as well as NSF-wide activities. The directorate or division may instruct the COV to provide answers addressing a cluster or group of programs – a portfolio of activities integrated as a whole – or to provide answers specific to the subactivities of the program, with the latter requiring more time but providing more detailed information. The Division or Directorate may choose to add questions relevant to the activities under review. NSF staff should work with the COV members in advance of the meeting to provide them with the report template, organized background materials, and to identify questions/goals that apply to the program(s) under review. Guidance to the COV: The COV report should provide a balanced assessment of NSF’s performance in two primary areas: (A) the integrity and efficiency of the processes related to proposal review; and (B) the quality of the results of NSF’s investments in the form of outputs and outcomes that appear over time. The COV also explores the relationships between award decisions and program/NSF-wide goals in order to determine the likelihood that the portfolio will lead to the desired results in the future. Discussions leading to answers for Part A of the Core Questions will require study of confidential material such as declined proposals and reviewer comments. COV reports should not contain confidential material or specific information about declined proposals. Discussions leading to answers for Part B of the Core Questions will involve study of nonconfidential material such as results of NSF-funded projects. It is important to recognize that the reports generated by COVs are used in assessing agency progress in order to meet governmentwide performance reporting requirements, and are made available to the public. Since material from COV reports is used in NSF performance reports, the COV report may be subject to an audit. We encourage COV members to provide comments to NSF on how to improve in all areas, as well as suggestions for the COV process, format, and questions.

- 1–
NSF FY 2005 CORE QUESTIONS FOR COVs

FY 2005 REPORT TEMPLATE FOR NSF COMMITTEES OF VISITORS (COVs) Date of COV: November 4 and 5, 2004 Program/Cluster: NSF Director's Award for Distinguished Teaching Scholars (DTS) Division: Division of Undergraduate Education (DUE) Directorate: Education and Human Resources (EHR) Number of actions reviewed by COV 1 : Awards: 19 Declinations: 31 Other: Total number of actions within Program/Cluster/Division during period being reviewed by COV 2 : 172 Awards: 19 Declinations: 153 Other: Manner in which reviewed actions were selected: In consultation with the COV chair, all awards were reviewed. Declined proposals were identified randomly by selecting every 5th declined proposal from a list of all proposals ordered by proposal ID number. This yielded 11 "highly competitive declines" and 20 "low declines." From FY 2001 there were 3 high declines and 9 low declines; from FY 2002 there were 6 high declines and 5 low declines; and from FY 2003 there were 2 high declines and 6 low declines. This is consistent with Chair's desire to have available 2-3 high and low declines from each year.

PART A.

INTEGRITY AND EFFICIENCY OF THE PROGRAM’S PROCESSES AND MANAGEMENT

Briefly discuss and provide comments for each relevant aspect of the program's review process and management. Comments should be based on a review of proposal actions (awards, declinations, and withdrawals) that were completed within the past three fiscal years. Provide comments for each program being reviewed and for those questions that are relevant to the program under review. Quantitative information may be required for some questions. Constructive comments noting areas in need of improvement are encouraged.

1 2

To be provided by NSF staff. To be provided by NSF staff.

- 2–
NSF FY 2005 CORE QUESTIONS FOR COVs

A.1 Questions about the quality and effectiveness of the program’s use of merit review procedures. Provide comments in the space below the question. Discuss areas of
concern in the space provided.

QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES

YES, NO, DATA NOT AVAILABLE, or NOT 3 APPLICABLE

YES 1. Is the review mechanism appropriate? (panels, ad hoc reviews, site visits) Comments:

The review mechanism is done in layers by panels. The mechanism has evolved over the last three years. The NSF staff is used as the final “vetting” panel for these proposals. The reasons for the final decisions are not entirely transparent to the members of the COV, due partly to the fact that there are many applicants who appear to be highly qualified and all could not be winners.
YES 2. Is the review process efficient and effective? Comments:

By using the NSF staff as the final “vetting” panel, the agency has avoided the cost (time and money) of convening yet another panel to bring together the recommendations from the discipline-based panels. The awardees are all clearly deserving, well-known teacher/scholars in their respective fields, thus adding prestige to their own portfolios and to their institutions.
YES 3. Are reviews consistent with priorities and criteria stated in the program’s solicitations, announcements, and guidelines? Comments:

Panelists perhaps need to be more carefully briefed before preparing their reviews. There was some inconsistency in reviews with regard to addressing the intent of the program and the criteria that were clearly stated in the program announcement. This was especially true for the earliest years of this program.
4. Do the individual reviews (either mail or panel) provide sufficient information for the principal investigator(s) to understand the basis for the reviewer’s recommendation? Comments:

So many of the applicants were so outstanding that it was difficult for the COV members to distinguish how final decisions were made. The quality was so high and the reviews were often outstanding. Thus, applicants who were turned
3

YES

If “Not Applicable” please explain why in the “Comments” section.

- 3–
NSF FY 2005 CORE QUESTIONS FOR COVs

down might find it difficult to understand why they were turned down.

5. Do the panel summaries provide sufficient information for the principal investigator(s) to understand the basis for the panel recommendation? Comments:

[See answer to 4 above.] However, the COV notes that a large fraction of the winners were not first-time applicants so that meaningful feedback was either given to these applicants or the different composition of panels from year to year allowed for the high quality of these applicants to shine through.
YES

YES 6. Is the documentation for recommendations complete, and does the program officer provide sufficient information and justification for her/his recommendation? Comments:

The documentation appears to be complete, but the final decision-making process is not entirely transparent. For example, in the case where two panels evaluated a single proposal, one panel clearly supported the applicant, one did not, and an award was made. Similarly, there were many very highly qualified and rated individuals who appeared to rank more highly than a few of the award winners. The final evaluation panel clearly has discretion to exercise and does so; there was no one who received an award who was not a well-qualified and highly deserving candidate.
YES 7. Is the time to decision appropriate? Comments:

All proposals were processed within 6 months in all years.
8. Discuss any issues identified by the COV concerning the quality and effectiveness of the program’s use of merit review procedures:

Please see the answers to questions 1-7.

- 4–
NSF FY 2005 CORE QUESTIONS FOR COVs

A.2 Questions concerning the implementation of the NSF Merit Review Criteria (intellectual merit and broader impacts) by reviewers and program officers.
Provide comments in the space below the question. Discuss issues or concerns in the space provided.

IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA

YES, NO, DATA NOT AVAILABLE, or NOT APPLICABLE 4

1. Have the individual reviews (either mail or panel) addressed both merit review criteria? Comments:

The COV answers yes to this question in general. Addressing both merit criteria became more consistent over time. The most recent year of awards was by far the most consistent in this regard.
YES

2. Have the panel summaries addressed both merit review criteria? Comments:

See above. This program has ONLY panel reviews, and the panel reviews did indeed address both merit review criteria.
YES

3. Have the review analyses (Form 7s) addressed both merit review criteria? Comments:

See above.
YES

4. Discuss any issues the COV has identified with respect to implementation of NSF’s merit review criteria.

It would have been useful to the COV if the final review analysis were broken down to show the synthesized comments explicitly for each of the individual merit review criteria, perhaps by using the same template for comments that individual reviewers used.

4

If “Not Applicable” please explain why in the “Comments” section.

- 5–
NSF FY 2005 CORE QUESTIONS FOR COVs

A.3 Questions concerning the selection of reviewers. Provide comments in the space
below the question. Discuss areas of concern in the space provided.

SELECTION OF REVIEWERS

YES , NO, DATA NOT AVAILABLE, or NOT APPLICABLE 5

YES
1. Did the program make use of an adequate number of reviewers? Comments:

YES
2. Did the program make use of reviewers having appropriate expertise and/or qualifications? Comments:

YES
3. Did the program make appropriate use of reviewers to reflect balance among characteristics such as geography, type of institution, and underrepresented groups? Comments:

We hope that more attention will be paid to underrepresented group participation both with respect to awardees and to reviewers.
YES
4. Did the program recognize and resolve conflicts of interest when appropriate? Comments:

Standard NSF procedures for avoiding conflicts were used by NSF program officers.
5. Discuss any issues the COV has identified relevant to selection of reviewers.

Please see response to 3 above.

5

If “Not Applicable” please explain why in the “Comments” section.

- 6–
NSF FY 2005 CORE QUESTIONS FOR COVs

A.4 Questions concerning the resulting portfolio of awards under review. Provide
comments in the space below the question. Discuss areas of concern in the space provided.

RESULTING PORTFOLIO OF AWARDS

APPROPRIATE, NOT APPROPRIATE 6 , OR DATA NOT AVAILABLE

APPROPRIATE
1. Overall quality of the research and/or education projects supported by the program. Comments:

In the opinion of the COV the quality of the projects varied from good to excellent. In some cases it was unclear whether the award was given for new and original projects or because of the long-standing stature of an individual in his or her field. Moreover, for some projects there was often no good plan for assessment of outcomes, which did not appear to affect the status of an award.
APPROPRIATE
2. Are awards appropriate in size and duration for the scope of the projects? Comments:

Many of the awards are leveraged by the awardees to attract additional funding which further enhances the ability of award winners to continue and expand their scholarly activities.
APPROPRIATE
3. Does the program portfolio have an appropriate balance of: • High risk projects? Comments:

The intent of this program is to support very innovative, potentially high risk projects and the COV believes this intent has been met and is appropriate.
APPROPRIATE
4. Does the program portfolio have an appropriate balance of: • Multidisciplinary projects? Comments:

This appears to be a hallmark of these awards. The combination of learning and scholarly endeavor almost always dictates that the projects are multidisciplinary.

6

If “Not Appropriate” please explain why in the “Comments” section.

- 7–
NSF FY 2005 CORE QUESTIONS FOR COVs

APPROPRIATE
5. Does the program portfolio have an appropriate balance of: • Innovative projects? Comments:

See answer to 3 above.
6. Does the program portfolio have an appropriate balance of: • Funding for centers, groups and awards to individuals? Comments:

DOES NOT APPLY TO THIS PROGRAM APPROPRIATE

7. Does the program portfolio have an appropriate balance of: • Awards to new investigators? Comments:

This program is intended to support well-established faculty who are bringing their research insights into undergraduate education, and it does so magnificently.
APPROPRIATE
8. Does the program portfolio have an appropriate balance of: • Geographical distribution of Principal Investigators? Comments:

While the COV felt that geographic diversity is not an appropriate criterion for this program, it appeared that greater effort in the future could be made to solicit proposals from a more geographically diverse set of institutions.
APPROPRIATE
9. Does the program portfolio have an appropriate balance of: • Institutional types? Comments:

Because part of the intent of this award appears to be to send a strong message in particular to research-intensive IHEs, the awards to institution types appear to be appropriate.
APPROPRIATE
10. Does the program portfolio have an appropriate balance of: • Projects that integrate research and education? Comments:

All projects clearly involved integration of research and education. The COV feels that efforts need to continue to emphasize the importance of the scholarship of learning. Accordingly, assessment plans need to be included in every proposal and evaluated for their value and effectiveness.

- 8–
NSF FY 2005 CORE QUESTIONS FOR COVs

APPROPRIATE
11. Does the program portfolio have an appropriate balance: • Across disciplines and subdisciplines of the activity and of emerging opportunities? Comments:

See answers to 3, 4 and 10 above. The quality and/or innovativeness of these projects in the context of scientific rigor should continue to be foremost in the minds of reviewers
12. Does the program portfolio have appropriate participation of underrepresented groups? Comments:

NOT APPROPRIATE

More effort to solicit proposals from members of underrepresented groups should be made.
APPROPRIATE
13. Is the program relevant to national priorities, agency mission, relevant fields and other customer needs? Include citations of relevant external reports. Comments:

The ASME Board on Engineering Education has recently published a report that addresses these subjects. A report entitled “The Engineer of 2020: Visions of Engineering in the New Century” from the National Academy of Engineering also addresses issues relevant to this program and its intent. See also the WESTAT review and references therein for other citations of relevant reports. There appears to be a growing body of literature very much interested in linkages between learning and discovery.
14. Discuss any concerns relevant to the quality of the projects or the balance of the portfolio.

The comments included with each of the questions in this section address the issues of quality and balance.

A.5 Management of the program under review. Please comment on:

- 9–
NSF FY 2005 CORE QUESTIONS FOR COVs

1. Management of the program. Comments:

Expanded efforts to solicit high quality proposals and to do so in ways that will potentially capture more interest from underrepresented groups need to be made. We recommend greater attention to inclusion of thoughtfully discussed assessment plans in each proposal. We also recommend more consistent follow up with investigators to provide annual reviews. The NSF staff is to be commended for management of this unique and important program in their usual highly efficient manner. Our recommendations are meant to reflect suggestions for possible improvements as this program continues to grow and mature.

2. Responsiveness of the program to emerging research and education opportunities. Comments:

The COV believes this program does an outstanding job of addressing emerging research and education opportunities.
3. Program planning and prioritization process (internal and external) that guided the development of the portfolio. Comments:

For a new and unique program, the planning and prioritization have been most appropriate thus far and will likely to continue to evolve in the future. For example, the decision to introduce a twostage process requiring nominations (preliminary proposals) from institutions is a positive step in forward planning.

4. Additional concerns relevant to the management of the program.

None noted. We continue to be impressed by the level of professionalism and responsiveness of the NSF staff.

- 10 –
NSF FY 2005 CORE QUESTIONS FOR COVs

PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS
NSF investments produce results that appear over time. The answers to the first three (People, Ideas and Tools) questions in this section are to be based on the COV’s study of award results, which are direct and indirect accomplishments of projects supported by the program. These projects may be currently active or closed out during the previous three fiscal years. The COV review may also include consideration of significant impacts and advances that have developed since the previous COV review and are demonstrably linked to NSF investments, regardless of when the investments were made. Incremental progress made on results reported in prior fiscal years may also be considered. The following questions are developed using the NSF outcome goals in the NSF Strategic Plan. The COV should look carefully at and comment on (1) noteworthy achievements of the year based on NSF awards; (2) the ways in which funded projects have collectively affected progress toward NSF’s mission and strategic outcomes; and (3) expectations for future performance based on the current set of awards. NSF asks the COV to provide comments on the degree to which past investments in research and education have contributed to NSF’s progress towards its annual strategic outcome goals and to its mission: • To promote the progress of science. • To advance national health, prosperity, and welfare. • To secure the national defense. • And for other purposes. Excellence in managing NSF underpins all of the agency’s activities. For the response to the Outcome Goal for Organizational Excellence, the COV should comment, where appropriate, on NSF providing an agile, innovative organization. Critical indicators in this area include (1) operation of a credible, efficient merit review system; (2) utilizing and sustaining broad access to new and emerging technologies for business application; (3) developing a diverse, capable, motivated staff that operates with efficiency and integrity; and (4) developing and using performance assessment tools and measures to provide an environment of continuous improvement in NSF’s intellectual investments as well as its management effectiveness.

B. Please provide comments on the activity as it relates to NSF’s Strategic Outcome Goals. Provide examples of outcomes (nuggets) as appropriate. Examples should reference the NSF award number, the Principal Investigator(s) names, and their institutions.

- 11 –
NSF FY 2005 CORE QUESTIONS FOR COVs

B.1 OUTCOME GOAL for PEOPLE: Developing “a diverse, competitive and globally engaged workforce of scientists, engineers, technologists and well-prepared citizens.” Comments: The examples illustrated by the projects proposed by award winners very clearly meet

the expected outcome goals that NSF has for PEOPLE. Many of the projects that award winners are conducting involve a diversity of participants, global engagement, strategic and interesting use of technology, and enhancement of the skills of others interested in learning and discovery in higher education. The award winners are superb role models for attracting others to pursue STEM careers, and in particular they impact everyone in the academy, from undergraduates, to graduate students, to postdoctoral fellows and faculty, who may be interested in integrating research and teaching as their respective careers develop.

B.2 OUTCOME GOAL for IDEAS: Enabling “discovery across the frontier of science and engineering, connected to learning, innovation, and service to society.” Comments: All of the projects embedded within the proposals of award winners are fundamentally

designed to create connections between discovery and learning across science and engineering disciplines. An intentional outcome of these awards is to facilitate potential culture changes at IHEs, wherein stronger connections between teaching and research are recognized, supported, and rewarded.

B.3 OUTCOME GOAL for TOOLS: Providing “broadly accessible, state-of-the-art S&E facilities, tools and other infrastructure that enable discovery, learning and innovation.” Comments: In a number of cases, the projects proposed by award winners involved the development

of new tools and/or technologies that enhance classroom teaching and/or effectiveness. The COV notes that while the development of tools was not a primary objective for this program, many new and innovative tools are being developed and disseminated nevertheless.

- 12 –
NSF FY 2005 CORE QUESTIONS FOR COVs

B.4 OUTCOME GOAL for ORGANIZATIONAL EXCELLENCE: Providing “an agile, innovative organization that fulfills its mission through leadership in state-of-the-art business practices.”

Comments:

The NSF is to be applauded for establishing this program in the first place. The use of a COV to evaluate progress and effectiveness is a positive statement of the desire for strengthening the program and holding it accountable to its goals and intent. By all measures available, the program appears to be setting outstanding priorities and meeting the goals that are laid out in the management plan.

- 13 –
NSF FY 2005 CORE QUESTIONS FOR COVs

PART C. OTHER TOPICS
C.1 Please comment on any program areas in need of improvement or gaps (if any) within program areas.

This is an excellent and unique program. It was the feeling of the COV that greater visibility for this program would be of enormous societal benefit. There needs to be both greater internal visibility at the NSF, internal visibility at the respective institutions of the awardees, and enhanced overall external visibility across the board. Some of this visibility could be created by more systematic and intentional dissemination of the activities of the award winners by both the NSF and the institutions accepting these awards.

C.2 Please provide comments as appropriate on the program’s performance in meeting program-specific goals and objectives that are not covered by the above questions.

Program goals and objectives are clearly being met.
C.3 Please identify agency-wide issues that should be addressed by NSF to help improve the program's performance.

The COV has made some suggestions for ways in which this program might continue to evolve throughout this report. We have no additional suggestions to add here and continue to applaud the care and efficiency that NSF staff exercise in managing this program.
C.4 Please provide comments on any other issues the COV feels are relevant.

The COV feels that all issues of substance have already been addressed in the body of this report.
C.5 NSF would appreciate your comments on how to improve the COV review process, format and report template.

The COV process in this case has been an intensive, thoughtful, collegial and thorough review process. We do not recommend changes to this, particularly in view of the fact that the COV has had ample opportunity to comment on points that might not directly be addressed in the report template.
SIGNATURE BLOCK:

- 14 –
NSF FY 2005 CORE QUESTIONS FOR COVs

__________________ For the EHR/DUE DST COV Sally Mason Chair

- 15 –
NSF FY 2005 CORE QUESTIONS FOR COVs

Committee of Visitors for the NSF Director’s Award for Distinguished Teaching Scholars for Fiscal Years 2001, 2002, and 2003: Response to Questions Posed by NSF Staff Regarding the Program’s Value and How to Improve Outcomes, and Future Directions and Emphases

A. The goal of the DTS program is to foster an academic culture that values a scholarly approach to both research and education. 1. Is this goal clear from a reading of the solicitation? • To members of the COV, the program goal might best be restated as: “The goal of the DTS program is to foster an academic culture that values faculty activities in research and education, both performed with rigor and both valued.” • The COV noted a change in the Program Description that was used in the solicitation for FY 2003 (NSF 02-131) and in the FY 2004 solicitation (NSF 03-591), which we believe altered inappropriately the values the program seeks to promote. We recommend retaining something like the wording contained in the following paragraph, which was omitted from the more recent solicitation. “The program aims to have an impact on: (a) the scholars themselves, (b) other faculty, (c) academic institutions, and (d) undergraduate students. The award will support scholars’ continued activities and growth as educators and researchers and enhance their visibility and influence as leaders in reforming the culture of institutions of higher education.” 2. Considering the goal of the program and expected outcomes, is the program worth the effort and investment? • The program is most certainly worth the effort and investment! • The novelty and uniqueness of this program are praiseworthy. • The awardees are uniformly of high caliber and deserving. Awardees are not merely outstanding teachers; they embody excellence across the academy in their efforts to tie together learning, discovery and engagement, particularly as these pertain to student learners. • Changing the culture in higher education is a lofty goal that will require long-term sustained investment and effort. This program is a means to promote this type of change.

- 16 –
NSF FY 2005 CORE QUESTIONS FOR COVs

B.

Objectives for achieving goal (see list of objectives; pages 20 and 21): Are the objectives appropriate and sufficiently comprehensive in scope? • The objectives as stated are appropriate and sufficiently comprehensive in scope to accomplish the goals of the program. The COV feels that these objectives should be stated in a way that clearly maintains the strength of the program in future program solicitations.

C.

Achievement of objectives: To what extent have the objectives been achieved? • The faculty who have received these awards are indeed scholars who have distinguished themselves both in their research disciplines and in their contributions to undergraduate education. The COV believes there is a fascinating and evolving story to be told about how awardees are influencing their disciplines and institutions and thus lessons that have been learned from these very personal stories. The COV suggests that one result of a study of these lessons might be the identification of institutional best practices. We urge that such a study be undertaken. • The DTS award winners have indeed integrated their disciplinary research into their educational activities. • The dissemination of the experience of these scholars should be vastly expanded and should be broader than the winners’ own disciplines or their own institutions. It is hoped these award winners are providing positive role models for students and other faculty at home and elsewhere. From WESTAT’s (the Program Evaluation Study for the NSF’s Director’s Award for Distinguished Teaching Scholars carried out under contract to WESTAT and provided to the COV to review) interviews with the awardees, there is apparently unevenness in how much progress is being made on dissemination and impact. The change in the program that begins now with a nomination from the institution acknowledges the institutional understanding and validation of what a potential award winner can contribute to the culture of an institution and to the broader scholarly community. Further ideas for addressing this objective are presented in D below. • The COV believes that awardees should clearly be engaging in mentoring of current and prospective faculty members. From WESTAT’s interviews with the awardees, there appears to be unevenness in this activity.

- 17 –
NSF FY 2005 CORE QUESTIONS FOR COVs

• Insistence on enhanced communication about the award, the award winners, and the importance of this to institutions will greatly promote the scholars’ influence and prestige and act as a catalyst to changing academic culture. NSF needs to take a more active role in this effort by developing a communication plan that better conveys the intent and results of this award both internally and externally. Anything that the NSF and its highest officials can do to promote greater interactions between awardees and professional societies or other possible venues for broader dissemination is to be encouraged. • As part of the suggestion to develop a communication plan for dissemination, more attention needs to be paid to showcasing the institutions where such activities are encouraged and nurtured. Other thoughts: • Another suggestion might involve creating linkages between DTS award winners and CAREER award winners and/or other groups of young and/or future faculty at particular institutions. If this is successful locally or regionally, perhaps it can be translated more broadly across the national scene. • Other ideas for how DTS award winners might be further showcased could include inviting awardees to participate more frequently in advocacy activities with Congress on behalf of scientific and educational initiatives.

D.

Potential to improve: How might the program be improved and enhanced to optimize the effort and achievements? • It is important that this award not be perceived as merely a good teaching award. A change in the name of the award to emphasize that it is truly special and unique is much needed. • In the WESTAT report, it was noted that these awards are currently distributed across a demographically limited group of individuals and institutions. We cannot emphasize more clearly the urgency and need to make a stronger effort to include individuals from underrepresented groups and/or from other types of institutions such as minority-serving institutions. • The COV feels that to enhance the prestige of this award, the NSF should consider: - 18 –
NSF FY 2005 CORE QUESTIONS FOR COVs

o Decreasing the number of awards given (make at least three but no more than five annually) o Increasing the size of the award to be commensurate with the Waterman Award o Suggesting to the Director and Deputy Director that they be more directly involved in speaking about and publicizing this award whenever and wherever possible • To enhance the prestige of the award, perhaps a different type of ceremony might be considered. With the fifth year anniversary coming up, there is an opportunity to “make a big fuss,” which should be encouraged. • Finally, our most significant suggestion for possible improvement has to do with the nature of the award. The COV feels strongly that this award is more a “prize” than a grant and thus should be treated as such, which in turn would enhance prestige and thus enhance and hasten much needed academic culture change. o For example, if the award were to be $500,000, then half could go to the recipient to continue and extend work and projects that support the goals of the program. The nominating institution would receive the other half of the award to be used expressly for the purpose of supporting appropriate projects and activities that also further the goals of the award. The institution would submit a plan for use of funds coincident with the potential award winner’s plans.

- 19 –
NSF FY 2005 CORE QUESTIONS FOR COVs

Questions to be addressed by DTS COV Concerning the Program’s Value and how to improve Outcomes, and Future Directions and Emphases A. Program Goal: The goal of the DTS program is to foster an academic culture that values a scholarly approach to both research and education. Questions: 1) Is this goal clear from a reading of the solicitation? 2) Is the goal consistent with NSF's strategic plan with respect to integrating research and education, and 3) Considering the goal of the program and expected outcomes, is the program worth the effort and investment? B. Objectives for achieving goal: The objectives of the program, as stated in the FY 2003 solicitation, are to:

Recognize faculty who have distinguished themselves as scholars in their research discipline and in educating undergraduates; Encourage scholars to continue their dual efforts and to explore and experiment with ways to integrate education and research; Disseminate exemplary experiences of scholars in the education of undergraduates; Support scholars to serve as mentors for other faculty who are trying to balance their contributions to science and engineering and to STEM education; Promote the scholars' influence and prestige so that balanced efforts in teaching and research by other faculty will be recognized and rewarded; and Recognize efforts of institutions of higher education that promote and commit resources to support faculty who effectively contribute to both discipline-related scholarship and science education.

Questions: 1) Are these objectives consistent with the program's goal? 2) Are the objectives appropriate and sufficiently comprehensive in scope? 3) Are the objectives written in terms of measurable outcomes? C. Achievement of objectives: Questions: To what extent have the objectives been achieved? For example, 1) Have individuals been identified who have distinguished themselves as leading scholars in both research and education? - 20 –
NSF FY 2005 CORE QUESTIONS FOR COVs

2) Are awardees serving as models and mentors for others? 3) To what extent are awardees and the program influencing the culture of academia to value the integration of research and education? 4) What specific metrics should be used to assess the extent to which each of the objectives is achieved? 5) Has the relative emphasis in the selection process on a candidate's credentials and the project proposed been appropriate to achieve the program's objectives? D. Potential to improve: How might the program be improved and enhanced to optimize the effort and achievements? For example: 1) Are candidates appropriately solicited? 2) Are awardees adequately recognized? 3) Is dissemination of information about the awardee's achievements and as models adequate? 4) Are the number of awardees per year, and the amount of each award appropriate? 5) What kinds of projects would best foster accomplishment of program objectives and goals? 6) To what extent should "research" or "outreach be emphasized in an awardee's project?" 7) Are the awardees active enough in the community? 8) Could the awardees contribute more to NSF? Note: two significant changes have been made in the program beginning in its 4th year: 1) The nomination of candidates is invited, and 2) Distinguished scholars whose educational impact has been on the K-12 sector are eligible.

- 21 –
NSF FY 2005 CORE QUESTIONS FOR COVs