You are on page 1of 16

CORE QUESTIONS and REPORT TEMPLATE for FY 2007 NSF COMMITTEE OF VISITOR (COV) REVIEWS

Guidance to NSF Staff: This document includes the FY 2007 set of Core Questions and the COV Report Template for use by NSF staff when preparing and conducting COVs during FY 2007. Specific guidance for NSF staff describing the COV review process is described in Subchapter 300Committee of Visitors Reviews (NSF Manual 1, Section VIII) that can be obtained at <www.inside.nsf.gov/od/oia/cov>. NSF relies on the judgment of external experts to maintain high standards of program management, to provide advice for continuous improvement of NSF performance, and to ensure openness to the research and education community served by the Foundation. Committee of Visitor (COV) reviews provide NSF with external expert judgments in two areas: (1) assessments of the quality and integrity of program operations and program-level technical and managerial matters pertaining to proposal decisions; and (2) comments on how the results generated by awardees have contributed to the attainment of NSF’s mission and strategic outcome goals.1 Many of the Core Questions are derived from NSF performance goals and apply to the portfolio of activities represented in the program(s) under review. The program(s) under review may include several subactivities as well as NSF-wide activities. The directorate or division may instruct the COV to provide answers addressing a cluster or group of programs – a portfolio of activities integrated as a whole – or to provide answers specific to the subactivities of the program, with the latter requiring more time but providing more detailed information. The Division or Directorate may choose to add questions relevant to the activities under review. NSF staff should work with the COV members in advance of the meeting to provide them with the report template, organized background materials, and to identify those questions/goals that apply to the program(s) under review. Guidance to the COV: The COV report should provide a balanced assessment of NSF’s performance in two primary areas: (A) the integrity and efficiency of the processes related to proposal review; and (B) the quality of the results of NSF’s investments that appear over time. The COV also explores the relationships between award decisions and program/NSF-wide goals in order to determine the likelihood that the portfolio will lead to the desired results in the future. Discussions leading to answers for Part A of the Core Questions will require study of confidential material such as declined proposals and reviewer comments. COV reports should not contain confidential material or specific information about declined proposals. Discussions leading to answers for Part B of the Core Questions will involve study of non-confidential material such as results of NSF-funded projects. The reports generated by COVs are used in assessing agency progress in order to meet government-wide performance reporting requirements, and are made available to the public. Since material from COV reports is used in NSF performance reports, the COV report may be subject to an audit. We encourage COV members to provide comments to NSF on how to improve in all areas, as well as suggestions for the COV process, format, and questions. For past COV reports, please see http://www.nsf.gov/od/oia/activities/cov/covs.jsp.

Please note that NSF will be adopting a new Strategic Plan in FY 2007. In the latter half of FY2007, the COV Template will be changed in accordance with the new Strategic Plan.

1

- 1–
NSF FY 2007 CORE QUESTIONS FOR COVs

FY 2007 REPORT TEMPLATE FOR NSF COMMITTEES OF VISITORS (COVs)
The table below should be completed by program staff.

Date of COV: January 4-5, 2007 Program/Cluster/Section: Federal Cyber Scholarships for Service (SFS) Program Division: Division of Undergraduate Education (DUE) Directorate: Education and Human Resources (EHR) Number of actions reviewed: Awards: 12 Declinations: 16 Other: Total number of actions within Program/Cluster/Division during period under review: Awards: 64 Declinations: 143 Other: NA Manner in which reviewed actions were selected: The NSF staff randomly selected award jackets and declinations for the SFS COV review by first sorting by proposal number the proposals from each track (Scholarship and Capacity Building). To select Scholarship Track jackets we randomly ordered the awards and selected every 3rd jacket. The same process was used to select declines. To select Capacity Building track jackets we randomly ordered the awards and selected every 5th jacket. The same process was used to select declines. We increased the selection interval for CB jackets because we have more of these awards/declines. Among the selected jackets we rejected 2 because we had repeat institutions/PIs. We re-sampled using the above procedure to select two alternate jackets. An additional four “high-decline” jackets were selected at the panel’s request.

PART A.

INTEGRITY AND EFFICIENCY OF THE PROGRAM’S PROCESSES AND MANAGEMENT

Briefly discuss and provide comments for each relevant aspect of the program's review process and management. Comments should be based on a review of proposal actions (awards, declinations, and withdrawals) that were completed within the past three fiscal years. Provide comments for each program being reviewed and for those questions that are relevant to the program under review. Quantitative information may be required for some questions. Constructive comments noting areas in need of improvement are encouraged.

A.1 Questions about the quality and effectiveness of the program’s use of merit review procedures. Provide comments in the space below the question. Discuss areas of
concern in the space provided.

QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES

YES, NO, DATA NOT AVAILABLE, or NOT 2 APPLICABLE

1. Is the review mechanism appropriate? (panels, ad hoc reviews, site visits)
2

Yes

If “Not Applicable” please explain why in the “Comments” section.

- 2–
NSF FY 2007 CORE QUESTIONS FOR COVs

Comments: Review panels are a good mechanism. We had no data on site visits.

2. Is the review process efficient and effective? Comments: The peer review process is very effective, but some members of the COV believe that in this time of constrained budgetary resources there may be a need to balance the costs associated with additional personnel versus the benefits derived from their on-site participation. See also A3.1 for a related issue. Yes

3. Do the individual reviews (either mail or panel) provide sufficient information for the Principal Investigator(s) to understand the basis for the reviewer’s recommendation? Comments: This is not consistent across files the COV reviewed. Some are outstanding and some have very little information in them. Panel summaries are the real value of the process. Yes

4. Do the panel summaries provide sufficient information for the Principal Investigator(s) to understand the basis for the panel recommendation? Comments: Panel summaries generally seem thorough and much improved over the individual reviews. However, some are overly brief or do not address broader impacts or intellectual merit. The COV suggests that panel members be provided with “worked examples” of good comments or reviews prior to starting the review process. Yes

5. Is the documentation for recommendations complete, and does the program officer provide sufficient information and justification (a) for her/his recommendation? (b) for the Principal Investigator(s)? Comments: See A.1.7 6. Is the time to decision (dwell time) appropriate? Comments: The COV is pleased to see that the lower 2005 completion percentage was

Yes

Yes

- 3–
NSF FY 2007 CORE QUESTIONS FOR COVs

not repeated in 2006. While 2005 did not complete all decisions within 6 months, both 2004 and 2006 did. 7. Additional comments on the quality and effectiveness of the program’s use of merit review procedures: There are two points in the review pipeline at which there could be more information sent forward to the next step. The first is the PO Review Analyses, which do not always address every concern expressed by the Panel Summary. The COV believes the PO has an obligation to address every major negative remark of the panel when making an award. The second point concerns the feedback to the PI on declines. We noticed that several declines, even on proposals with scores as high as 3.67 or 4.00 (so-called “high declines”), received only “boilerplate” or “form letter” PO Comments. In other cases, there was more substantive and constructive feedback to the PIs, often using language drawn from the Panel Summary. The latter seems more appropriate, and the COV believes this should be the general practice.

- 4–
NSF FY 2007 CORE QUESTIONS FOR COVs

A.2 Questions concerning the implementation of the NSF Merit Review Criteria (intellectual merit and broader impacts) by reviewers and program officers.
Provide comments in the space below the question. Discuss issues or concerns in the space provided.

IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA

YES, NO, DATA NOT AVAILABLE, or NOT APPLICABLE3

1. Have the individual reviews (either mail or panel) addressed both merit review criteria? Comments: Yes

2. Have the panel summaries addressed both merit review criteria? Comments: Yes

3. Have the review analyses (Form 7s) addressed both merit review criteria? Comments:

Yes

4. Additional comments with respect to implementation of NSF’s merit review criteria:

3

In “Not Applicable” please explain why in the “Comments” section.

- 5–
NSF FY 2007 CORE QUESTIONS FOR COVs

A.3 Questions concerning the selection of reviewers. Provide comments in the space
below the question. Discuss areas of concern in the space provided.

SELECTION OF REVIEWERS

YES , NO, DATA NOT AVAILABLE, or NOT APPLICABLE4

1. Did the program make use of an adequate number of reviewers? Comments: The COV felt that in some instances there are more reviewers than may be needed. The minimum number of members required for a panel is 3, but this number does not allow for unforeseen absences or member recusal due to conflict of interest. The number of panelists in proposals the COV reviewed varied from 5 to 7, with 6 being typical. The COV suggests 6 as the maximum number.

Yes

2. Did the program make use of reviewers having appropriate expertise and/or qualifications? Comments: Most of the review panelists were from CS/IS/IT departments of academic institutions. A variety of experience and backgrounds, especially those found in current practitioners, should be added. As the graduates of the SFS are to serve in federal agencies, government personnel should be encouraged to sit on review panels. No

3. Did the program make appropriate use of reviewers to reflect balance among characteristics such as geography, type of institution, and underrepresented groups?5 Comments: The COV identified an issue regarding the number of academic versus non-academic (i.e., federal government and industry practitioner) participants. Specifically, in 2004-2006, less than twenty-five percent (25%) of the reviewers in the scholarship track were from outside academia. In addition, their percentage was aggregate; some panels were exclusively
4 5

No

If “Not Applicable” please explain why in the “Comments” section. Please note that less than 35 percent of reviewers report their demographics last fiscal year, so the data may be limited.

- 6–
NSF FY 2007 CORE QUESTIONS FOR COVs

academics and the three non-academics serving in 2004 were all on the same panel. The trend is a small improvement of the ratio of nonacademics to academics over the time period. Regarding capacity building programs, the ratio is even lower, at ten percent (10%), and seems to be declining. In fact, in FY 2006, none of the reviewers were from non-academic organizations.

4. Did the program recognize and resolve conflicts of interest when appropriate? Comments: In the three instances that occurred in the COV jackets, the issues were resolved. Yes

5. Additional comments on reviewer selection:

- 7–
NSF FY 2007 CORE QUESTIONS FOR COVs

A.4 Questions concerning the resulting portfolio of awards under review. Provide
comments in the space below the question. Discuss areas of concern in the space provided.

RESULTING PORTFOLIO OF AWARDS

APPROPRIATE, NOT APPROPRIATE6, OR DATA NOT AVAILABLE

1. Overall quality of the research and/or education projects supported by the program. Comments: The COV saw no problems in the sample jackets examined. Appropriate

2. Are awards appropriate in size and duration for the scope of the projects? Comments: The COV saw no problems in the sample jackets examined. Appropriate

3. Does the program portfolio have an appropriate balance of: • Innovative/high-risk projects?7 Data Not Available

Comments:

4. Does the program portfolio have an appropriate balance of: • Multidisciplinary projects?

Comments: The COV applauds the trend to broader multi-disciplinary proposals from academic institutions. While the preponderance of awards has been to PIs from CS/IS/IT departments, NSF should continue to encourage the involvement of other departments (business, legal, public policy, and ethics) in the SFS program. Such a broad approach reflects the importance of “Risk Management” in information security/assurance theory and practice. This is particularly true within the federal government where the Federal Information Security Management Act has placed a premium on the adoption of a risk-based approach to IT security. The COV believes that future SFS solicitations should reflect this emphasis on risk management.

Not Appropriate

If “Not Appropriate” please explain why in the “Comments” section. For examples and concepts of high risk and innovation, please see Appendix III, p. 66 of the Report of the Advisory Committee for GPRA Performance Assessment for FY 2005, available at <www.nsf.gov/about/performance/reports.jsp>.
7

6

- 8–
NSF FY 2007 CORE QUESTIONS FOR COVs

5. Does the program portfolio have an appropriate balance of: • Funding for centers, groups and awards to individuals? Not Applicable

Comments:

6. Does the program portfolio have an appropriate balance of: • Awards to new investigators? Not Appropriate

Comments: The number of new investigators for capacity building seems satisfactory. The scholarship ratio is troublesome: only 1/16 in 3 years. Understanding the reason for this may require study by NSF.

7. Does the program portfolio have an appropriate balance of: • Geographical distribution of Principal Investigators? Appropriate

Comments: The distribution seems to have improved since the previous COV, but it appears that the Midwest and the Rocky Mountain states remain underrepresented. The NSF might arrange targeting workshops to stimulate submissions from these regions. Reliance on having attained CAEIAE status may exacerbate the problem.

8. Does the program portfolio have an appropriate balance of: • Institutional types? Appropriate

Comments:

9. Does the program portfolio have an appropriate balance of: • Projects that integrate research and education? Appropriate

Comments: It is in the nature of the program to integrate research and education. 10. Does the program portfolio have an appropriate balance: • Across disciplines and subdisciplines of the activity and of emerging opportunities? Appropriate

Comments: As mentioned in A.4.4, it appears that awards are being distributed disproportionately to CS/IS/IT departments as opposed to more broadly

- 9–
NSF FY 2007 CORE QUESTIONS FOR COVs

interdisciplinary entities.

11. Does the program portfolio have appropriate participation of underrepresented groups? Comments: There is quite a difference between students and PIs. The student ratios, both minority and gender, are right at (or better than) the overall student population proportions. However, there is weaker representation among PIs in the new awards in 2004-06. Here the ratios are not good on minorities, and they are even worse on gender. Appropriate

12. Is the program relevant to national priorities, agency mission, relevant fields and other customer needs? Include citations of relevant external reports. Comments:

Appropriate

13. Additional comments on the quality of the projects or the balance of the portfolio:

- 10 –
NSF FY 2007 CORE QUESTIONS FOR COVs

A.5 Management of the program under review. Please comment on:

1. Management of the program. Comments: The SFS program involves a great deal of collaboration with Federal agencies and with the Office of Personnel Management (OPM), so it requires continuity. This need for continuity extends beyond management to operational issues. For example, the COV noted that the program had 100% completion within a six-month window after the solicitation due date in 2004 and 2006, but not in 2005 when the last program officer transition took place (though the COV noted that the NSF target was still achieved in 2005). Thus, the COV believes that this program should be managed by a permanent program officer.

2. Responsiveness of the program to emerging research and education opportunities. Comments: The Information Assurance (IA) world has grown far beyond the boundaries of technical programs and includes many legal, ethical, and business issues. The SFS program needs to go beyond CS/IS/IT to include multi-disciplinary approaches to IA, while continuing to track the latest technological developments.

3. Program planning and prioritization process (internal and external) that guided the development of the portfolio. Comments: The SFS program should ensure that the federal agencies using SFS graduates have the most current resources to manage their systems, and the program should seek ways to identify these resources. For example, in the annual review of the capacity building portion of the SFS program solicitation, a selected group of federal agency IA program managers should be engaged to identify issues that the program could productively address. The COV expects that a continuing examination of IA directions would ensure that issues such as those in A.5.2 are included as they emerge. 4. Additional comments on program management: Comments: The SFS program should obtain statistics on the retention of SFS graduates in federal employment after the end of their two-year commitment. This would give Principal Investigators and the lead Program Director data that could help them understand the features of agencies and institutions that lead to long-term federal employment. The COV recognizes the significant efforts that Diana Burley has contributed to the SFS program and the major progress made in the program under her leadership.

- 11 –
NSF FY 2007 CORE QUESTIONS FOR COVs

PART B. RESULTS OF NSF INVESTMENTS
NSF investments produce results that appear over time. The answers to the first three (People, Ideas and Tools) questions in this section are to be based on the COV’s study of award results, which are direct and indirect accomplishments of projects supported by the program. These projects may be currently active or closed out during the previous three fiscal years. The COV review may also include consideration of significant impacts and advances that have developed since the previous COV review and are demonstrably linked to NSF investments, regardless of when the investments were made. Incremental progress made on results reported in prior fiscal years may also be considered. The following questions are developed using the NSF outcome goals in the NSF Strategic Plan. The COV should look carefully at and comment on (1) noteworthy achievements of the year based on NSF awards; (2) the ways in which funded projects have collectively affected progress toward NSF’s mission and strategic outcomes; and (3) expectations for future performance based on the current set of awards. NSF asks the COV to provide comments on the degree to which past investments in research and education have contributed to NSF’s progress towards its annual strategic outcome goals and to its mission: • To promote the progress of science. • To advance national health, prosperity, and welfare. • To secure the national defense. • And for other purposes. Excellence in managing NSF underpins all of the agency’s activities. For the response to the Outcome Goal for Organizational Excellence, the COV should comment, where appropriate, on NSF providing an agile, innovative organization. Critical indicators in this area include (1) operation of a credible, efficient merit review system; (2) utilizing and sustaining broad access to new and emerging technologies for business application; (3) developing a diverse, capable, motivated staff that operates with efficiency and integrity; and (4) developing and using performance assessment tools and measures to provide an environment of continuous improvement in NSF’s intellectual investments as well as its management effectiveness.

B. Please provide comments on the activity as it relates to NSF’s Strategic Outcome Goals. Provide examples of outcomes as appropriate. Examples should reference the NSF award number, the Principal Investigator(s) names, and their institutions.
B.1 OUTCOME GOAL for PEOPLE: Developing “a diverse, competitive and globally engaged workforce of scientists, engineers, technologists and well-prepared citizens.” Comments: The students graduating from this program and entering the federal government are well prepared. They are highly regarded and eagerly recruited by federal agencies.

B.2 OUTCOME GOAL for IDEAS: Enabling “discovery across the frontier of science and engineering, connected to learning, innovation, and service to society.” Comments:

- 12 –
NSF FY 2007 CORE QUESTIONS FOR COVs

This question is not directly relevant to the SFS program. It would be useful to undertake a study of the impact of the capacity building program as the effects of this activity are not clear as of the date of this review. It would also be useful to develop a bibliography of papers written by SFS participants.

B.3 OUTCOME GOAL for TOOLS: Providing “broadly accessible, state-of-the-art S&E facilities, tools and other infrastructure that enable discovery, learning and innovation.” Comments: The COV are not well-placed to answer this question. B.4 OUTCOME GOAL for ORGANIZATIONAL EXCELLENCE: Providing “an agile, innovative organization that fulfills its mission through leadership in state-of-the-art business practices.”8 Comments: The 92% placement success in 2006 is impressive. There is no comparable metric for the capacity building component of the program.

For examples and further detail on the Organizational Excellence Goal, please refer to pp. 19-21 of NSF’s Strategic Plan, FY 2003-2008, at <http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201>. Please note that there will be a new Strategic Plan in FY 2007.

8

- 13 –
NSF FY 2007 CORE QUESTIONS FOR COVs

PART C. OTHER TOPICS
C.1 Please comment on any program areas in need of improvement or gaps (if any) within program areas.
a) While the NFS SFS program is an extremely well-managed, thoroughly reviewed, and highly successful initiative, the COV was struck by the bifurcation between the SFS role and that of OPM. The COV was extremely interested in the placement and retention of students in federal agencies. The COV believes there should be comprehensive program information for review. o There should be statistics on retention rate within the federal government, now that the program has been in existence for 6 years. o An analysis should be conducted of what happens when scholarship commitments have ended. Specifically, identifying those scholarship recipients leaving the federal government, and if they are leaving, where are they going? b) The COV noted that a disproportionate number of SFS graduates go to non-civilian federal agencies despite the intent of Congress to focus this program on the civil agencies of the federal government. o 66% of SFS students have gone to DoD/NSA and 10% to national laboratories o DoD has its own similar (though smaller) program. o Civilian agencies do not have adequate vacant positions to absorb the SFS graduates. How do we solve this? • OPM should have a position pool for SFS graduates for which civilian agencies, small and large, could compete. • These positions would be temporarily assigned to an employing agency and subsequently returned to OPM for reallocation. c) The COV believes the membership of the ICC should include representatives from large and small civilian agencies. The current composition is dominated by law enforcement, defense, and intelligence agencies. d) The COV encourages SFS to allow student placement in the governments of states and larger municipalities. e) In general, the COV is opposed to allowing scholarship recipients to be placed with federal contractors, with the exception of employment by major Department of Energy (DOE) and other National Labs managed by contractors. The COV recommends expanding the current limit of 5 at National Labs. f) The strong emphasis placed upon CAEIAE designation for SFS participation is a limiting factor that should be reexamined.

C.2 Please provide comments as appropriate on the program’s performance in meeting program-specific goals and objectives that are not covered by the above questions. C.3 Please identify agency-wide issues that should be addressed by NSF to help improve the program's performance.

- 14 –
NSF FY 2007 CORE QUESTIONS FOR COVs

C.4 Please provide comments on any other issues the COV feels are relevant.
The COV would appreciate an overview of the 5 Federal Cyber Service Training and Education Initiatives and how they relate to one another (2006 Annual Report, Pg. 1).

C.5 NSF would appreciate your comments on how to improve the COV review process, format and report template.
The COV would have preferred to understand the scope of the review task at the initial invitation to participate on the panel. The initial readings (from the email of December 20) came at an unexpected time for some panelists, and there was a significant amount of additional reading required of the COV participants in the meeting room. This COV believes that it would be more efficient to know exactly what is expected, and to have additional reading time allotted at the beginning of the meeting. The on-site support was outstanding, from the comprehensive binders, to the computer facilities, to the environmental conditions. The COV commends Diana Burley and her team for the professional management of the SFS program and its review functions.

SIGNATURE BLOCK:

__________________ For the Federal Cyber Service: Scholarship for Service (SFS) Committee of Visitors Joseph O’Rourke Chair

- 15 –
NSF FY 2007 CORE QUESTIONS FOR COVs

MEMORANDUM DATE: TO: Bernice Anderson, Senior Program Director for Evaluation Directorate for Education and Human Resources

FROM: SUBJECT: COV for Federal Cyber Service: Scholarship for Service (SFS) COI and Diversity Memo

The Committee of Visitors report for the Federal Cyber Service: Scholarship for Service (SFS) Program was approved at the EHR Advisory Committee meeting held at NSF on May 2-3, 2007. The COV consisted of four members selected for their expertise related to the goals of the program. They provided a balance with respect to the type of institutions supported through the program, gender, and representation from underrepresented groups. The following table shows the main features of the COV’s diversity. Category of COV Membership Member of EHR Advisory Committee…………. Institution Type: University………………………………… Four-year College………………………. Two-year College………………………. K-12 School or LEA…………………… Industry………………………………….. Federal Agency…………………………. Location East……………………………………….. Midwest/North …………………………. West………………………………………. South……………………………………… Gender Female……………………………………. Male………………………………………. Race/Ethnicity White……………………………………… Black……………………………………… Hispanic………………………………….. Asian/Pacific Islander…………………… Native American.……………………….. No. of COV Members in Category ……1……. …………. ……2……. …………. …………. ……1……. ……1…… ……3…… ……1…… ………… ………… ……1…… ……3…… ……4…… ………… ………… ………… …………

The COV was briefed on Conflict of Interest issues and each COV member completed a COI form. COV members had no conflicts with any of the proposals or files.