Industrial and Government Applications Track Paper Mining for Proposal Reviewers Lessons learned at the national science foundation Seth hettich Michael Pazzani Google, Inc Rutgers University 1600 Amphitheatre Parkway CoRE Building, Rm 706 Mou View. ca 994043 sh@ics. uci. edu Piscataway, NJ 08854-8018 Pa rutgers ed ABSTRACT directors who In this paper, we discuss a prototype application deployed at the ultimately make all decisions. Altho paper reports on U.S. National Science Foundation for assisting program directors reviewing proposals, we argue that the and technology in identifying reviewers for proposals. The application helps would also apply to the reviewing of papers submitted to program directors sort proposals into panels and find reviewers conferences and journals for proposals. To accomplish these tasks, it extracts information Many proposals are reviewed in panels, i.e, a group of from the full text of proposals both to learn about the topics of pically 8-15 reviewers who meet to discuss a set of 20-4 proposals and the expertise of reviewers. We discuss a of related proposals, with each panelist typically reviewing 6- alternatives that were explored, the solution plemented, and the experience in using the solution proposals. Most proposals are submitted in response to a particular solicitation(e.g,"Information Technology Research") workflow of NSF or to a specific program(e. g, " Human Computer Interaction) Individual program directors, or for larger solicitations teams of Categories and Subject Descriptors program officers, perform a number of tasks to insure that H 2.8 Database Applications Data Mining proposals are reviewed equitably. These tasks include 1. Divide the proposals into"clusters"of 20-40 related General terms proposals to create panels Algorithms. Human Factors 2. Finding reviewers. ging applications, technology, Identify potential external reviewers to invite for Keyword Keyword extraction, similarity functions, clustering, information If there is not adequate expertise on a panel to review a proposal, obtain "ad hoc" reviews from 1. INTRODUCTION people with that expertise who are not on a panel The National Science Foundation receives over 40.000 proposals a year. Each proposal is reviewed by several extemal In addition to this lengthy process, reviewers must not have a conflict of interest with proposals they are reviewing(e.g, they reviewers. It is critical to the mission of the agency and the may not be from the same department as the proposals autho integrity of the review process that every proposal is reviewed by and a diverse group of panelists (both scientifically and researchers with the expertise necessary to comment on the merit of the proposal. If there is not a good match between the topic of demographically)is desirable to insure that multiple perspectives a proposal and the expertise of the reviewers, then it is possible are represented in the review process. Furthermore, due to that a project is funded that will not advance the progress of scheduling or workload conflicts, not every invited review science or that a very promising proposal is declined. We explore ccepts the invitation, requiring an iterative process of inviting a the problem of using data mining technology to assist progra batch of reviewers and then inviting others to fill in gaps after the initial reviewers respond to the invitation directors in the review of proposals. Care is taken to match the technology to the existing workflow of the agency and to A particular consideration at NSF is that many proposals are multidiscipline lining genome data. To determine if such a proposal is meritorious, it is important to consult some experts Permission to make digital or hard copies of all or part of this work for ersonal or classroom use is granted without fee provided that copies with backgrounds in data mining(to insure that the method ot made or distributed for profit or commercial advantage and that proposed are likely to work)and in the biological sciences(te copies bear this notice and the full citation on the first page. To copy Insure that the problem addressed is an important open problem) otherwise, or republish, to post on servers redistribute to lists If all reviewers have expertise in one area, it's possible that an d/or a fee important problem would be addressed by a technique that isn't KDD06, August 20-23, 2006, Philadelphia, Pennsylvania, USA. Copyright2006ACMl-59593-3395060008.5500Mining for Proposal Reviewers: Lessons Learned at the National Science Foundation Seth Hettich Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 9 94043 sjh@ics.uci.edu Michael J. Pazzani Rutgers University CoRE Building, Rm 706 96 Frelinghuysen Rd Piscataway, NJ 08854-8018 Pazzani @ rutgers.edu ABSTRACT In this paper, we discuss a prototype application deployed at the U.S. National Science Foundation for assisting program directors in identifying reviewers for proposals. The application helps program directors sort proposals into panels and find reviewers for proposals. To accomplish these tasks, it extracts information from the full text of proposals both to learn about the topics of proposals and the expertise of reviewers. We discuss a variety of alternatives that were explored, the solution that was implemented, and the experience in using the solution within the workflow of NSF. Categories and Subject Descriptors H.2.8 [Database Applications]: Data Mining General Terms Algorithms, Human Factors, Emerging applications, technology, and issues Keywords Keyword extraction, similarity functions, clustering, information retrieval. 1. INTRODUCTION The National Science Foundation receives over 40,000 proposals a year. Each proposal is reviewed by several external reviewers. It is critical to the mission of the agency and the integrity of the review process that every proposal is reviewed by researchers with the expertise necessary to comment on the merit of the proposal. If there is not a good match between the topic of a proposal and the expertise of the reviewers, then it is possible that a project is funded that will not advance the progress of science or that a very promising proposal is declined. We explore the problem of using data mining technology to assist program directors in the review of proposals. Care is taken to match the technology to the existing workflow of the agency and to use technology to offer suggestions to program directors who ultimately make all decisions. Although this paper reports on reviewing proposals, we argue that the lessons and technology would also apply to the reviewing of papers submitted to conferences and journals. Many proposals are reviewed in panels, i.e., a group of typically 8-15 reviewers who meet to discuss a set of 20-40 related proposals, with each panelist typically reviewing 6-10 proposals. Most proposals are submitted in response to a particular solicitation (e.g., “Information Technology Research”) or to a specific program (e.g., “Human Computer Interaction”). Individual program directors, or for larger solicitations teams of program officers, perform a number of tasks to insure that proposals are reviewed equitably. These tasks include: 1. Divide the proposals into “clusters” of 20-40 related proposals to create panels. 2. Finding reviewers: • Identify potential external reviewers to invite for each panel. • Assign panelists as reviewers of proposals. • If there is not adequate expertise on a panel to review a proposal, obtain “ad hoc” reviews from people with that expertise who are not on a panel. In addition to this lengthy process, reviewers must not have a conflict of interest with proposals they are reviewing (e.g., they may not be from the same department as the proposal’s author), and a diverse group of panelists (both scientifically and demographically) is desirable to insure that multiple perspectives are represented in the review process. Furthermore, due to scheduling or workload conflicts, not every invited reviewer accepts the invitation, requiring an iterative process of inviting a batch of reviewers and then inviting others to fill in gaps after the initial reviewers respond to the invitation. A particular consideration at NSF is that many proposals are multidisciplinary, e.g., mining genome data. To determine if such a proposal is meritorious, it is important to consult some experts with backgrounds in data mining (to insure that the methods proposed are likely to work) and in the biological sciences (to insure that the problem addressed is an important open problem). If all reviewers have expertise in one area, it’s possible that an important problem would be addressed by a technique that isn’t Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. KDD'06, August 20–23, 2006, Philadelphia, Pennsylvania, USA. Copyright 2006 ACM 1-59593-339-5/06/0008...$5.00. 862 Industrial and Government Applications Track Paper