Expert Systems with Applications 37(2010)6948-6956 Contents lists available at Science Direct Expert Systems with Applications ELSEVIER journalhomepagewww.elsevier.com/locate/eswa a decision support approach for assigning reviewers to proposals Yunhong Xua.b,, Jian Ma, Yonghong Sun, Gang Hao, Wei Xu Dingtao Zhao b PR China g Kong Kowloon, Hong Kong PR China d school of Management, Chinese Academy of Sciences. PR China ARTICLE IN FO A BSTRACT Peer review plays an nt role in research project selection at funding agencies. Quality of pe view ree of ma g This kind of matching is largely determined by the process of assigning proposals to reviewers. As the Research project selection umber of proposals submitted to funding agencies continues to grow, the traditional approaches of find- ng appropriate reviewers for each individual proposal fail to satisfy the practical needs. This paper pro- poses an alternative approach whose basic idea is grouping proposals first and then assigning appropriate reviewers for each proposal group. Based on the idea, a decision support approach is proposed to identify valid proposals and reviewers, classify proposals and reviewers according to their disciplines, partition proposals into groups and assign reviewers to proposal groups. A system has been developed based the proposed approach to facilitate the decision making process of assigning proposals to reviewers. e 2010 Elsevier Ltd. All rights reserved. 1 Introduction Traditional assignment methods rely heavily on a single deci sion maker (e.g, panel chair) to manually analyze the title, ab- Selecting appropriate research project for funding is an impor- stract, keywords and other parts of the proposals and then tant task in government funding agencies(Tian, Ma, liu, 2002). identify a set of reviewers who are most likely to review the pro- Research project selection is also a complex one which often in- posals. Furthermore, constraints that should be considered when volves many sub-tasks (Cook, Golany, Kress, Penn, Raviv, assigning proposals to reviewers make the assignment problem 2005). It generally begins with a call for proposals(CFP)which de- more complicated. For example, proposals should be evaluated scribes funding opportunities and requirements of the funding by reviewers who are knowledgeable in the corresponding re- agencies, and then the CfP is distributed to relevant communities, search area, and relationships between applicants and reviewers such as universities and research institutions. Researchers in rele- that could affect the justice of review process (e.g, co-authorships) vant communities who are interested in CFP-related topics can need to be avoided. It presents a great challenge for managers who then submit their proposals to the corresponding funding agencies. are responsible for the assignment task, especially when there are Submitted proposals are validated, compiled and then assigned to large amounts of proposals and reviewers. field experts who are invited as reviewers to provide their opin Several studies have been carried out to solve the reviewer ions. These reviewers comment on the proposals based on their assignment problem from different perspectives, and various ap- orofessional knowledge and with reference to the specific criteria proaches have been proposed. These approaches can be classified issued by the funding agencies. The reviewing results are aggre into two major categories based on their underlying techniques: gated to determine which proposals would be funded (tian et al., approaches based on information retrieval, and approaches based 2002). Assigning proposals to reviewers is one of the most impor ion(Wang, Chen, Miao, 2008). The former ap- tant and challenging tasks(Sun, Ma, Fan, Wang, 2008). It must be proaches focus on using information retrieval techniques to com lone appropriately because assigning proper reviewers to a re- pute the matching degree between proposals and reviewers search proposal ensures reviewers have enough expertise to judge (david Andrew, 2007: Dumais Nielsen, 1992; Hettich Pazza- the quality of the proposal ni,2006: Rodriguez Bollen, 2008). The latter approaches uses theory, modeling and algorithms to formulate and solve the prob lem from mathematical or operational research directions( Cook Technology of China, Hefei, Anhui Province, PR China. Tel. +86 512 87161393 et al, 2005: Janak, Taylor, Floudas, Burka, Mountziaris, 2006 fax:+8651287161381 Merelo-Guervos Castillo-Valdivieso, 2004). Many assignment systems have been developed and implemented to support the 0957-4174 front matter o 2010 Elsevier Ltd. All rights reserved. oi:10.1016/eswa201003.027A decision support approach for assigning reviewers to proposals Yunhong Xu a,b,*, Jian Ma a , Yonghong Sun a , Gang Hao c , Wei Xu d , Dingtao Zhao b aDepartment of Information Systems, City University of Hong Kong, Kowloon, Hong Kong, PR China b Management School, University of Science and Technology of China, Hefei, Anhui Province, PR China cDepartment of Management Sciences, City University of Hong Kong, Kowloon, Hong Kong, PR China d School of Management, Chinese Academy of Sciences, Beijing, PR China article info Keywords: Proposal assignment Reviewer assignment Proposal grouping Research project selection abstract Peer review plays an important role in research project selection at funding agencies. Quality of peer review greatly depends on the degree of matching between reviewers and assigned research proposals. This kind of matching is largely determined by the process of assigning proposals to reviewers. As the number of proposals submitted to funding agencies continues to grow, the traditional approaches of find￾ing appropriate reviewers for each individual proposal fail to satisfy the practical needs. This paper pro￾poses an alternative approach whose basic idea is grouping proposals first and then assigning appropriate reviewers for each proposal group. Based on the idea, a decision support approach is proposed to identify valid proposals and reviewers, classify proposals and reviewers according to their disciplines, partition proposals into groups and assign reviewers to proposal groups. A system has been developed based on the proposed approach to facilitate the decision making process of assigning proposals to reviewers. 2010 Elsevier Ltd. All rights reserved. 1. Introduction Selecting appropriate research project for funding is an impor￾tant task in government funding agencies (Tian, Ma, & Liu, 2002). Research project selection is also a complex one which often in￾volves many sub-tasks (Cook, Golany, Kress, Penn, & Raviv, 2005). It generally begins with a call for proposals (CFP) which de￾scribes funding opportunities and requirements of the funding agencies, and then the CFP is distributed to relevant communities, such as universities and research institutions. Researchers in rele￾vant communities who are interested in CFP-related topics can then submit their proposals to the corresponding funding agencies. Submitted proposals are validated, compiled and then assigned to field experts who are invited as reviewers to provide their opin￾ions. These reviewers comment on the proposals based on their professional knowledge and with reference to the specific criteria issued by the funding agencies. The reviewing results are aggre￾gated to determine which proposals would be funded (Tian et al., 2002). Assigning proposals to reviewers is one of the most impor￾tant and challenging tasks (Sun, Ma, Fan, & Wang, 2008). It must be done appropriately because assigning proper reviewers to a re￾search proposal ensures reviewers have enough expertise to judge the quality of the proposal. Traditional assignment methods rely heavily on a single deci￾sion maker (e.g., panel chair) to manually analyze the title, ab￾stract, keywords and other parts of the proposals and then identify a set of reviewers who are most likely to review the pro￾posals. Furthermore, constraints that should be considered when assigning proposals to reviewers make the assignment problem more complicated. For example, proposals should be evaluated by reviewers who are knowledgeable in the corresponding re￾search area, and relationships between applicants and reviewers that could affect the justice of review process (e.g., co-authorships) need to be avoided. It presents a great challenge for managers who are responsible for the assignment task, especially when there are large amounts of proposals and reviewers. Several studies have been carried out to solve the reviewer assignment problem from different perspectives, and various ap￾proaches have been proposed. These approaches can be classified into two major categories based on their underlying techniques: approaches based on information retrieval, and approaches based on optimization (Wang, Chen, & Miao, 2008). The former ap￾proaches focus on using information retrieval techniques to com￾pute the matching degree between proposals and reviewers (David & Andrew, 2007; Dumais & Nielsen, 1992; Hettich & Pazza￾ni, 2006; Rodriguez & Bollen, 2008). The latter approaches uses theory, modeling and algorithms to formulate and solve the prob￾lem from mathematical or operational research directions (Cook et al., 2005; Janak, Taylor, Floudas, Burka, & Mountziaris, 2006; Merelo-Guervos & Castillo-Valdivieso, 2004). Many assignment systems have been developed and implemented to support the 0957-4174/$ - see front matter 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2010.03.027 * Corresponding author at: Management School, University of Science and Technology of China, Hefei, Anhui Province, PR China. Tel.: +86 512 87161393; fax: +86 512 87161381. E-mail address: xuyunhong@mail.ustc.edu.cn (Y. Xu). Expert Systems with Applications 37 (2010) 6948–6956 Contents lists available at ScienceDirect Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa