《电子商务 E-business》阅读文献：A decision support approach for assigning reviewers to proposals

Expert Systems with Applications 37(2010)6948-6956 Contents lists available at Science Direct Expert Systems with Applications ELSEVIER journalhomepagewww.elsevier.com/locate/eswa a decision support approach for assigning reviewers to proposals Yunhong Xua.b,, Jian Ma, Yonghong Sun, Gang Hao, Wei Xu Dingtao Zhao b PR China g Kong Kowloon, Hong Kong PR China d school of Management, Chinese Academy of Sciences. PR China ARTICLE IN FO A BSTRACT Peer review plays an nt role in research project selection at funding agencies. Quality of pe view ree of ma g This kind of matching is largely determined by the process of assigning proposals to reviewers. As the Research project selection umber of proposals submitted to funding agencies continues to grow, the traditional approaches of find- ng appropriate reviewers for each individual proposal fail to satisfy the practical needs. This paper pro- poses an alternative approach whose basic idea is grouping proposals first and then assigning appropriate reviewers for each proposal group. Based on the idea, a decision support approach is proposed to identify valid proposals and reviewers, classify proposals and reviewers according to their disciplines, partition proposals into groups and assign reviewers to proposal groups. A system has been developed based the proposed approach to facilitate the decision making process of assigning proposals to reviewers. e 2010 Elsevier Ltd. All rights reserved. 1 Introduction Traditional assignment methods rely heavily on a single deci sion maker (e.g, panel chair) to manually analyze the title, ab- Selecting appropriate research project for funding is an impor- stract, keywords and other parts of the proposals and then tant task in government funding agencies(Tian, Ma, liu, 2002). identify a set of reviewers who are most likely to review the pro- Research project selection is also a complex one which often in- posals. Furthermore, constraints that should be considered when volves many sub-tasks (Cook, Golany, Kress, Penn, Raviv, assigning proposals to reviewers make the assignment problem 2005). It generally begins with a call for proposals(CFP)which de- more complicated. For example, proposals should be evaluated scribes funding opportunities and requirements of the funding by reviewers who are knowledgeable in the corresponding re- agencies, and then the CfP is distributed to relevant communities, search area, and relationships between applicants and reviewers such as universities and research institutions. Researchers in rele- that could affect the justice of review process (e.g, co-authorships) vant communities who are interested in CFP-related topics can need to be avoided. It presents a great challenge for managers who then submit their proposals to the corresponding funding agencies. are responsible for the assignment task, especially when there are Submitted proposals are validated, compiled and then assigned to large amounts of proposals and reviewers. field experts who are invited as reviewers to provide their opin Several studies have been carried out to solve the reviewer ions. These reviewers comment on the proposals based on their assignment problem from different perspectives, and various ap- orofessional knowledge and with reference to the specific criteria proaches have been proposed. These approaches can be classified issued by the funding agencies. The reviewing results are aggre into two major categories based on their underlying techniques: gated to determine which proposals would be funded (tian et al., approaches based on information retrieval, and approaches based 2002). Assigning proposals to reviewers is one of the most impor ion(Wang, Chen, Miao, 2008). The former ap- tant and challenging tasks(Sun, Ma, Fan, Wang, 2008). It must be proaches focus on using information retrieval techniques to com lone appropriately because assigning proper reviewers to a re- pute the matching degree between proposals and reviewers search proposal ensures reviewers have enough expertise to judge (david Andrew, 2007: Dumais Nielsen, 1992; Hettich Pazza- the quality of the proposal ni,2006: Rodriguez Bollen, 2008). The latter approaches uses theory, modeling and algorithms to formulate and solve the prob lem from mathematical or operational research directions( Cook Technology of China, Hefei, Anhui Province, PR China. Tel. +86 512 87161393 et al, 2005: Janak, Taylor, Floudas, Burka, Mountziaris, 2006 fax:+8651287161381 Merelo-Guervos Castillo-Valdivieso, 2004). Many assignment systems have been developed and implemented to support the 0957-4174 front matter o 2010 Elsevier Ltd. All rights reserved. oi:10.1016/eswa201003.027

A decision support approach for assigning reviewers to proposals Yunhong Xu a,b,*, Jian Ma a , Yonghong Sun a , Gang Hao c , Wei Xu d , Dingtao Zhao b aDepartment of Information Systems, City University of Hong Kong, Kowloon, Hong Kong, PR China b Management School, University of Science and Technology of China, Hefei, Anhui Province, PR China cDepartment of Management Sciences, City University of Hong Kong, Kowloon, Hong Kong, PR China d School of Management, Chinese Academy of Sciences, Beijing, PR China article info Keywords: Proposal assignment Reviewer assignment Proposal grouping Research project selection abstract Peer review plays an important role in research project selection at funding agencies. Quality of peer review greatly depends on the degree of matching between reviewers and assigned research proposals. This kind of matching is largely determined by the process of assigning proposals to reviewers. As the number of proposals submitted to funding agencies continues to grow, the traditional approaches of find￾ing appropriate reviewers for each individual proposal fail to satisfy the practical needs. This paper pro￾poses an alternative approach whose basic idea is grouping proposals first and then assigning appropriate reviewers for each proposal group. Based on the idea, a decision support approach is proposed to identify valid proposals and reviewers, classify proposals and reviewers according to their disciplines, partition proposals into groups and assign reviewers to proposal groups. A system has been developed based on the proposed approach to facilitate the decision making process of assigning proposals to reviewers. 2010 Elsevier Ltd. All rights reserved. 1. Introduction Selecting appropriate research project for funding is an impor￾tant task in government funding agencies (Tian, Ma, & Liu, 2002). Research project selection is also a complex one which often in￾volves many sub-tasks (Cook, Golany, Kress, Penn, & Raviv, 2005). It generally begins with a call for proposals (CFP) which de￾scribes funding opportunities and requirements of the funding agencies, and then the CFP is distributed to relevant communities, such as universities and research institutions. Researchers in rele￾vant communities who are interested in CFP-related topics can then submit their proposals to the corresponding funding agencies. Submitted proposals are validated, compiled and then assigned to field experts who are invited as reviewers to provide their opin￾ions. These reviewers comment on the proposals based on their professional knowledge and with reference to the specific criteria issued by the funding agencies. The reviewing results are aggre￾gated to determine which proposals would be funded (Tian et al., 2002). Assigning proposals to reviewers is one of the most impor￾tant and challenging tasks (Sun, Ma, Fan, & Wang, 2008). It must be done appropriately because assigning proper reviewers to a re￾search proposal ensures reviewers have enough expertise to judge the quality of the proposal. Traditional assignment methods rely heavily on a single deci￾sion maker (e.g., panel chair) to manually analyze the title, ab￾stract, keywords and other parts of the proposals and then identify a set of reviewers who are most likely to review the pro￾posals. Furthermore, constraints that should be considered when assigning proposals to reviewers make the assignment problem more complicated. For example, proposals should be evaluated by reviewers who are knowledgeable in the corresponding re￾search area, and relationships between applicants and reviewers that could affect the justice of review process (e.g., co-authorships) need to be avoided. It presents a great challenge for managers who are responsible for the assignment task, especially when there are large amounts of proposals and reviewers. Several studies have been carried out to solve the reviewer assignment problem from different perspectives, and various ap￾proaches have been proposed. These approaches can be classified into two major categories based on their underlying techniques: approaches based on information retrieval, and approaches based on optimization (Wang, Chen, & Miao, 2008). The former ap￾proaches focus on using information retrieval techniques to com￾pute the matching degree between proposals and reviewers (David & Andrew, 2007; Dumais & Nielsen, 1992; Hettich & Pazza￾ni, 2006; Rodriguez & Bollen, 2008). The latter approaches uses theory, modeling and algorithms to formulate and solve the prob￾lem from mathematical or operational research directions (Cook et al., 2005; Janak, Taylor, Floudas, Burka, & Mountziaris, 2006; Merelo-Guervos & Castillo-Valdivieso, 2004). Many assignment systems have been developed and implemented to support the 0957-4174/$ - see front matter 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2010.03.027 * Corresponding author at: Management School, University of Science and Technology of China, Hefei, Anhui Province, PR China. Tel.: +86 512 87161393; fax: +86 512 87161381. E-mail address: xuyunhong@mail.ustc.edu.cn (Y. Xu). Expert Systems with Applications 37 (2010) 6948–6956 Contents lists available at ScienceDirect Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa

Y. Xu et al/ Expert Systems with Applications 37(2010)6948-6956 6949 assignment of candidate reviewers for each proposal, using either the aforementioned decision process, including the matching de- one or both of these approaches(Papagelis, Plexousakis, Niko- gree calculation model, the proposal grouping model and assignment model to assign reviewers to proposal groups, etc. According to our literature review, the majority of existing ap- The remainder of this paper is organized as follows. The re- proaches either from information retrieval stream or from optimi- search background is introduced in Section 2. Section 3 proposes ation stream are based on the same strategy which seeks an approach for solving the assignment problem. Section 4 pre earch appropriate reviewers for each individual proposal (as sents a prototype system based on the proposed approach. Section shown in Part A of Fig. 1). This strategy is in an attempt to ensure 5 validates the proposed approach and discusses the potential that every reviewer has sufficient expertise to evaluate the merits application in government funding agencies. Section 6 concludes of the proposals assigned to him/her. This strategy has several lim- the paper. itations: first, selecting appropriate reviewers for every individual proposal is time-consuming. For large funding agencies, the num ber of proposals received and the number of qualified reviewers 2. Backgro can be very large. Besides, the review process must be usually com- pleted with a tight deadline(Wang et al, 2008). Thus, partitioning 2.1. Background of NSFC proposals into groups and searching reviewers for each proposal up can be an efficient alternative. The time spent on reviewer one of the largest and most reputable research funding agen- assignment can be reduced by assigning reviewers to proposal cies in China, NSFC (National Natural Science Foundation of China) groups instead of individual proposals. Second, assigning individ- aims to fund research projects that have great potential of ual proposals to reviewers can hardly meet the specific require- tific and social impacts. NSFC has one general office, five bureaus, nents of the funding agencies, e.g. balancing the backgrounds of and seven scientific departments(Tian, Ma, Liang, Kwok, &Liu, plicants and affiliations of proposals assigned to reviewers. The 2005). The general office and bure mainly in charge of pol limitations of this strategy call for another approach in this con- icy making, operational management, administrative work and re- text. In this research, a new assignment strategy is proposed which lated affaires. The scientific departments are the key parts of NSFC. is based on grouping proposals first and then search qualified and they are responsible for the selection and management of re- reviewers for each proposal group(shown in Fig. 1B). To the best search projects. The scientific departments include Department of of our knowledge, it has never been addressed in the literature. Mathematicaland Physical Sciences, Department of Chemical Sciences, Furthermore, previous research usually used exact algorithms Department of Life Sciences, Department of Earth Sciences, Depart to solve the reviewer assignment problem. But it may be difficult ment of Engineering and Materials Sciences, Department of Informa- to solve the assignment problem using exact algorithms when tion Sciences and Department of Management Sciences. These seven the numbers of both proposals and reviewers are very large. Genet- scientific departments are further divided into divisions focusing various assignment and combinational applications where it dem- Management Sciences is further divided into three divisions: Man- onstrated satisfactory performances(Deep Das, 2008: Harper, de agement Science and Engineering, Macro Management and Policy Senna, vieira, Shahani, 2005: Huang Lim, 2006). While genetic and Business Administration algorithms in their elementary forms can be designed to tackle a here are various categories of programs in NSFC. The general real-world problem, the incorporation of domain knowledge and Program(including Project for young scientists'fund and Project local search techniques may improve the computational perfor- for developing regions)is the major one. The number of proposals mance significantly submitted to NSFC for the General Program increased dramati- This paper proposes an integrated approach assisting in assign- cally from 23, 636 in year 2001 to 73, 785 in year 2008(see ing proposals to reviewers where proposals need to be partitioned Fig. 2). Note that the average funded rate( funded over submit- into groups. The proposed approach facilitates the reviewer assign- ted )in 2005-2008 is only about 18%. In order to support and fi- nent through the following steps: identify valid proposals and nance the most promising proposals a limited budget, a viewers, classify proposals and reviewers according to their dis- fair and unbiased project selection pi ciplines, partition proposals into groups and assign reviewers to one of the most important tasks is to approprlate review proposal groups. Knowledge rules and models are used to support ers to proposals Proposals Reviewers Proposals Grouping Reviewers (A)Individual proposal assignment (B)Proposal assignment based on grouping them first

assignment of candidate reviewers for each proposal, using either one or both of these approaches (Papagelis, Plexousakis, & Niko￾laou, 2005; Sun et al., 2008). According to our literature review, the majority of existing ap￾proaches either from information retrieval stream or from optimi￾zation stream are based on the same strategy which seeks to search appropriate reviewers for each individual proposal (as shown in Part A of Fig. 1). This strategy is in an attempt to ensure that every reviewer has sufficient expertise to evaluate the merits of the proposals assigned to him/her. This strategy has several lim￾itations: first, selecting appropriate reviewers for every individual proposal is time-consuming. For large funding agencies, the num￾ber of proposals received and the number of qualified reviewers can be very large. Besides, the review process must be usually com￾pleted with a tight deadline (Wang et al., 2008). Thus, partitioning proposals into groups and searching reviewers for each proposal group can be an efficient alternative. The time spent on reviewer assignment can be reduced by assigning reviewers to proposal groups instead of individual proposals. Second, assigning individ￾ual proposals to reviewers can hardly meet the specific require￾ments of the funding agencies, e.g. balancing the backgrounds of applicants and affiliations of proposals assigned to reviewers. The limitations of this strategy call for another approach in this con￾text. In this research, a new assignment strategy is proposed which is based on grouping proposals first and then search qualified reviewers for each proposal group (shown in Fig. 1B). To the best of our knowledge, it has never been addressed in the literature. Furthermore, previous research usually used exact algorithms to solve the reviewer assignment problem. But it may be difficult to solve the assignment problem using exact algorithms when the numbers of both proposals and reviewers are very large. Genet￾ic algorithm (GA) has been widely used as a search algorithm in various assignment and combinational applications where it dem￾onstrated satisfactory performances (Deep & Das, 2008; Harper, de Senna, Vieira, & Shahani, 2005; Huang & Lim, 2006). While genetic algorithms in their elementary forms can be designed to tackle a real-world problem, the incorporation of domain knowledge and local search techniques may improve the computational perfor￾mance significantly. This paper proposes an integrated approach assisting in assign￾ing proposals to reviewers where proposals need to be partitioned into groups. The proposed approach facilitates the reviewer assign￾ment through the following steps: identify valid proposals and reviewers, classify proposals and reviewers according to their dis￾ciplines, partition proposals into groups and assign reviewers to proposal groups. Knowledge rules and models are used to support the aforementioned decision process, including the matching de￾gree calculation model, the proposal grouping model and the assignment model to assign reviewers to proposal groups, etc. The remainder of this paper is organized as follows. The re￾search background is introduced in Section 2. Section 3 proposes an approach for solving the assignment problem. Section 4 pre￾sents a prototype system based on the proposed approach. Section 5 validates the proposed approach and discusses the potential application in government funding agencies. Section 6 concludes the paper. 2. Background 2.1. Background of NSFC As one of the largest and most reputable research funding agen￾cies in China, NSFC (National Natural Science Foundation of China) aims to fund research projects that have great potential of scien￾tific and social impacts. NSFC has one general office, five bureaus, and seven scientific departments (Tian, Ma, Liang, Kwok, & Liu, 2005). The general office and bureaus are mainly in charge of pol￾icy making, operational management, administrative work and re￾lated affaires. The scientific departments are the key parts of NSFC, and they are responsible for the selection and management of re￾search projects. The scientific departments include Department of Mathematical and Physical Sciences, Department of Chemical Sciences, Department of Life Sciences, Department of Earth Sciences, Depart￾ment of Engineering and Materials Sciences, Department of Informa￾tion Sciences and Department of Management Sciences. These seven scientific departments are further divided into divisions focusing on more specific research areas. For example, the Department of Management Sciences is further divided into three divisions: Man￾agement Science and Engineering, Macro Management and Policy and Business Administration. There are various categories of programs in NSFC. The General Program (including Project for young scientists’ fund and Project for developing regions) is the major one. The number of proposals submitted to NSFC for the General Program increased dramati￾cally from 23,636 in year 2001 to 73,785 in year 2008 (see Fig. 2). Note that the average funded rate (funded over submit￾ted) in 2005–2008 is only about 18%. In order to support and fi- nance the most promising proposals within a limited budget, a fair and unbiased project selection process is necessary where one of the most important tasks is to assign appropriate review￾ers to proposals. (A) Individual proposal assignment (B) Proposal assignment based on grouping them first Proposals Reviewers Proposals’ Grouping Reviewers Fig. 1. Two strategies for assigning proposals. Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956 6949

Y. Xu et aL/ Expert Systems with Applications 37(2010)6948-6956 80000 O Submitted propos 73785 9329 39665 20000 2003 20042005200620072008 Year Fig.2.thEnumberofsubmittedandfundedproposalsfrom2001to2008.Sourcehttp://www.nsfc.gov.cn 2.2. Proposal assignment at NSFC In most situations, reviewing proposal is done on a voluntary basis. If the workload for a reviewer is too heavy. reviewers may refuse The proposal assignment at NSFC underwent two phases and the work assigned to them or pass on the job to their colleagues employed different strategies in the past as shown in Fig. 1. Ini- or subordinates. (4)For a fair review process, the number of tially, it adopted the strategy to find appropriate reviewers for each reviewers assigned to each proposal group should be about the dividual proposal. Subsequently, with large number of proposals same. For example, General Program in NSFC requires that each submitted and with special requirements of NSFC (e.g. balancing proposal be reviewed by five reviewers. (5) Reviewers who have the background information of applicants, balancing the affilia- conflicts of interests with an applicant in a proposal group should tions of proposals etc. ) it was necessary to group proposals first not be assigned to that group, e.g., the reviewers who co-authored and then search qualified reviewers for each proposal group a paper before with an applicant should be excluded f the review p ess, several issues should be considered when assigning reviewers 3. The proposed approach to proposal groups in NSFC:(1)The composition of proposals in The basic idea of our proposed approach is p proposals ants of proposals in each proposal group are from different uni- first and then assign reviewers to each proposal t versities or institutions. (2)The number of proposals in each proposed approach is composed of six steps as shown in Fig. 3. A group should be about the same. otherwise, there are proposal detailed explanation of each step is provided in the following groups with large group size and small group size. Under this con- suDs text, it is possible that some reviewers receive a large number of proposals, even when the number of proposal groups assigned to 3. 1. Step 1. Identifying valid proposals and reviewe each reviewer is same. (3)To ensure that reviewers have sufficient time to complete their review task, the number of proposal groups The aim of this step is to identify valid proposals and reviewers assigned to each reviewer should not exceed a certain number In NSFC, there are various types of programs such as General Pro (e.g. 16 proposals in 2 proposal groups each with 8 proposals ). gram, Distinguished Young Scholar Program, etc. The requirements validity identification Classification based on Grouping proposals up proposals based on ify valid proposals an fy proposals and reviewers research disciplines Result adjustment Reviewer assignment calculation Adjust assignment results by Assign reviewers to proposal between proposal groups and groups Fig 3. The assignment process

2.2. Proposal assignment at NSFC The proposal assignment at NSFC underwent two phases and employed different strategies in the past as shown in Fig. 1. Ini￾tially, it adopted the strategy to find appropriate reviewers for each individual proposal. Subsequently, with large number of proposals submitted and with special requirements of NSFC (e.g., balancing the background information of applicants, balancing the affilia￾tions of proposals etc.), it was necessary to group proposals first and then search qualified reviewers for each proposal group. The assignment results reflect the fairness of the review process to some extent. In order to ensure a fair and unbiased review pro￾cess, several issues should be considered when assigning reviewers to proposal groups in NSFC: (1) The composition of proposals in each group need to be as diverse as possible. For example, appli￾cants of proposals in each proposal group are from different uni￾versities or institutions. (2) The number of proposals in each group should be about the same. Otherwise, there are proposal groups with large group size and small group size. Under this con￾text, it is possible that some reviewers receive a large number of proposals, even when the number of proposal groups assigned to each reviewer is same. (3) To ensure that reviewers have sufficient time to complete their review task, the number of proposal groups assigned to each reviewer should not exceed a certain number (e.g., 16 proposals in 2 proposal groups each with 8 proposals). In most situations, reviewing proposal is done on a voluntary basis. If the workload for a reviewer is too heavy, reviewers may refuse the work assigned to them or pass on the job to their colleagues or subordinates. (4) For a fair review process, the number of reviewers assigned to each proposal group should be about the same. For example, General Program in NSFC requires that each proposal be reviewed by five reviewers. (5) Reviewers who have conflicts of interests with an applicant in a proposal group should not be assigned to that group, e.g., the reviewers who co-authored a paper before with an applicant should be excluded. 3. The proposed approach The basic idea of our proposed approach is to group proposals first and then assign reviewers to each proposal group. The entire proposed approach is composed of six steps as shown in Fig. 3. A detailed explanation of each step is provided in the following subsections. 3.1. Step 1. Identifying valid proposals and reviewers The aim of this step is to identify valid proposals and reviewers. In NSFC, there are various types of programs such as General Pro￾gram, Distinguished Young Scholar Program, etc. The requirements 27590 31791 39665 49329 58811 4435 5808 6359 7711 9111 10271 11608 14355 64649 23636 73785 Year Number Submitted proposals Funded projects Fig. 2. The number of submitted and funded proposals from 2001 to 2008. Source: http://www.nsfc.gov.cn Validity identification Identify valid proposals and reviewers Grouping proposals Group proposals based on criteria predetermined by managers Matching degree calculation Calculate matching degree between proposal groups and reviewers Result adjustment Adjust assignment results by managers Reviewer assignment Assign reviewers to proposal groups Classification based on disciplines Classify proposals and reviewers according to their research disciplines Fig. 3. The assignment process. 6950 Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956

Y. Xu et al/ Expert Systems with Applications 37(2010)6948-6956 Table Sample rules for identifying valid proposals. Sample rules for classifying reviewers. Rules for general progra Suppose that there are k discipline areas, and Dx g denotes reviewer j (k=1,2,- K): Assume that there are reviewers THEN C=1, 2,-J): g denotes the mth research area of reviewer R] S denotes the IF Unregistered(PD)or Unregistered (affiliated organization eviewer sets of discipline area k, then Sk can be calculated as follows: THEN Status= incomplete or k=1 to K IF Position of Pl= Full-Time Graduate student Forj=1 to J OR Position of Pl= Part-Time Graduate Student If R #(Categorized reviewers] Then umber of discipline areas of reviewer Ry IF PI in Bad Reputation List Record If R f= D Then THEN Status undetermined ill be added to Sk End If Next for applicants vary for different programs. For example, in the dis en tinguished Young Scholar Program, the age of the applicants should R, will be added to set(Categorized reviewers] be less than 35 years old. After proposals are submitted to NSFC, End If knowledge rules(see Table 1 for an example)can help managers End if plete, undetermined and valid. If a proposal is valid. it can be di- Next incomplete and undetermined proposals are rejected or returned to applicants for further considerations, and for possible resubmis- weights are determined in advance. The detail of the model and al,2002) the solution are presented in our previous research(Fan, Chen, iewers are selected based on their recent research publica Ma, Zhu, 2008). The goal of this step is to guarantee that the com- in research, e.g. more published papers and more funded projects, of NFc f the proposal groups is consistent with the requirements tions and funded projects. Experts who have better track records position have higher probability to be selected as reviewers, while those who are not active in the past few years are usually excluded 3. 4. Step 4. Calculating the matching degree (Sun, Ma, Fan, Wang. 2008). Knowledge rules can be used to facilitate identifying valid reviewers(see Table 2 for an example). One of the most important issues in reviewer assignment proce dure is to determine how good a reviewer matches a proposal 3. 2. Step 2. Classifying proposals and reviewers in terms of their group in terms of the extent to which the reviewer is familiar with he research area of the corresponding proposals. This judgment can be extracted ad hoc via expert knowledge, but it is impractical After identifying valid reviewers and proposals, the next step is to do so if there are a large number of proposal group and reviewer lassify these proposals and reviewers according to their disc pline areas. As mentioned before, there are seven scientific depart reviewers can be calculated as follows ents in NSFC. The departments are composed of different divisions. The divisions are further divided into many disciplines 3. 4.1. Calculating the matching degree between proposal groups and (Sun et al., 2008). NSFC maintains a dictionary of discipline areas reviewers which is arranged in a tree structure. This discipline dictionary is To determine the expertise level of reviewers in NSFC, review released to applicants and reviewers. It facilitates applicants to de- ers are required to fill in a form declaring their expertise level in clare more carefully the discipline areas that their proposals belong the discipline areas with supporting evidence of publication re- to The reviewers also declare their research areas according to the cords. The expertise level of the reviewers is measured along a same dictionary. However, due to large number of proposals, scale using 1-3, where level 3 signifies that a reviewer is very reviewers and discipline areas, it is a time-consuming process for familiar with the corresponding research area, level 2 familiar the managers to manually create the appropriate matching pro- and level 1 less familiar. The matching degree between a specific osal sets and reviewer sets in each discipline area. The knowledge reviewer and a proposal group can be calculated as follows ules can thus be used to aid the managers to classify proposal If we use Ck to denote the expertise level of reviewer j to disci- and reviewers(see Table 3 as an example). 3.3. Step 3. Partitioning proposals into proposal groups 3, if reviewer is"very familiar"with discipline area k Ck= 2, if reviewer is"familiar"with discipline area k As mentioned in Section 2, it is necessary to consider the fund 1. if reviewer is"less familiar"with discipline area k ing policies and requirements of NSFC in the initial grouping pro cess before assigning proposals to reviewers. The attributes that dgk is a binary value whose value is 1 if proposal g belongs to disci- are considered when grouping proposals and their corresponding pline k, otherwise, it is zero i proposal group(i=1,2,., n). The matching degree between proposal group i and reviewerjis Table 2 Sample rules for identifying valid reviewers. obtained by Mingei Max( Ck*dgk). The matching degree between an individual proposal g(in group i) and reviewer j is measured by ot activ Max(Ci*dgk). It uses one of a specific proposal's discipline with which a reviewer is most familiar to obtain the matching degree ad Reputation Record us invalid between a proposal and a reviewer while the minimal value of matching degree between a reviewer and proposals in a particular

for applicants vary for different programs. For example, in the Dis￾tinguished Young Scholar Program, the age of the applicants should be less than 35 years old. After proposals are submitted to NSFC, knowledge rules (see Table 1 for an example) can help managers to classify the proposals into four categories, i.e., invalid, incom￾plete, undetermined and valid. If a proposal is valid, it can be di￾rectly passed to the next selection phase. While invalid, incomplete and undetermined proposals are rejected or returned to applicants for further considerations, and for possible resubmis￾sion (Tian et al., 2002). Reviewers are selected based on their recent research publica￾tions and funded projects. Experts who have better track records in research, e.g. more published papers and more funded projects, have higher probability to be selected as reviewers, while those who are not active in the past few years are usually excluded (Sun, Ma, Fan, & Wang, 2008). Knowledge rules can be used to facilitate identifying valid reviewers (see Table 2 for an example). 3.2. Step 2. Classifying proposals and reviewers in terms of their discipline areas After identifying valid reviewers and proposals, the next step is to classify these proposals and reviewers according to their disci￾pline areas. As mentioned before, there are seven scientific depart￾ments in NSFC. The departments are composed of different divisions. The divisions are further divided into many disciplines (Sun et al., 2008). NSFC maintains a dictionary of discipline areas which is arranged in a tree structure. This discipline dictionary is released to applicants and reviewers. It facilitates applicants to de￾clare more carefully the discipline areas that their proposals belong to. The reviewers also declare their research areas according to the same dictionary. However, due to large number of proposals, reviewers and discipline areas, it is a time-consuming process for the managers to manually create the appropriate matching pro￾posal sets and reviewer sets in each discipline area. The knowledge rules can thus be used to aid the managers to classify proposals and reviewers (see Table 3 as an example). 3.3. Step 3. Partitioning proposals into proposal groups As mentioned in Section 2, it is necessary to consider the fund￾ing policies and requirements of NSFC in the initial grouping pro￾cess before assigning proposals to reviewers. The attributes that are considered when grouping proposals and their corresponding weights are determined in advance. The detail of the model and the solution are presented in our previous research (Fan, Chen, Ma, & Zhu, 2008). The goal of this step is to guarantee that the com￾position of the proposal groups is consistent with the requirements of NSFC. 3.4. Step 4. Calculating the matching degree One of the most important issues in reviewer assignment proce￾dure is to determine how good a reviewer matches a proposal group in terms of the extent to which the reviewer is familiar with the research area of the corresponding proposals. This judgment can be extracted ad hoc via expert knowledge, but it is impractical to do so if there are a large number of proposal group and reviewer combinations. The matching degree between proposal groups and reviewers can be calculated as follows. 3.4.1. Calculating the matching degree between proposal groups and reviewers To determine the expertise level of reviewers in NSFC, review￾ers are required to fill in a form declaring their expertise level in the discipline areas with supporting evidence of publication re￾cords. The expertise level of the reviewers is measured along a scale using 1–3, where level 3 signifies that a reviewer is very familiar with the corresponding research area, level 2 familiar and level 1 less familiar. The matching degree between a specific reviewer and a proposal group can be calculated as follows. If we use Cjk to denote the expertise level of reviewer j to disci￾pline area k, then: Cjk ¼ 3; if reviewer is “very familiar” with discipline area k 2; if reviewer is “familiar” with discipline area k 1; if reviewer is “less familiar” with discipline area k 8 >: dgk is a binary value whose value is 1 if proposal g belongs to disci￾pline k, otherwise, it is zero. i proposal group (i = 1,2,...,n). The matching degree between proposal group i and reviewer j is obtained by Ming2iMaxk(Cjk dgk). The matching degree between an individual proposal g (in group i) and reviewer j is measured by Max k ðCjk dgkÞ. It uses one of a specific proposal’s discipline with which a reviewer is most familiar to obtain the matching degree between a proposal and a reviewer. While the minimal value of matching degree between a reviewer and proposals in a particular Table 1 Sample rules for identifying valid proposals. Rules for general program IF Number of on-going projects + Number of Submitted proposals > 2 THEN Status = Undetermined IF Unregistered (PI) or Unregistered (affiliated organization) THEN Status = incomplete IF Position of PI = Full-Time Graduate Student OR Position of PI = Part-Time Graduate Student OR Position of PI = Retired THEN Status = Invalid IF PI in Bad Reputation List Record THEN Status = undetermined Table 2 Sample rules for identifying valid reviewers. IF reviewer not active in recent five years THEN Status = undetermined IF reviewer in Bad Reputation Record THEN Status = invalid Table 3 Sample rules for classifying reviewers. Suppose that there are K discipline areas, and Dk denotes discipline area k (k = 1,2,...,K); Assume that there are J reviewers; Rj denotes reviewer j (j = 1,2,...,J); Rn j denotes the nth research area of reviewer Rj, Sk denotes the reviewer sets of discipline area k, then Sk can be calculated as follows: For k = 1 to K For j = 1 to J If Rj R {Categorized reviewers} Then N = Number of discipline areas of reviewer Rj For n = 1 to N If Rn j ¼ Dk Then Rj will be added to Sk End Loop End If Next If n = N Then Rj will be added to set {Categorized reviewers} End If End If Next Next Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956 6951

Y. Xu et aL/ Expert Systems with Applications 37(2010)6948-6956 group is used to measure the matching degree between a proposal Initialize population P(r) based on GRASP group and a reviewer. 3. 4.2. Eliminating possible conflicts of interests Select parents from P(1) In order to ensure the fairness of the project selection, potential conflicts of interests between reviewers and applicants must be avoided as much as possible. For example, the applicants and on operator on parents eviewers cannot come form same institution or have co-authe relationship. If conflicts of interests between any applicant of proposal group and a reviewer exist, then the matching degree be- ween this reviewer and the proposal group is set to be o or a neg- Applyrepair operator on infeasible children tive number so as to avoid the matching. 3.5. Step 5. Assigning reviewers to proposal groups Use local search to improve the quality of children In step 5, the proposal groups generated in step 3 are assigned Evaluate the fitness of childre to reviewers based on the matching degree between proposal whether groups and reviewers calculated in step 4 and other considerations as mentioned in Section 2. Recall that the research problem is to find the most qualified reviewers for each proposal group. That t=t+I is, maximize the matching degree between the proposal groups d reviewers while satisfying the funding constraints. This is ex- pressed in an assignment model as discussed in Section 3.5.1. Then a hybrid approach based on gRASP (Greedy randomized Adaptive Search Procedure)and genetic algorithm is proposed to solve the assignment model for the large volume problems in the following 3.5.1. The assignment model Let n be the number of proposal groups needed to be assigned m be the total number of available reviewers, i be the index of pro- posal groups (i=1,2,., n) be G=1, 2,..., m), a be the required number of reviewers for evaluat- Fig 4. The structure of the hy brid GRASP and gA ing each proposal group(in most funding it is usually 3 or 5) and b be the maximum number of proposal groups that a re- ewer can be assigned to. xy is a binary variable whose value is g algorithm may not be superior to other heuristic 1 if group i is assigned to reviewer j, and 0 otherwise. The mathe algorit when a genetic algorithm is combined with a local matical model can be formulated as follows: search he efficiency of the algorithm can be improve Ahuja, Orlin, Tiwari, 2000). A combination of hybrid GRASI Maximize∑∑Mn2sMax(C*4)k and genetic algorithm can be used to solve the above mathematical model while overcoming the weaknesses of the general GA. It Subject to a vi (1) cludes the steps of initialization, selection, crossover, mutation and other operators as shown in Fig 4 ∑刈≤b可 3.5.3. The hybrid GRASP and GA Xi=0.1 (3) 3.5.3.1. Representation. A matrix whose elements are either 0 1 can be used to represent a solution to the reviewer assignmen Constraint(1)guarantees that the number of proposal groups as- to the jth reviewer, then the value of the corresponding element of signed to each reviewer should not exceed a predetermined number to ensure the review quality when the number of proposals and the solution matrix is 1 or o otherwise. The combination of 0 and 1 reviewers is small, the exact algorithm can work ell to solvi variables in the matrix do are as- signed to which reviewers(as shown in Fig. 5). When the matrix ciently by exact algorithm in reasonable time due to the dual prob- is too sparse, another representation technique is used where the lems of large volume of proposals and reviewers and limited available time. Thus, we propose here a heuristic algorithm based on a hybrid GRASP and genetic algorithm for an efficient solution R1R2R3….R。 when the number of proposals and reviewers is large G1101 G,01 3.5.2. The proposed solution to the assignment model G3110 Genetic algorithms have been applied to several kinds of com- tory performance(Huang Lim, 2006: Nebro, Luque, Luna, Alba 2008: Tang, Quek, Ng, 2005). It has also been shown that a Fig. 5. Representation of an individuals chromosome

group is used to measure the matching degree between a proposal group and a reviewer. 3.4.2. Eliminating possible conflicts of interests In order to ensure the fairness of the project selection, potential conflicts of interests between reviewers and applicants must be avoided as much as possible. For example, the applicants and reviewers cannot come form same institution or have co-author relationship. If conflicts of interests between any applicant of a proposal group and a reviewer exist, then the matching degree be￾tween this reviewer and the proposal group is set to be 0 or a neg￾ative number so as to avoid the matching. 3.5. Step 5. Assigning reviewers to proposal groups In step 5, the proposal groups generated in step 3 are assigned to reviewers based on the matching degree between proposal groups and reviewers calculated in step 4 and other considerations as mentioned in Section 2. Recall that the research problem is to find the most qualified reviewers for each proposal group. That is, maximize the matching degree between the proposal groups and reviewers while satisfying the funding constraints. This is ex￾pressed in an assignment model as discussed in Section 3.5.1. Then a hybrid approach based on GRASP (Greedy Randomized Adaptive Search Procedure) and genetic algorithm is proposed to solve the assignment model for the large volume problems in the following subsections. 3.5.1. The assignment model Let n be the number of proposal groups needed to be assigned, m be the total number of available reviewers, i be the index of pro￾posal groups (i = 1,2,...,n),j be the index of reviewers (j = 1,2,...,m), a be the required number of reviewers for evaluat￾ing each proposal group (in most funding agencies, it is usually 3 or 5), and b be the maximum number of proposal groups that a re￾viewer can be assigned to. xij is a binary variable whose value is 1 if group i is assigned to reviewer j, and 0 otherwise. The mathe￾matical model can be formulated as follows: Maximize X i2I X j2J ½Ming2iMaxkðCjk dgkÞxij Subject to X j xij ¼ a 8i ð1Þ X i xij 6 b 8j ð2Þ xij ¼ 0; 1 ð3Þ Constraint (1) expresses the requirement that each group of propos￾als should be evaluated by a reviewers. Constraint (2) implies that each reviewer should be assigned at most to b proposal groups. Constraint (1) guarantees that the number of proposal groups as￾signed to each reviewer should not exceed a predetermined number to ensure the review quality. When the number of proposals and reviewers is small, the exact algorithm can work very well to solve the above model. However, the above model cannot be solved effi- ciently by exact algorithm in reasonable time due to the dual prob￾lems of large volume of proposals and reviewers and limited available time. Thus, we propose here a heuristic algorithm based on a hybrid GRASP and genetic algorithm for an efficient solution when the number of proposals and reviewers is large. 3.5.2. The proposed solution to the assignment model Genetic algorithms have been applied to several kinds of com￾binatorial optimization problems and have demonstrated satisfac￾tory performance (Huang & Lim, 2006; Nebro, Luque, Luna, & Alba, 2008; Tang, Quek, & Ng, 2005). It has also been shown that a general genetic algorithm may not be superior to other heuristic algorithms. But when a genetic algorithm is combined with a local search strategy, the efficiency of the algorithm can be improved (Ahuja, Orlin, & Tiwari, 2000). A combination of hybrid GRASP and genetic algorithm can be used to solve the above mathematical model while overcoming the weaknesses of the general GA. It in￾cludes the steps of initialization, selection, crossover, mutation and other operators as shown in Fig. 4. 3.5.3. The hybrid GRASP and GA 3.5.3.1. Representation. A matrix whose elements are either 0 or 1 can be used to represent a solution to the reviewer assignment problem, which is referred to as chromosome in genetic algorithm. Using R1, R2,...,Rm, and G1, G2,...,Gn to represent reviewers and proposal groups, respectively, if the ith proposal group is assigned to the jth reviewer, then the value of the corresponding element of the solution matrix is 1 or 0 otherwise. The combination of 0 and 1 variables in the matrix determines which proposal groups are as￾signed to which reviewers (as shown in Fig. 5). When the matrix is too sparse, another representation technique is used where the Initialize population P(t) based on GRASP Select parents from P(t) Apply crossover and mutation operator on parents to yield children Apply repair operator on infeasible children Use local search to improve the quality of children Evaluate the fitness of children to determine whether replace the individuals or not t=t+1 Stop criterion satified Stop Yes No Fig. 4. The structure of the hybrid GRASP and GA. 123 1 2 3 ... 1 0 1 ... 0 0 1 1 ... 0 1 1 0 ... 0 ... ... ... ... ... ... 0 0 1 ... 1 m n RRR R G G G G Fig. 5. Representation of an individual’s chromosome. 6952 Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956

Y. Xu et al/ Expert Systems with Applications 37(2010)6948-6956 1"element is taken place by the code numbers of the reviewers, 2007: Feo Resende, 1995). The neighborhood structure N for a and"0"element is deleted to reduce the space size solution s to a specific problem is a subset of the solutions N(s).Gi- ven two solutions represented by two matrices p and g the differ 3.5.3.2. Initialization. The initial population is generated in the con- ence between p and q can be defined as: &(pq)=(i,D)lp(i, J) struction phase of GRASP. In the construction phase of GRASP, a * q(i,)), and the distance between p and g is defined to be feasible solution is iteratively constructed, one element at a time d(p, q)= 8(p, q) For the majority of GRASP, the most commonly (Aiex, Resende, Pardalos, Toraldo, 2005: Alvarez-Valdes, Crespo, used neighborhood structure is the so-called k-exchange neighbor- Tamarit,&Villa, 2008). The element added to the solution is ran- hood structure. The k-exchange neighborhood of a solution p can domly selected from a restricted candidate list(Resende, Mart, be defined allego, duarte, 2008), which is formed by ordering all candidate lements according to the benefit(here it is the matching degree p)={qld(p,q)≤k}, where2≤k≤min(n,m between a proposal group and a reviewer)of selecting this element This neighborhood structure is also employed in the proposed is above the threshold can be selected given a n m reviewer and method. However, the common exchange technique of just swap- proposal group assignment matrix P. a row index ie(1,.,n). and assignment problem because it leads to infeasible solutions. Thus a column indexjE[1,., m), and let X(P, i, j) be the following com- a new local search mechanism is proposed for this problem. patibility function Each step of the local search is composed of two operations X(P ii-1, for matrix p, assigning group i to reviewerjis allowed deleting and inserting operation. Deleting operation is based on lO. otherwise eliminating some nonzero elements and changing their values from X(P, i.) can be used as an index to determine whether assigning a forming an incomplete assignment matrix. The inserting operation proposal group to a reviewer is available. a candidate list is com- is based on completely constructing the incomplete assignment ment pair can be added to the candidate list. Chromosomes (or phase sing the constructing phase described in the initialization posed of all available elements. If X(P, ij)=1, then this(i, )assign- matrix solutions)are generated by iteratively adding available elements 3.5.3.6. Termination criteria. Genetic algorithm continues the above 3.5.3.3. The selection, crossover and mutation operator. The selection procedure until some predetermined criteria are achieved. Several operator determines from which chromosomes the next genera- criteria were commonly used in previous research, such as the tion will be generated(Michalewicz, 1996: Pezzella, Morganti, number of executed generation(Chen, Pan, Lin, 2008). a particu- Ciaschetti, 2008). It ensures that better members of the population lar objective function, and the homogeneity of the population (with higher fitness value)will have higher probability of being se-(Huang Lim, 2006). This research uses a fixed number of gener have a small probability of being selected. For this problem, the fit- ness value of the chromosome is measured by objective function in the assignment model. And the roulette wheel selection method 3.6. Step 6. Adjusting the assignment results (Hung, 2008) is used to select parents from population. The aim of crossover is to generate new individuals which The purpose of the proposed approach is to help managers to maintain some characteristics of their parents and at the meantime perform the reviewer assignment task efficiently and effectively extend the search space. The crossover scheme used here is similar Atter assignment results are obtained using the proposed ap to the traditional two-point crossover scheme in genetic algorithm proach, they can be checked and adjusted by managers according where the row position replacing the bit position to the specific funding policies. The mutation operator can maintain the diversity of the popu lation and prevent the premature convergence. In this algorithm, 4. System design and development the mutation operator can be described as follows: for any ran- domly selected individual expressed as a binary array, randomly This section describes the design and development of a prote select two column sites and interchange the values of the corre- type system for supporting reviewer assignment based on the ap- proach proposed in Section 3. The high level architecture of the system is shown in Fig. 6. The upper layer of the system provides 3.5.3.4. The repair operator. From the earlier mentioned crossover an interface for users who can access the system via the Web. It and mutation operator, it is obvious that constrains #(1)in the also enables users to adjust the assignment results based on the assignment model are always maintained, while constraints #(2) outcome provided by the system may be violated. So some infeasible solutions may be generated This system supports the whole proposal assignment process during the evolution process, and their proportion in the popula- through the fulfillment of six tasks: identifying valid proposals tion may become large. One way to deal with this is using a penalty and reviewers, classifying them according to their disciplines, par function method to guide the search direction because of its sim- titioning proposals into proposal groups based on the practical plicity and ease of implementation(Gen Cheng, 1997). But this needs of funding agencies, calculating the matching degree be- method cannot guarantee that constrains are satisfied and may tween each proposal group and reviewer, assigning proposal lead to infeasible results. Besides, a good penalty function is diffi- groups to reviewers, and adjusting the assignment results. In the ult to determine. Thus, a repair operator is used to fix infeasible phase of validity identification, valid proposals and reviewers are solutions and make them feasible, like using a greedy approach selected for further consideration. These valid proposals and to transfer the work of overloaded reviewers to others reviewers are classified according to their disciplines. Proposals are grouped based on funding policies and requirements. Then, 3.5.3.5. Local se search algorithm works by succe the matching degree between proposal groups and reviewers are searching a better solution in the neighborhood of the current solu- calculated using a matching model Based on the matching degre tion and replac e current solution(Boudia, Louly, Prins, proposal groups are assigned to reviewers. Also the managers

‘‘1” element is taken place by the code numbers of the reviewers, and ‘‘0” element is deleted to reduce the space size. 3.5.3.2. Initialization. The initial population is generated in the con￾struction phase of GRASP. In the construction phase of GRASP, a feasible solution is iteratively constructed, one element at a time (Aiex, Resende, Pardalos, & Toraldo, 2005; Alvarez-Valdes, Crespo, Tamarit, & Villa, 2008). The element added to the solution is ran￾domly selected from a restricted candidate list (Resende, Martı´ , Gallego, & Duarte, 2008), which is formed by ordering all candidate elements according to the benefit (here it is the matching degree between a proposal group and a reviewer) of selecting this element and use a threshold to ensure that only the elements whose benefit is above the threshold can be selected. Given a n m reviewer and proposal group assignment matrix P, a row index i 2 {1,...,n}, and a column index j 2 {1,...,m}, and let X(P,i,j) be the following com￾patibility function: XðP;i;jÞ ¼ 1; for matrix p; assigning group i to reviewer j is allowed 0; otherwise X(P,i,j) can be used as an index to determine whether assigning a proposal group to a reviewer is available. A candidate list is com￾posed of all available elements. If X(P,i,j) = 1, then this (i,j) assign￾ment pair can be added to the candidate list. Chromosomes (or solutions) are generated by iteratively adding available elements. 3.5.3.3. The selection, crossover and mutation operator. The selection operator determines from which chromosomes the next genera￾tion will be generated (Michalewicz, 1996; Pezzella, Morganti, & Ciaschetti, 2008). It ensures that better members of the population (with higher fitness value) will have higher probability of being se￾lected for mating, while the worse members of the population still have a small probability of being selected. For this problem, the fit￾ness value of the chromosome is measured by objective function in the assignment model. And the roulette wheel selection method (Hung, 2008) is used to select parents from population. The aim of crossover is to generate new individuals which maintain some characteristics of their parents and at the meantime extend the search space. The crossover scheme used here is similar to the traditional two-point crossover scheme in genetic algorithm where the row position replacing the bit position. The mutation operator can maintain the diversity of the popu￾lation and prevent the premature convergence. In this algorithm, the mutation operator can be described as follows: for any ran￾domly selected individual expressed as a binary array, randomly select two column sites and interchange the values of the corre￾sponding two columns. 3.5.3.4. The repair operator. From the earlier mentioned crossover and mutation operator, it is obvious that constrains # (1) in the assignment model are always maintained, while constraints # (2) may be violated. So some infeasible solutions may be generated during the evolution process, and their proportion in the popula￾tion may become large. One way to deal with this is using a penalty function method to guide the search direction because of its sim￾plicity and ease of implementation (Gen & Cheng, 1997). But this method cannot guarantee that constrains are satisfied and may lead to infeasible results. Besides, a good penalty function is diffi- cult to determine. Thus, a repair operator is used to fix infeasible solutions and make them feasible, like using a greedy approach to transfer the work of overloaded reviewers to others. 3.5.3.5. Local search. A local search algorithm works by successively searching a better solution in the neighborhood of the current solu￾tion and replacing the current solution (Boudia, Louly, & Prins, 2007; Feo & Resende, 1995). The neighborhood structure N for a solution s to a specific problem is a subset of the solutions N(s). Gi￾ven two solutions represented by two matrices p and q, the differ￾ence between p and q can be defined as: d(p,q) = {(i,j)jp(i,j) – q(i,j)}, and the distance between p and q is defined to be: d(p,q) = jd(p,q)j. For the majority of GRASP, the most commonly used neighborhood structure is the so-called k-exchange neighbor￾hood structure. The k-exchange neighborhood of a solution p can be defined as NkðpÞ¼fqjdðp; qÞ 6 kg; where 2 6 k 6 minðn; mÞ This neighborhood structure is also employed in the proposed method. However, the common exchange technique of just swap￾ping k elements of the solution is not suitable for this reviewer assignment problem because it leads to infeasible solutions. Thus, a new local search mechanism is proposed for this problem. Each step of the local search is composed of two operations: deleting and inserting operation. Deleting operation is based on eliminating some nonzero elements and changing their values from one to zero in a complete group-reviewer assignment matrix, forming an incomplete assignment matrix. The inserting operation is based on completely constructing the incomplete assignment matrix using the constructing phase described in the initialization phase. 3.5.3.6. Termination criteria. Genetic algorithm continues the above procedure until some predetermined criteria are achieved. Several criteria were commonly used in previous research, such as the number of executed generation (Chen, Pan, & Lin, 2008), a particu￾lar objective function, and the homogeneity of the population (Huang & Lim, 2006). This research uses a fixed number of gener￾ations as the stopping rule. 3.6. Step 6. Adjusting the assignment results The purpose of the proposed approach is to help managers to perform the reviewer assignment task efficiently and effectively. After assignment results are obtained using the proposed ap￾proach, they can be checked and adjusted by managers according to the specific funding policies. 4. System design and development This section describes the design and development of a proto￾type system for supporting reviewer assignment based on the ap￾proach proposed in Section 3. The high level architecture of the system is shown in Fig. 6. The upper layer of the system provides an interface for users who can access the system via the Web. It also enables users to adjust the assignment results based on the outcome provided by the system. This system supports the whole proposal assignment process through the fulfillment of six tasks: identifying valid proposals and reviewers, classifying them according to their disciplines, par￾titioning proposals into proposal groups based on the practical needs of funding agencies, calculating the matching degree be￾tween each proposal group and reviewer, assigning proposal groups to reviewers, and adjusting the assignment results. In the phase of validity identification, valid proposals and reviewers are selected for further consideration. These valid proposals and reviewers are classified according to their disciplines. Proposals are grouped based on funding policies and requirements. Then, the matching degree between proposal groups and reviewers are calculated using a matching model. Based on the matching degree, proposal groups are assigned to reviewers. Also the managers Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956 6953

aL/Expert Systems wi Results Adjustment just assignment User Interface Classification Matching Degre Identification Proposal Grouping into based on the roposal groups their disciplines determined criteria ups and reviewers Databas Knowledge b Model base Proposal Data Know ledge rules Grouping N Human Resource Data Assignment Model Matching Model Fig. 6. High level system architecture. could adjust the results according to the funding agency' s used to partition proposals into balanced groups with the consid- requirements. erations of the specific requirements of NSFC (Fan et al., 2008). The database, model base and knowledge base of this DSS are The model for determining the matching degree between a pro- designed to support the decision making process of the proposal posal group and a reviewer helps to decide whether a reviewer assignment. There are two major categories of data stored in the appropriate for a specific proposal group and to what degree based database: human resource data and proposal data. Human re- on the information provided by the reviewers and the applicants. ource data include the information about the panel chair, panel The assignment model aims to assign reviewers to proposal groups members, reviewers, applicants, and their obligations and respon- according to their matching degree and the requirements of NSFC. sibilities. Applicants data contain information about proposal The assignment model incorporates GRASP and ga to search qual applicants, such as name, gender, professional title, educational ified reviewers for each proposal group for large volume problem. ckground, etc. Each individual applicant should have an affil Knowledge rules extend the capabilities of model-based and ated organization. Individual applicants submit their proposals data-based system and enable dealing with semi-structured or posal data consists of title, abstract, keywords and other elements are usually embodied in the work process d a roposal assignment through their affiliated organizations to the funding agency. Pro- unstructured problems. Knowledge rules for be implied from of a research prop documents of NSFC Knowledge rules can be exacted from these The decision models and knowledge rules used in the approach documents to form knowledge base. Knowledge rules are used to are summarized in Table 4. The model base is designed for parti- continuously monitor the accuracy of the models and to refine ioning proposals into groups, calculating matching degree be- the results as needed They are also designed for identifying valid ween proposal groups and reviewers and assigning proposal proposals and reviewer classifying proposals and reviewers based groups to reviewers. For grouping proposals, a grouping model is on their research discipline areas, partition proposals to form Table 4 Decision models and knowledge rules used in the system. Task ing whether proposals are valid, invalid, incomplete and underdetermined whether reviewers are qualified Proposal grou dge rules for classifying proposals and reviewers according to their discipli Knowledge rules to determine the criteria to measure whether grouping results are appropriate Matching degree calculation nowledge rules for determining whether there is a conflict of interest between an applicant and a reviewer Matching model to calculate the matching degree for each combination of a proposal group and a reviewer Reviewer assignmen ssignment model to assign proposal groups to reviewers based on the matching degree

could adjust the results according to the funding agency’s requirements. The database, model base and knowledge base of this DSS are designed to support the decision making process of the proposal assignment. There are two major categories of data stored in the database: human resource data and proposal data. Human re￾source data include the information about the panel chair, panel members, reviewers, applicants, and their obligations and respon￾sibilities. Applicants’ data contain information about proposal applicants, such as name, gender, professional title, educational background, etc. Each individual applicant should have an affili￾ated organization. Individual applicants submit their proposals through their affiliated organizations to the funding agency. Pro￾posal data consists of title, abstract, keywords and other elements of a research proposal. The decision models and knowledge rules used in the approach are summarized in Table 4. The model base is designed for parti￾tioning proposals into groups, calculating matching degree be￾tween proposal groups and reviewers and assigning proposal groups to reviewers. For grouping proposals, a grouping model is used to partition proposals into balanced groups with the consid￾erations of the specific requirements of NSFC (Fan et al., 2008). The model for determining the matching degree between a pro￾posal group and a reviewer helps to decide whether a reviewer is appropriate for a specific proposal group and to what degree based on the information provided by the reviewers and the applicants. The assignment model aims to assign reviewers to proposal groups according to their matching degree and the requirements of NSFC. The assignment model incorporates GRASP and GA to search qual￾ified reviewers for each proposal group for large volume problem. Knowledge rules extend the capabilities of model-based and data-based system and enable dealing with semi-structured or unstructured problems. Knowledge rules for proposal assignment are usually embodied in the work process and can be implied from documents of NSFC. Knowledge rules can be exacted from these documents to form knowledge base. Knowledge rules are used to continuously monitor the accuracy of the models and to refine the results as needed. They are also designed for identifying valid proposals and reviewer, classifying proposals and reviewers based on their research discipline areas, partition proposals to form User Interface Model Base Grouping Model Assignment Model Knowledge Base Knowledge Rules Database Proposal Data Human Resource Data Validity Identification Identify valid proposals and reviewers Classification Classify proposals and reviewers according to their disciplines Proposal Grouping Partition proposals into groups based on the predetermined criteria Matching Degree Calculation Calculating matching degree between proposal groups and reviewers Reviewer Assignment Assigning reviewers to proposal groups Matching Model Results Adjustment Adjust assignment results Fig. 6. High level system architecture. Table 4 Decision models and knowledge rules used in the system. Task name Support components Validity identification Knowledge rules for determining whether proposals are valid, invalid, incomplete and underdetermined Knowledge rules to determine whether reviewers are qualified Classification Knowledge rules for classifying proposals and reviewers according to their disciplines Proposal grouping Knowledge rules to determine the criteria to measure whether grouping results are appropriate Grouping model to partition proposals into groups Matching degree calculation Knowledge rules for determining whether there is a conflict of interest between an applicant and a reviewer Matching model to calculate the matching degree for each combination of a proposal group and a reviewer Reviewer assignment Assignment model to assign proposal groups to reviewers based on the matching degree 6954 Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956

Y. Xu et al/ Expert Systems with Applications 37(2010)6948-6956 the proposal groups were assigned to reviewers using the proposed hybrid genetic algorithm. The experiment was conducted five times using the hybrid genetic algorithm and the average results 品号88已杀 were recorded. The experiment results are shown in Figs. 7 and 8. From Fig. 7, we can see that the lowest expertise level for each proposal group is 2. 2 on a scale of 1-3, which means that most reviewers were very familiar with the discipline areas of their as- signed proposal groups And no conflicts of interests exist between applicants in proposal groups and the reviewers. Since the number of proposal groups assigned to all revier less than 3. the workload of the reviewers has been controlled as well(see Fig 8) Group number 6. Conclusions Fig. 7. The average expertise for each group This paper presents a novel approach to solve the proposal in funding age posals is large. In the proposed approach, knowledge rules an decision models complement each other to improve the assign- ment process. Based on this approach, a decision support system 2.5 has been designed to enable program directors or department managers in funding agencies to complete the proposal assignment task efficiently and effectively. The major benefits of our method are:(1)it is in compliant with the current workflow and standards of the proposal assignment process of NSFC: (2)it frees program directors or department managers from manual and time-consu ing task of the proposal assignment; ( 3)the web-based DSS en- 100120 ables the users to access it at any place and time: (4)the system can be easily incorporated with the existing ISIS to support the en- tire project selection process in NSFC. Fig 8. The workload of assigned reviewers. he major contribution of this research can be summarized as follows. First, it proposes a novel approach to support the task of assigning proposals to reviewers. This approach addresses charac balanced groups and improve the matching degree between teristics of proposals assigned to each reviewer by grouping pro- reviewers and their assigned proposal groups posals first, which has been long ignored in the literature. Second, a hybrid GRASP and GA is used to search for the desired solutions of assigning proposal groups to reviewers for large vol 5. Evaluation and potential application ume problem. Third, this research provides a comprehensive framework integrating models and knowledge rules AnInternet-basedScienceiNformationSystem(iSIs,http:/facilitateassigningproposalstoreviewersinlargefundingagency isis nsfc.gov. cn) has been developed by us and in use in NSFC for where knowledge rules are used to deal with semi-structured electronic submission, online evaluation of proposals, and problems and the mathematical models are used to handle struc nnouncement of funding results since May 2000. However, the tured problems ISIs system was designed for assigning individual proposals to reviewers without automating the grouping of proposals and the needs. the assignment problem is evidenced in the current process ensuing task for assigning proposal groups to reviewers. This pre- of NSFC, so the proposed approach aims to solve the specific NSFC. However, the roach can be mod- managers, where about 80,000 proposals need to be grouped and ified and adopted for similar use in other funding agencies assigned within short period of time annually. Therefore, the pro- posed system can be integrated into the existing ISIS system to im prove the task of assigning proposals to reviewers. References NSFC has internal guidelines to ensure the quality of the pro- posal assignment. The first is to guarantee that reviewers have en ough expertise in the discipline areas to review proposals assigned quadratic assignment problem. Computers and Operations Research, 27(10). to them. The second is that the number of proposals assigned to each 917-934. Aiex. R M. Resende, M. G. C. Pardal reviewer should be controlled In NSFC, each proposal is generally re- ewed by five external reviewers, and the number of proposals as- relinking for three-index assignment. Informs Journal on Computing 17(2). signed to each reviewer must be smaller than a predetermined Alvarez-Valdes, R, Crespo, E, Tamarit, J M.& villa. F(2008) GRASP and path number which is set by the different scientific departments. search,1893)1153-1170. To validate the proposed approach, an experiment was con- Boudia, M Louly. M.A O.& Prins, C(2007). A reactive GRASP and path relinking ducted based on historical data of proposals and reviewers in one NSFC department. The data was composed of 100 reviewers and Chen, ]. S, Pan, j C H.& Lin, C M(2008) 300 proposals. The proposals were first partitioned into 30 pro- ystems with Applications, 34(1), posal groups, about 10 proposals in each proposal group. Then Cook, w. D, Golany, B. Kress, M, Penn, M, T(2005)op sals to reviewers to facilitate effective ranking. Man Science, 51(4). Ihttp://www.nsfcgov.cn

balanced groups and improve the matching degree between reviewers and their assigned proposal groups. 5. Evaluation and potential application An Internet-based Science Information System (ISIS, http:// isis.nsfc.gov.cn) has been developed by us and in use in NSFC for electronic submission, online evaluation of proposals, and announcement of funding results since May 2000. However, the ISIS system was designed for assigning individual proposals to reviewers without automating the grouping of proposals and the ensuing task for assigning proposal groups to reviewers. This pre￾sents great challenges to the program directors and department managers, where about 80,000 proposals need to be grouped and assigned within short period of time annually. Therefore, the pro￾posed system can be integrated into the existing ISIS system to im￾prove the task of assigning proposals to reviewers. NSFC has internal guidelines1 to ensure the quality of the pro￾posal assignment. The first is to guarantee that reviewers have en￾ough expertise in the discipline areas to review proposals assigned to them. The second is that the number of proposals assigned to each reviewer should be controlled. In NSFC, each proposal is generally re￾viewed by five external reviewers, and the number of proposals as￾signed to each reviewer must be smaller than a predetermined number which is set by the different scientific departments. To validate the proposed approach, an experiment was con￾ducted based on historical data of proposals and reviewers in one NSFC department. The data was composed of 100 reviewers and 300 proposals. The proposals were first partitioned into 30 pro￾posal groups, about 10 proposals in each proposal group. Then the proposal groups were assigned to reviewers using the proposed hybrid genetic algorithm. The experiment was conducted five times using the hybrid genetic algorithm and the average results were recorded. The experiment results are shown in Figs. 7 and 8. From Fig. 7, we can see that the lowest expertise level for each proposal group is 2.2 on a scale of 1–3, which means that most reviewers were very familiar with the discipline areas of their as￾signed proposal groups. And no conflicts of interests exist between applicants in proposal groups and the reviewers. Since the number of proposal groups assigned to all reviewers was less than 3, the workload of the reviewers has been controlled as well (see Fig. 8). 6. Conclusions This paper presents a novel approach to solve the proposal assignment problem in funding agencies where the number of pro￾posals is large. In the proposed approach, knowledge rules and decision models complement each other to improve the assign￾ment process. Based on this approach, a decision support system has been designed to enable program directors or department managers in funding agencies to complete the proposal assignment task efficiently and effectively. The major benefits of our method are: (1) it is in compliant with the current workflow and standards of the proposal assignment process of NSFC; (2) it frees program directors or department managers from manual and time-consum￾ing task of the proposal assignment; (3) the web-based DSS en￾ables the users to access it at any place and time; (4) the system can be easily incorporated with the existing ISIS to support the en￾tire project selection process in NSFC. The major contribution of this research can be summarized as follows. First, it proposes a novel approach to support the task of assigning proposals to reviewers. This approach addresses charac￾teristics of proposals assigned to each reviewer by grouping pro￾posals first, which has been long ignored in the literature. Second, a hybrid GRASP and GA is used to search for the desired solutions of assigning proposal groups to reviewers for large vol￾ume problem. Third, this research provides a comprehensive framework integrating models and knowledge rules together to facilitate assigning proposals to reviewers in large funding agency, where knowledge rules are used to deal with semi-structured problems and the mathematical models are used to handle struc￾tured problems. This research is motivated by real-world funding agencies’ needs. The assignment problem is evidenced in the current process of NSFC, so the proposed approach aims to solve the specific assignment problem in NSFC. However, the approach can be mod￾ified and adopted for similar use in other funding agencies. References Ahuja, R. K., Orlin, J. B., & Tiwari, A. (2000). A greedy genetic algorithm for the quadratic assignment problem. Computers and Operations Research, 27(10), 917–934. Aiex, R. M., Resende, M. G. C., Pardalos, P. M., & Toraldo, G. (2005). GRASP with path relinking for three-index assignment. Informs Journal on Computing, 17(2), 224–247. Alvarez-Valdes, R., Crespo, E., Tamarit, J. M., & Villa, F. (2008). GRASP and path relinking for project scheduling under partially renewable resources. European Journal of Operational Research, 189(3), 1153–1170. Boudia, M., Louly, M. A. O., & Prins, C. (2007). A reactive GRASP and path relinking for a combined production–distribution problem. Computers and Operations Research, 34(11), 3402–3419. Chen, J. S., Pan, J. C. H., & Lin, C. M. (2008). A hybrid genetic algorithm for the re￾entrant flow-shop scheduling problem. Expert Systems with Applications, 34(1), 570–577. Cook, W. D., Golany, B., Kress, M., Penn, M., & Raviv, T. (2005). Optimal allocation of proposals to reviewers to facilitate effective ranking. Management Science, 51(4), 655–661. 0 0.5 1 1.5 2 2.5 3 3.5 0 10 20 30 40 Group number Averge expertise for each group Fig. 7. The average expertise for each group. 0 0.5 1 1.5 2 2.5 3 3.5 0 20 40 60 80 100 120 Reviewers Workload (Number of groups) Fig. 8. The workload of assigned reviewers. 1 http://www.nsfc.gov.cn. Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956 6955

Y. Xu et al/ Expert Systems with Applications 37(2010)6948-6956 David, M.& Andrew, M.(2007). Expertise modeling for matching papers with conference on parallel problem solving from nature(PPSN VIll), Birmingham, reviewers. In Proceedings of the 13th ACM SIGKDD international conference on Michalewicz, Z(1996). Genetic algorithms data structures=evolution programs. Deep, K,& Das, K N.(2008). Quadratic approximation based hybrid genetic Berlin: Springer-Verlag gorithm for function optimization. Applied Mathematics and Computation, Ebro, A, Luque, G. Luna, F.& Alba, E(2008). DNA fragment assembly using a 203(1).86-98. Dumais, S. T.& Nielsen, J. (1992) Automating the assignment of submitted scripts to reviewers. Research and Development in information Retrieval, Papagelis, M, Plexousakis, D,& N.(2005) Fan,Z P. Chen, Y, Ma, - Zhu, Y (2008). Decision support for proposal grouping: A h using knowledge rule and genetic algorithm. Expert Syster Pezzella, F, Morganti, G,& Ciaschetti, G (2008). A genetic algor or the flexible b-shop scheduling problem. Computers and Operations Research, 35(10), search Harper, P R, de senna, v. vieira, I. T, Shahani, A K(2005). A genetic algorith Rodriguez, M. A& Bollen, J(2008). An algorithm to determine peer-reviewers. In Me, 1255-126sject assignment problem. Computers and Operations Research, at the national science foundation. In Proceedings of the 12th ACM valuate experts for R&D project selection. IEEE Transactions on Engineering intemational conference on knowledge discovery and data mining. Philadelphia, Sun, Y. H, Ma,J- Fan, Z. P& Wang. J(2008) ge and mode Huang. G. F& Lim, A.(2006 A hybrid genetic algorithm for the three-index approach for reviewer assignment. Expert Systems with Applications, 34(2), ssignment problem. European Jourmal of operational Research, 172(1). 249-257. Tang. A M, Quek, C,& Ng, G. S(2005). GA-TSKEn: Parameters tuning of fuzzy Hung, J C (2008) A genetic algorithm approach to the spectral estimation of time neural network using genetic algorithms. Expert Systems with Applications, ries with noise and missed observations. Information Sciences, 178(24). Tian, Q. Ma,J- Liang ]. z, Kwok, R C. w,& Liu, O(2005). An organizational Janak, S. L, Taylor, M. S, Floudas, C. A, Burka, M, Mountziaris, T.J.(2006). ovel and effective integer optimization approach for the NSF stms. 39(D)0403-415m0 for erective RaD project selecton. Decsion Support Tian, QJ, Ma. ]& Liu, O(2002) A hybrid knowledge and model system for R&D Wang F, Chen, B, Miao, Z w.(2008 Merelo-Guervos, J.& Castillo-Valdivieso, P(2004) Confere n 21st international conference on industrial, engineering and other applications of using a combined greedy /evolutionary algorithm. In Eighth international applied intelligent systems, Wroclaw, Poland

David, M., & Andrew, M. (2007). Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD international conference on knowledge discovery and data mining. CA, USA: ACM, San Jose. Deep, K., & Das, K. N. (2008). Quadratic approximation based hybrid genetic algorithm for function optimization. Applied Mathematics and Computation, 203(1), 86–98. Dumais, S. T., & Nielsen, J. (1992). Automating the assignment of submitted manuscripts to reviewers. Research and Development in information Retrieval, 233–244. Fan, Z. P., Chen, Y., Ma, J., & Zhu, Y. (2008). Decision support for proposal grouping: A hybrid approach using knowledge rule and genetic algorithm. Expert Systems with Applications. Feo, T. A., & Resende, M. G. C. (1995). Greedy randomized adaptive search procedures. Journal of Global Optimization, 6(2), 109–133. Gen, M., & Cheng, R. (1997). Genetic algorithms and engineering design. New York: Wiley. Harper, P. R., de Senna, V., Vieira, I. T., & Shahani, A. K. (2005). A genetic algorithm for the project assignment problem. Computers and Operations Research, 32(5), 1255–1265. Hettich, S., & Pazzani, M. J. (2006). Mining for proposal reviewers: Lessons learned at the national science foundation. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining. Philadelphia, PA, USA: ACM. Huang, G. F., & Lim, A. (2006). A hybrid genetic algorithm for the three-index assignment problem. European Journal of Operational Research, 172(1), 249–257. Hung, J. C. (2008). A genetic algorithm approach to the spectral estimation of time series with noise and missed observations. Information Sciences, 178(24), 4632–4643. Janak, S. L., Taylor, M. S., Floudas, C. A., Burka, M., & Mountziaris, T. J. (2006). Novel and effective integer optimization approach for the NSF panel￾assignment problem: A multiresource and preference-constrained generalized assignment problem. Industrial and Engineering Chemistry Research, 45(1), 258–265. Merelo-Guervos, J. J. & Castillo-Valdivieso, P. (2004). Conference paper assignment using a combined greedy/evolutionary algorithm. In Eighth international conference on parallel problem solving from nature (PPSN VIII), Birmingham, England. Michalewicz, Z. (1996). Genetic algorithms + data structures = evolution programs. Berlin: Springer-Verlag. Nebro, A. J., Luque, G., Luna, F., & Alba, E. (2008). DNA fragment assembly using a grid-based genetic algorithm. Computers and Operations Research, 35(9), 2776–2790. Papagelis, M., Plexousakis, D., & Nikolaou, P. N. (2005). CONFIOUS: Managing the electronic submission and reviewing process of scientific conferences. In Sixth international conference on web information systems engineering, NY, USA. Pezzella, F., Morganti, G., & Ciaschetti, G. (2008). A genetic algorithm for the flexible job-shop scheduling problem. Computers and Operations Research, 35(10), 3202–3212. Resende, M. G. C., Martı´, R., Gallego, M., & Duarte, A. (2008). GRASP and path relinking for the max–min diversity problem. Computers and Operations Research. Rodriguez, M. A. & Bollen, J. (2008). An algorithm to determine peer-reviewers. In Proceeding of the 17th ACM conference on information and knowledge management, Napa Valley, CA, USA. Sun, Y. H., Ma, J., Fan, Z. P., & Wang, J. (2008). A group decision support approach to evaluate experts for R&D project selection. IEEE Transactions on Engineering Management, 55(1), 158–170. Sun, Y. H., Ma, J., Fan, Z. P., & Wang, J. (2008). A hybrid knowledge and model approach for reviewer assignment. Expert Systems with Applications, 34(2), 817–824. Tang, A. M., Quek, C., & Ng, G. S. (2005). GA-TSKfnn: Parameters tuning of fuzzy neural network using genetic algorithms. Expert Systems with Applications, 29(4), 769–781. Tian, Q. J., Ma, J., Liang, J. Z., Kwok, R. C. W., & Liu, O. (2005). An organizational decision support system for effective R&D project selection. Decision Support Systems, 39(3), 403–413. Tian, Q. J., Ma, J., & Liu, O. (2002). A hybrid knowledge and model system for R&D project selection. Expert Systems with Applications, 23(3), 265–271. Wang, F., Chen, B., & Miao, Z. W. (2008). A survey on reviewer assignment problem. In 21st international conference on industrial, engineering and other applications of applied intelligent systems, Wroclaw, Poland. 6956 Y. Xu et al. / Expert Systems with Applications 37 (2010) 6948–6956