《电子商务 E-business》阅读文献：A Survey on Reviewer Assignment Problem

A Survey on Reviewer Assignment Problem Fan wang. Ben Chen. and Zhaowei Miao I School of Business, Sun Yat-Sen University, Guangzhou, PRChina chenben819@gmail co M School, Xiamen University, Xiamen, P R China Abstract. Research into Reviewer Assignment Problem(RAP)is still in its early stage but there is great world-wide interest, as the foregoing process of peer-review which is the brickwork of science authentication. The RAP ap- proach can be divided into three phases: identifying assignment procedure omputing the matching degree between manuscripts and reviewers, and opti- mizing the assignment so as to achieve the given objectives. Methodologies for addressing the above three phases have been developed from a variety of re- search disciplines, including information retrieval, artificial intelligent, opera- tions research, etc. This survey is not only to cover variations of rAP that have ppeared in the literature, but also to identify the practical challenge and current progress for developing intelligent RAP systems Keywords: reviewer assign information retrieval; conference system. 1 Introduction Assigning submitted manuscripts to reviewers known as Reviewer Assignment Prob lem(RAP), is an important but tough task for journal editors, conference program chairs, and research councils. The essence of RAP can be divided into three phases: (1)identifying the assignment procedure, (2)computing the match between manu- scripts and reviewers, and (3)optimizing the assignment so as to maximize the match within the feasible restriction Reviewer assignment, traditionally handled by a single person (or at most a few people), is approached to satisfy four conditions: matching manuscripts and review- ers, fulfilling manuscript slots, balancing reviewers' workload and avoiding conflicts of interest. As reviewer assignment must be completed under severe timing con- straints, along with a very large number of submissions arriving near an announced deadline, it makes RAP a stressed dual problem of time and labor intensive. In re- sponse to the need of automatic mechanism, a number of studies have addressed the assignment solutions since Dumais and Nielsens[14] breakthrough orresponding Author. This research is supported by National Natural Science Foundation of China under Project No. 60704048. The author Dr. Zhaowei Miao thanks the support by Hu manities and Social Science Project(No. 07JC630047)of National Ministry of Education, China N.T. Nguyen et al.(Eds IEA/AIE 2008, LNAl 5027, PP. 718-727, 2008. o Springer-Verlag Berlin Heidelberg 2008

N.T. Nguyen et al. (Eds.): IEA/AIE 2008, LNAI 5027, pp. 718–727, 2008. © Springer-Verlag Berlin Heidelberg 2008 A Survey on Reviewer Assignment Problem Fan Wang1 , Ben Chen1,*, and Zhaowei Miao2 1 School of Business, Sun Yat-Sen University, Guangzhou, P.R. China chenben819@gmail.com 2 Management School, Xiamen University, Xiamen, P.R. China Abstract. Research into Reviewer Assignment Problem (RAP) is still in its early stage but there is great world-wide interest, as the foregoing process of peer-review which is the brickwork of science authentication. The RAP ap￾proach can be divided into three phases: identifying assignment procedure, computing the matching degree between manuscripts and reviewers, and opti￾mizing the assignment so as to achieve the given objectives. Methodologies for addressing the above three phases have been developed from a variety of re￾search disciplines, including information retrieval, artificial intelligent, opera￾tions research, etc. This survey is not only to cover variations of RAP that have appeared in the literature, but also to identify the practical challenge and current progress for developing intelligent RAP systems. Keywords: reviewer assignment; information retrieval; conference system. 1 Introduction Assigning submitted manuscripts to reviewers known as Reviewer Assignment Prob￾lem (RAP), is an important but tough task for journal editors, conference program chairs, and research councils. The essence of RAP can be divided into three phases: (1) identifying the assignment procedure, (2) computing the match between manu￾scripts and reviewers, and (3) optimizing the assignment so as to maximize the match within the feasible restriction. Reviewer assignment, traditionally handled by a single person (or at most a few people), is approached to satisfy four conditions: matching manuscripts and review￾ers, fulfilling manuscript slots, balancing reviewers’ workload and avoiding conflicts of interest. As reviewer assignment must be completed under severe timing con￾straints, along with a very large number of submissions arriving near an announced deadline, it makes RAP a stressed dual problem of time and labor intensive. In re￾sponse to the need of automatic mechanism, a number of studies have addressed the assignment solutions since Dumais and Nielsen’s [14] breakthrough. * Corresponding Author. This research is supported by National Natural Science Foundation of China under Project No. 60704048. The author Dr. Zhaowei Miao thanks the support by Hu￾manities and Social Science Project (No. 07JC630047) of National Ministry of Education, China

A Survey on Reviewer Assignment Problem 719 In general, RAP described in this paper, should be relevant whenever one has two large sets of objects that need to be matched such that each object in one set gets as- mall number of objects from the other, and have many practical appl tions, such as resource allocation(matching funding agencies to research projects[10] scheduling on parallel machines [13], assigning managers to construction projects [27], classroom assignment [8]), staff scheduling (assigning graduating students to interviewer, assigning press releases to newspaper reporters, matching staff to pro- jects in consulting companies [7], crew scheduling in airline companies [19], posting military servicemen [3, 25)), and construction decision making(corrective action rec ommendation for material management (38)) This wide range of real-world applications has constituted a major motivation for scholars in developing solutions for RAP. Reviewer assignment has been discussed as a process of peer-review in the voice of improving review mechanism. One of the earliest papers that addressed an assignment solution is by Dumais and Nielsen [14] who presented the matching degree computation by Latent Semantic Indexing(LSD), which was refined by Yarowsky and Florian [41] on using reviewers' online publica tion instead of their autobiography. Different from content-based Information Re trieval (IR), Watanabe et al. [39 raised the matching degree computation by constructing collaborative network, which leads to the hybrid approach of the two technologies. On the other hand, Operations Research(OR) has been used to build the assignment models and design algorithms of optimization RAP, which has been widely approached along different disciplines, can be applied into various application fields and represented by multifarious items, thus making it more difficult to search for previous work on the topic. This paper is intended to pro- vide a comprehensive survey according to the researches which have appeared in the literature sorting by the three phases of RaP. The value of such a study is in provid- ing an opportunity to reflect on what has been achieved, to identify gaps that need to be addressed. and to set direction for future research The reminder of this paper is organized as follows. Section 2 provides a review on the assignment procedure. Section 3 presents IR technology for computing matching degree between manuscripts and reviewers. Section 4 is a summary on the method- ologies for optimization modeling. Finally, conclusions are presented and suggestions are made for further research 2 Reviewer Assignment Procedure Researches defended on reviewer assignment procedure involve the specification of identification, and its influence in the peer-review mode( utions of qualified reviewer four issues: assignment criteria, assignment processes, so The assignment criteria play an important role in the methodology and procedure of reviewer assignment, and indirectly affect the peer-review quality. Fixed assign ment criteria are widely used in the science community, which require authors to select keywords from one list of disciplines or prorate the keywords. Fixed lists make up of two or three columns sometimes(e.g." Methodology","Application"and"Oth ers"[33D, which makes the chosen keywords more comprehensively. As it is argued that the update of fixed lists cannot catch up with the development of disciplines, the

A Survey on Reviewer Assignment Problem 719 In general, RAP described in this paper, should be relevant whenever one has two large sets of objects that need to be matched such that each object in one set gets as￾signed a small number of objects from the other, and have many practical applica￾tions, such as resource allocation (matching funding agencies to research projects[10], scheduling on parallel machines [13], assigning managers to construction projects [27], classroom assignment [8]), staff scheduling (assigning graduating students to interviewer, assigning press releases to newspaper reporters, matching staff to pro￾jects in consulting companies [7], crew scheduling in airline companies [19], posting military servicemen [3,25]), and construction decision making (corrective action rec￾ommendation for material management [38]). This wide range of real-world applications has constituted a major motivation for scholars in developing solutions for RAP. Reviewer assignment has been discussed as a process of peer-review in the voice of improving review mechanism. One of the earliest papers that addressed an assignment solution is by Dumais and Nielsen [14], who presented the matching degree computation by Latent Semantic Indexing (LSI), which was refined by Yarowsky and Florian [41] on using reviewers’ online publica￾tion instead of their autobiography. Different from content-based Information Re￾trieval (IR), Watanabe et al. [39] raised the matching degree computation by constructing collaborative network, which leads to the hybrid approach of the two technologies. On the other hand, Operations Research (OR) has been used to build the assignment models and design algorithms of optimization. RAP, which has been widely approached along different disciplines, can be applied into various application fields and represented by multifarious items, thus making it more difficult to search for previous work on the topic. This paper is intended to pro￾vide a comprehensive survey according to the researches which have appeared in the literature sorting by the three phases of RAP. The value of such a study is in provid￾ing an opportunity to reflect on what has been achieved, to identify gaps that need to be addressed, and to set direction for future research. The reminder of this paper is organized as follows. Section 2 provides a review on the assignment procedure. Section 3 presents IR technology for computing matching degree between manuscripts and reviewers. Section 4 is a summary on the method￾ologies for optimization modeling. Finally, conclusions are presented and suggestions are made for further research. 2 Reviewer Assignment Procedure Researches defended on reviewer assignment procedure involve the specification of four issues: assignment criteria, assignment processes, solutions of qualified reviewer identification, and its influence in the peer-review model. The assignment criteria play an important role in the methodology and procedure of reviewer assignment, and indirectly affect the peer-review quality. Fixed assign￾ment criteria are widely used in the science community, which require authors to select keywords from one list of disciplines or prorate the keywords. Fixed lists make up of two or three columns sometimes (e.g. “Methodology”, “Application” and “Oth￾ers” [33]), which makes the chosen keywords more comprehensively. As it is argued that the update of fixed lists cannot catch up with the development of disciplines, the

F Wang B. Chen. and Z miao unfixed criteria are also used in some conference procedure, especially when the dis- cipline information is not integrated or hard to represent by several keywords. In that case, IR technology is applied to compute the similarity between manuscripts and reviewers' biographies [14]. In the mean while, data mining is used to extract the keyword-list by doing unsupervised clustering or supervised learning by using pervi ous accepted papers as training set [5] Reviewer assignment is the foregoing process of peer-review in the manuscr selection procedure, which directly influences the review result and evaluation aggre gating. Conference chair usually works on the reviewer assignment with the regis tered information of manuscripts and reviewers, and then solution is addressed within the certain community. Meanwhile, in the project selection of national funding com- mittee(e.g. NSF), the assignment is firstly run for an optimal solution, and new re- viewers are invited in case no satisfied solution exists. As the potential reviewer pool is large, committee chair never worries about reviewers workload intensive [21, 36 Moreover, some large science communities(e.g. AAAD)ask reviewers to bid on manuscripts by scanning abstracts, which is taken into consideration for the assign- ment. The bid behavior had been studied as human -factor noise which influences the preference very much rather than disciplines of manuscripts [3 Subjects of domain and conflicts of interest are the two main factors in identifying qualified reviewers. Collecting information of these two factors, known as building of knowledge-database, attracts attention of many conference chairs. Domain informa tion can be submitted by authors and reviewers during registration; but information of conflict, including collaborative relation, student-advisor-relationship, colleague rela- tion, is hard to collect. Geller [15] raised this problem to challenge the Al committee to call for an intelligent solution. Furthermore, Geller and Scherl [16] described how to search Internet to generate a potential-reviewers-list There exists a rich body of literature on peer-review that point out the inadequacies of the current systems. Weber[40] presented his manifestos for changing the journal review processes, since the assignment between manuscripts and reviewers works irrationally and inefficiently. Casati et al.[9] asked for more awareness on the open efficient review model and the reasonable assigning manuscripts to reviewers using information technology along with internet. Some scholars argued that the automatic reviewer assignment approaches bereaved their rights on classifying their own prob- lems which were treated as the scientists'most precious possession [34]. And it is said that the taxonomy of disciplines is not changed momentarily, which makes some interdisciplinary researches and frontier of science will never be recognized by the 3 Assignment Based on Information Retrieval IR used on reviewer assignment focuses on the second phase of RAP, which is com- puting the matching degree between manuscripts and reviewers. This phase had been approached mainly in four ways: content-based IR, collaborative filtering, hybrid approach of the former two and data mining One of the earliest RAP solutions found in literature is by IR, since inefficiency of ee scoring manually was firstly raised. Using the content-based IR

720 F. Wang, B. Chen, and Z. Miao unfixed criteria are also used in some conference procedure, especially when the dis￾cipline information is not integrated or hard to represent by several keywords. In that case, IR technology is applied to compute the similarity between manuscripts and reviewers’ biographies [14]. In the mean while, data mining is used to extract the keyword-list by doing unsupervised clustering or supervised learning by using pervi￾ous accepted papers as training set [5]. Reviewer assignment is the foregoing process of peer-review in the manuscripts selection procedure, which directly influences the review result and evaluation aggre￾gating. Conference chair usually works on the reviewer assignment with the regis￾tered information of manuscripts and reviewers, and then solution is addressed within the certain community. Meanwhile, in the project selection of national funding com￾mittee (e.g. NSF), the assignment is firstly run for an optimal solution, and new re￾viewers are invited in case no satisfied solution exists. As the potential reviewer pool is large, committee chair never worries about reviewers’ workload intensive [21,36]. Moreover, some large science communities (e.g. AAAI) ask reviewers to bid on manuscripts by scanning abstracts, which is taken into consideration for the assign￾ment. The bid behavior had been studied as human-factor noise which influences the preference very much rather than disciplines of manuscripts [31]. Subjects of domain and conflicts of interest are the two main factors in identifying qualified reviewers. Collecting information of these two factors, known as building of knowledge-database, attracts attention of many conference chairs. Domain informa￾tion can be submitted by authors and reviewers during registration; but information of conflict, including collaborative relation, student-advisor-relationship, colleague rela￾tion, is hard to collect. Geller [15] raised this problem to challenge the AI committee to call for an intelligent solution. Furthermore, Geller and Scherl [16] described how to search Internet to generate a potential-reviewers-list. There exists a rich body of literature on peer-review that point out the inadequacies of the current systems. Weber [40] presented his manifestos for changing the journal review processes, since the assignment between manuscripts and reviewers works irrationally and inefficiently. Casati et al. [9] asked for more awareness on the open efficient review model and the reasonable assigning manuscripts to reviewers using information technology along with internet. Some scholars argued that the automatic reviewer assignment approaches bereaved their rights on classifying their own prob￾lems which were treated as the scientists’ most precious possession [34]. And it is said that the taxonomy of disciplines is not changed momentarily, which makes some interdisciplinary researches and frontier of science will never be recognized by the corresponding committee. 3 Assignment Based on Information Retrieval IR used on reviewer assignment focuses on the second phase of RAP, which is com￾puting the matching degree between manuscripts and reviewers. This phase had been approached mainly in four ways: content-based IR, collaborative filtering, hybrid approach of the former two and data mining. One of the earliest RAP solutions found in literature is by IR, since inefficiency of matching degree scoring manually was firstly raised. Using the content-based IR

A Survey on Reviewer Assignment Problem 721 method known as LSI, Dumais and Nielsen [14] represented each manuscripts and reviewers'autobiography by a matrix containing nearly 100 item vectors of factors weight, the matching degree was computed as the dot product of the two matrixes Then, assignment was done by picking several reviewers from those with high match ing degree. A similar task was performed by Yarowsky and Florian [41], but review ers' biographies were replaced by their publications which were submitted by them- selves or downloaded from internet. Beginning with Dumais and Nielsens [14] paper, there are nine papers addressed the solution by using IR techniques. Table 1 reveals that content-based methodologies are acceptable, as text is the most important factor for manuscripts and reviewers' biographies, but not the only one for assignment Biswas and Hasan [5] compared the applicability of different content-based filterin and indicated that hybrid approaches might be a more comprehensive way Table 1. Review on the use of Ir solutions for rAp Methodology Yarowsky and Florian [41] Content-Based VSM/Naive Bayes Classifier Basu et al. [2] Popescul et al. [30] Hybrid Watanabe et al.[39 Collaborative Filtering Scale-free Network Hettich and Pazzani [21] Data Mining Data mining Rodriguez and Bollen [32] Collaborative Filtering Relative-rank Particle-Sw Algorithm Biswas and Hasan 5] Content-Based VSM( Comparison Study) Hvbrid Semantic web Studies by ir develop with the progresses of the technology itself. Content-based IR looks only at the contents of an artifact(e. g, the words on a paper), whereas col laborative filtering, which also consider the opinions of other like-minded people with respect to these artifacts, has been used to recommend NetNews articles [26], movies [1, 22], music[ll], and even jokes [18]. Scale-free network, which can continuously expand with the addition of new vertices, is a useful mechanism of collaborative fil tering. Watanabe et al. [39]had constructed a scale-free network whose vertices were keywords of reviewers'expertise and manuscripts' topic, and similarity between two keywords was the probability of connecting between the corresponding vertices. The matching degree is the weighted average of similarities between each pair of key words of manuscripts and reviewers. Instead of keywords, Rodriguez and Bollen [32] had approached a co-authorship network with vertices representing experts, edges representing a tie between two experts, and weights representing the strength of tie. with the application of collaborative filtering in other operations, a hybrid ap- roach was achieved before collaborative filtering by using co-authors and authors of reference to approach the collaborative method in the paper of Basu et al. [2]. Their framework provides a more flexible alternative to simple keyword-based search algo rithms and a less intrusive alternative to collaborative methods. Popescul et al. [30]

A Survey on Reviewer Assignment Problem 721 method known as LSI, Dumais and Nielsen [14] represented each manuscripts and reviewers’ autobiography by a matrix containing nearly 100 item vectors of factors weight, the matching degree was computed as the dot product of the two matrixes. Then, assignment was done by picking several reviewers from those with high match￾ing degree. A similar task was performed by Yarowsky and Florian [41], but review￾ers’ biographies were replaced by their publications which were submitted by them￾selves or downloaded from internet. Beginning with Dumais and Nielsen’s [14] paper, there are nine papers addressed the solution by using IR techniques. Table 1 reveals that content-based methodologies are acceptable, as text is the most important factor for manuscripts and reviewers’ biographies, but not the only one for assignment. Biswas and Hasan [5] compared the applicability of different content-based filtering and indicated that hybrid approaches might be a more comprehensive way. Table 1. Review on the use of IR solutions for RAP Study Methodology Techniques Dumais and Nielsen [14] Content-Based Latent Semantic Indexing Yarowsky and Florian [41] Content-Based VSM/Naive Bayes Classifier Basu et al. [2] Hybrid N/A Popescul et al. [30] Hybrid N/A Watanabe et al. [39] Collaborative Filtering Scale-free Network Hettich and Pazzani [21] Data Mining Data Mining Rodriguez and Bollen [32] Collaborative Filtering Relative-rank Particle-swarm Algorithm Biswas and Hasan [5] Content-Based VSM (Comparison Study) Cameron et al. [6] Hybrid Semantic Web Studies by IR develop with the progresses of the technology itself. Content-based IR looks only at the contents of an artifact (e.g., the words on a paper), whereas col￾laborative filtering, which also consider the opinions of other like-minded people with respect to these artifacts, has been used to recommend NetNews articles [26], movies [1,22], music[11], and even jokes [18]. Scale-free network, which can continuously expand with the addition of new vertices, is a useful mechanism of collaborative fil￾tering. Watanabe et al. [39] had constructed a scale-free network whose vertices were keywords of reviewers’ expertise and manuscripts’ topic, and similarity between two keywords was the probability of connecting between the corresponding vertices. The matching degree is the weighted average of similarities between each pair of key￾words of manuscripts and reviewers. Instead of keywords, Rodriguez and Bollen [32] had approached a co-authorship network with vertices representing experts, edges representing a tie between two experts, and weights representing the strength of tie. With the application of collaborative filtering in other operations, a hybrid ap￾proach was achieved before collaborative filtering by using co-authors and authors of reference to approach the collaborative method in the paper of Basu et al. [2]. Their framework provides a more flexible alternative to simple keyword-based search algo￾rithms and a less intrusive alternative to collaborative methods. Popescul et al. [30]

722 F Wang B Chen and Z miao proposed a unified probabilistic framework for combining content-based and collabo- rative IR by extending Hofmann's [23] aspect model to incorporate the information source among scholar, manuscript and manuscript content. Semantic Web technology was also brought to collect data, represent researches'expertise and co-authorship and find relevant reviewers in a peer-review setting [6]. As the increase of internet resource, data mining can be used to identify relative reviewers within an existing expert pool by mining online information through search engines, potential external reviewers can also be found in the online academic community In respond to the difficulty for a scientific community to agree up k eywords and maintain such a keyword database over time, Hettich and Pazzani [21] described the data mining method deployed at the U.S. NSF for assisting program directors in identifying reviewers for proposals. To the best of our knowledge, thei paper is the only one work found in literature which combined IR and optimization to achieve rap 4 Reviewer Assignment Optimization Optimization on reviewer assignment which mainly focus on the theory, modeling algorithm of assigning manuscripts to reviewers, can be viewed as an enhanced version of the Generalized Assignment Problem(GAP). However, most of these stud- ies are not based on the matching degree evaluated by IR, but simply in sense of simi- larity by the selected keywords. In making it easier to read, the notations used in the cited sources have, where necessary, been changed to try to keep consistent as those of traditional gap The RAP optimization we study is the following. We are given a set P=(l,,P) of manuscripts and a set R=(lr) of reviewers; and a parameter cy denoting the matching degree"of manuscript i for reviewer j, where iE ter a, is the certain number of reviewers that manuscript i should be assigned to: Parameter b, is the certain number of manuscripts that reviewer j should be as- signed to no more than; a given threshold T can be set as boundary of c to identify reviewers'qualification o The simplest version of RAP just distinguishes whether each reviewer and each anuscript can be matched or not, and represents the matching degree as a px matrix C=Ci(i=lp,j=lr), where cy is a binary parameter whose value equals l if there is overlap between reviewers expertise and manuscript's topic and no conflicts of interest, otherwise 0. As the binary ci is too general to present the grade of matching, discrete matching degree becomes a more popular method. A typical but exhausted way is requiring reviewers to rate their preference according the abstracts, with I for the lowest preference to 10 for the highest, sometimes p is used for the highest, where p is the total number of manuscripts. Once there is con flict of interest, cy is set to 0 [24]. Instead of rating for all the submissions, reviewers are usually required to score their expertise level on different disciplines(keywords)

722 F. Wang, B. Chen, and Z. Miao proposed a unified probabilistic framework for combining content-based and collabo￾rative IR by extending Hofmann’s [23] aspect model to incorporate the information source among scholar, manuscript and manuscript content. Semantic Web technology was also brought to collect data, represent researches’ expertise and co-authorship, and find relevant reviewers in a peer-review setting [6]. As the increase of internet resource, data mining can be used to identify relative reviewers within an existing expert pool by mining online information through search engines, potential external reviewers can also be found in the online academic community. In respond to the difficulty for a scientific community to agree upon taxonomy of keywords and maintain such a keyword database over time, Hettich and Pazzani [21] described the data mining method deployed at the U.S. NSF for assisting program directors in identifying reviewers for proposals. To the best of our knowledge, their paper is the only one work found in literature which combined IR and optimization to achieve RAP. 4 Reviewer Assignment Optimization Optimization on reviewer assignment which mainly focus on the theory, modeling and algorithm of assigning manuscripts to reviewers, can be viewed as an enhanced version of the Generalized Assignment Problem (GAP). However, most of these stud￾ies are not based on the matching degree evaluated by IR, but simply in sense of simi￾larity by the selected keywords. In making it easier to read, the notations used in the cited sources have, where necessary, been changed to try to keep consistent as those of traditional GAP. The RAP optimization we study is the following. We are given a set P = {1,..., p} of manuscripts and a set R = {1,...,r} of reviewers; and a parameter ij c denoting the “matching degree” of manuscript i for reviewer j , where i ∈ P and j ∈ R . Parame￾ter i a is the certain number of reviewers that manuscript i should be assigned to; Parameter bj is the certain number of manuscripts that reviewer j should be as￾signed to no more than; a given threshold T can be set as boundary of ij c to identify reviewers’ qualification. The simplest version of RAP just distinguishes whether each reviewer and each manuscript can be matched or not, and represents the matching degree as a p × r matrix ij C = c ( i = 1,..., p , j = 1,...,r ), where ij c is a binary parameter whose value equals 1 if there is overlap between reviewer’s expertise and manuscript’s topic and no conflicts of interest, otherwise 0. As the binary ij c is too general to present the grade of matching, discrete matching degree becomes a more popular method. A typical but exhausted way is requiring reviewers to rate their preference according to the abstracts, with 1 for the lowest preference to 10 for the highest, sometimes p is used for the highest, where p is the total number of manuscripts. Once there is con￾flict of interest, ij c is set to 0 [24]. Instead of rating for all the submissions, reviewers are usually required to score their expertise level on different disciplines (keywords)

A Survey on Reviewer Assignment Problem 723 from the corresponding category, and matching degrees are acquired by a computa- tion mechanism set by organizer [20]. The above computation methods are raised as increasing matching degree, which is appropriate to the optimizations of maximizing utility. In fact, the smaller value for higher preference, with the evaluation on the converse way, so as to be adapted for minimum optimization. With the"matching degree"matrix C, a binary variable xi, whose value is I if manuscript i is assigned to reviewer j and 0 otherwise, is brought to present the integer programming(IP) formulation of the RAP(here we take the increasing match ing degree as example) max x ∑号=a1 x≤b x 0 or 1 Note that constraint (4) along with(5)force that reviewer would not be assigned to a manuscript whenever ci is smaller than the given threshold T.However, some conference chairs just require that at least one assigned reviewer is exactly expert for that manuscript, constraint(6) instead of(4), is brought into the mathematical model to ensure that at least one reviewer whose matching degree for manuscript i is greater than or equal to T[20 max{cnxn}≥T In case the above model does not have a feasible solution or takes too much run ing-time, the formulation will be reformed as a multi-objective(7) by relaxing con- strains(2)and (3). The parameter a>0 is the penalty weight for missing reviewers (a=0.5 in the work of [23]), and B20 is for reviewers'over-workload(B is set to be o when the over-workload problem is not taken into consideration). These two free control parameters enable us to freely shape the solution structure. Once an opti- mal/feasible solution is found, the assignment procedure comes to the end, otherwise

A Survey on Reviewer Assignment Problem 723 from the corresponding category, and matching degrees are acquired by a computa￾tion mechanism set by organizer [20]. The above computation methods are raised as increasing matching degree, which is appropriate to the optimizations of maximizing assignment utility. In fact, the matching degree ij c can easily be transformed as smaller value for higher preference, with the evaluation on the converse way, so as to be adapted for minimum optimization. With the “matching degree” matrix C , a binary variable ij x , whose value is 1 if manuscript i is assigned to reviewer j and 0 otherwise, is brought to present the integer programming (IP) formulation of the RAP (here we take the increasing match￾ing degree as example): 1 1 max p r ij ij i j c x = = ∑∑ (1) Subject to 1 r ij i j x a = ∑ = (2) 1 p ij j i x b = ∑ ≤ (3) ij ij c x T ⎢ ⎥ ≤ ⎢ ⎥ ⎣ ⎦ (4) 0 1 ij x or = (5) Note that constraint (4) along with (5) force that reviewer would not be assigned to a manuscript whenever ij c is smaller than the given threshold T . However, some conference chairs just require that at least one assigned reviewer is exactly expert for that manuscript, constraint (6) instead of (4), is brought into the mathematical model to ensure that at least one reviewer whose matching degree for manuscript i is greater than or equal to T [20]. { } 1 max ij ij j r cx T ≤ ≤ ≥ (6) In case the above model does not have a feasible solution or takes too much run￾ning-time, the formulation will be reformed as a multi-objective (7) by relaxing con￾strains (2) and (3). The parameter α > 0 is the penalty weight for missing reviewers ( 5 α = 0. in the work of [23]), and β ≥ 0 is for reviewers’ over-workload ( β is set to be 0 when the over-workload problem is not taken into consideration). These two free control parameters enable us to freely shape the solution structure. Once an opti￾mal/feasible solution is found, the assignment procedure comes to the end, otherwise

724 F Wang B Chen and Z miao new reviewers are invited and the original model is run( 24, 36]. Reviewers'workload balancing can also be considered in the manner of (7)as a multi-objective problem m∑∑c-∑mx104-2x}-B∑mx0∑x-b}( OR was brought into RAP in the latest decade. GAP formulation is the most popu ar method, as rAP plays a good incident for OR researchers to examine their algo- rithms. Network Flow Model and set-covering optimization were also discussed. The review presented in Table 2 shows that most of the researches are IP formulation of GAP with binary variables which indicate the assignment, and MIP models are lately laxation Network Flow Model, whose polynomial-time algorithm has been proposed also works well for RAP when manuscripts and reviewers are represented as nodes in bipartite graph, with weights of arcs connected nodes representing the matching de gree; manuscripts slots and reviewers'workload limitation can be fulfilled by adding a source and a sink, and set the capabilities of all the arcs. Studies that focus on mod eling seldom consider algorithm, and usually solve the model by the professional mathematic software (e.g. ILOG CPLEX, MATALAB), thus the solution is only examined in communities related to computer science, while conference chair of art communities cannot handle Table 2. Review on the RAP optimization Algorithm Hartvigsen et al. [201 Network Flow N/A(using SAS/OR) Benferhat and Lang [41 GAP(IP) Tian et al. 371 GAP(IP) Merelo-Guervos et al. [29 GAP(IP) Evolutionary Merelo-Guervos and Casti- GAP(IP) Hybrid( Greedy/Evolutionary) llo-Valdivieso [28 Cook et al. [12] Set-Covering(IP) Greedy Janak et al. [241 GAP(IP/MIP N/Amusing CPleX) Goldsmith and Solan [17 GAP(IP) Note: GAP-General Assignment Problem; IP-Integer Programming: MIP-Mixed Integer Pro- 5 Concluding Remarks and Future Perspectives The focal point of interest in this paper is to review the up-to-date research on RAP which has important practical interests in several field including resource allocation staff scheduling decision making etc. This article outlined the existing discussion and solution for RAP, which focused on the three different phases: (1)recognizing

724 F. Wang, B. Chen, and Z. Miao new reviewers are invited and the original model is run [24,36]. Reviewers’ workload balancing can also be considered in the manner of (7) as a multi-objective problem. 11 1 1 1 1 max max 0, max 0, pp p r rr ij ij i ij ij j ij i j j i cx a x x b α β == = = = = ⎧ ⎫ ⎧ ⎫ −⋅ − −⋅ − ⎨⎬ ⎨⎬ ⎩ ⎭ ⎩ ⎭ ∑∑ ∑ ∑ ∑ ∑ (7) OR was brought into RAP in the latest decade. GAP formulation is the most popu￾lar method, as RAP plays a good incident for OR researchers to examine their algo￾rithms. Network Flow Model and set-covering optimization were also discussed. The review presented in Table 2 shows that most of the researches are IP formulation of GAP with binary variables which indicate the assignment, and MIP models are lately raised by bringing continual variables into the objective function for constrains re￾laxation. Network Flow Model, whose polynomial-time algorithm has been proposed, also works well for RAP when manuscripts and reviewers are represented as nodes in bipartite graph, with weights of arcs connected nodes representing the matching de￾gree; manuscripts slots and reviewers’ workload limitation can be fulfilled by adding a source and a sink, and set the capabilities of all the arcs. Studies that focus on mod￾eling seldom consider algorithm, and usually solve the model by the professional mathematic software (e.g. ILOG CPLEX, MATALAB), thus the solution is only examined in communities related to computer science, while conference chair of arts communities cannot handle. Table 2. Review on the RAP optimization Study Model Algorithm Hartvigsen et al. [20] Network Flow N/A(using SAS/OR) Benferhat and Lang [4] GAP(IP) VCSP Tian et al. [37] GAP(IP) N/A Merelo-Guervós et al. [29] GAP(IP) Evolutionary Merelo-Guervós and Casti￾llo-Valdivieso [28] GAP(IP) Hybrid(Greedy/Evolutionary) Cook et al. [12] Set-Covering(IP) Greedy Janak et al. [24] GAP(IP/MIP) N/A(using CPLEX) Goldsmith and Solan [17] Survey N/A Sun et al. [36] GAP(IP) N/A Schirrer et al. [33] GAP(IP/MIP)/Network Flow Memetic/(using XPressTM) Note: GAP-General Assignment Problem; IP-Integer Programming; MIP-Mixed Integer Pro￾gramming. 5 Concluding Remarks and Future Perspectives The focal point of interest in this paper is to review the up-to-date research on RAP which has important practical interests in several field including resource allocation, staff scheduling, decision making, etc. This article outlined the existing discussion and solution for RAP, which focused on the three different phases: (1) recognizing

A Survey on Reviewer Assignment Problem 725 assignment criteria and procedure, (2)computing the matching degree between manu- scripts and reviewers, (3)approaching the assignment. Hence, discussions on the first phase, IR that computed the match, OR that optimized the assignment are addressed in the above ordinal sections. Moreover, all these researches based themselves mostly in one field Future research on this field can be extended towards a variety of directions. First, widely acceptable reviewer assignment criteria and procedure are required for a cer- tain field to ensure the democracy and equity: study on this field should be raised along with peer-review. Then, solutions employ IR work really well in the situation of time and labor intensive, but new technology and supervised can be brought in to improve the match computation to be closer to the actual demand. Further, innovation in mathematical modeling and algorithm is encouraged, such as network solution and set covering methodology are brought in to enrich RAP which is a traditional GAP Another further, empirical study or tracking research can be done to study reviewer performance(such as review efficiency, quality, etc), in order to take reviewer per- formance into the future assignment consideration. Another future research direction, which could be of interest at all levels of research and practice, is the investigation of implementing RAP solution by popular office software. Seeing that previous research are all in respond to the predicament in science community and solution are addressed using a corresponding professional tool of the certain field(e.g. Al, Computer Science, Decision Science, OR, etc), additional re- source should be called for to deal with RAP in arts community. On condition that RAP solution can be implemented by popular software tool(e.g. Excel), it will be more convenient for program chair of arts community to manage conference proceeding There is a clear need for more field work where studies involving actual stake holders with their own problems may reveal ways of improving technologies, meth dologies and models, or suggesting where new approaches may be needed. This paper has identified certain issues in RAP research approach and direction. While, as pointed out above, the literature review carried out has some limitations, it is intended that a follow-up study will examine the principle and performance of reviewer as- signment module in the conference managing software(e.g. Cyber Chair, ConfMan Conftool, AAA S/W, Puma, SIGACT). It is also expected that this historical analysis will be continuously updated and reported at regular intervals in the futur References 1. Basu, C, Hirsh, H, Cohen, w: Recommendation as Classification: Using Social and Con- tent-based Information in Recommendation. In: 15th national/tenth conference on artifi- cial intelligence/Innovative applications of artificial intelligence, pp. 714-720. AAAL, USA(1998) 2. Basu, C, Hirsh, H, Cohen, w, Nevill-Manning, C: Technical paper recommendation: A study in combining multiple information sources. Journal of Artificial Intelligence Re arch14,231-252(2001) 3. Bausch, DO, Brown, G.G., Hundley, D R, Rapp, S.H., Rosenthal, R E: Mobilizing Ma- rine Corps Officers. Interfaces 21(4), 26-38(1991) 4. Benferhat, S, Lang, J. Conference Paper Assignment. International Journal of Intelligent Systems16,1183-1192(2001)

A Survey on Reviewer Assignment Problem 725 assignment criteria and procedure, (2) computing the matching degree between manu￾scripts and reviewers, (3) approaching the assignment. Hence, discussions on the first phase, IR that computed the match, OR that optimized the assignment are addressed in the above ordinal sections. Moreover, all these researches based themselves mostly in one field. Future research on this field can be extended towards a variety of directions. First, widely acceptable reviewer assignment criteria and procedure are required for a cer￾tain field to ensure the democracy and equity; study on this field should be raised along with peer-review. Then, solutions employ IR work really well in the situation of time and labor intensive, but new technology and supervised can be brought in to improve the match computation to be closer to the actual demand. Further, innovation in mathematical modeling and algorithm is encouraged, such as network solution and set covering methodology are brought in to enrich RAP which is a traditional GAP. Another further, empirical study or tracking research can be done to study reviewer performance (such as review efficiency, quality, etc), in order to take reviewer per￾formance into the future assignment consideration. Another future research direction, which could be of interest at all levels of research and practice, is the investigation of implementing RAP solution by popular office software. Seeing that previous research are all in respond to the predicament in science community and solution are addressed using a corresponding professional tool of the certain field (e.g. AI, Computer Science, Decision Science, OR, etc), additional re￾source should be called for to deal with RAP in arts community. On condition that RAP solution can be implemented by popular software tool (e.g. Excel), it will be more convenient for program chair of arts community to manage conference proceeding. There is a clear need for more field work where studies involving actual stake￾holders with their own problems may reveal ways of improving technologies, meth￾odologies and models, or suggesting where new approaches may be needed. This paper has identified certain issues in RAP research approach and direction. While, as pointed out above, the literature review carried out has some limitations, it is intended that a follow-up study will examine the principle and performance of reviewer as￾signment module in the conference managing software (e.g. CyberChair, ConfMan, Conftool, AAA S/W, Puma, SIGACT). It is also expected that this historical analysis will be continuously updated and reported at regular intervals in the future. References 1. Basu, C., Hirsh, H., Cohen, W.: Recommendation as Classification: Using Social and Con￾tent-based Information in Recommendation. In: 15th national/tenth conference on Artifi￾cial intelligence/Innovative applications of artificial intelligence, pp. 714–720. AAAI, USA (1998) 2. Basu, C., Hirsh, H., Cohen, W., Nevill-Manning, C.: Technical paper recommendation: A study in combining multiple information sources. Journal of Artificial Intelligence Re￾search 14, 231–252 (2001) 3. Bausch, D.O., Brown, G.G., Hundley, D.R., Rapp, S.H., Rosenthal, R.E.: Mobilizing Ma￾rine Corps Officers. Interfaces 21(4), 26–38 (1991) 4. Benferhat, S., Lang, J.: Conference Paper Assignment. International Journal of Intelligent Systems 16, 1183–1192 (2001)

F Wang B. Chen. and Z miao 5. Biswas, H.K., Hasan, M. M. Using Publications and Domain Knowledge to Build search Profiles: an Application in Automatic Reviewer Assignment. In: 2007 Intermat Conference on Information and Communication Technology, pp 82-86(2007) 6. Cameron, D, Aleman-Meza, B, Arpinar, B. Collecting Expertise of Researchers for Find- ing Relevant Experts in a Peer-Review Setting. In: First International ExpertFinder Work 7. Caron, G, Hansen, P, Jaumard. B. The assignment problem with seniority and job prior- ity constraints. Operations Research 47(3), 449-454(1999) 8. Carter, M.W., Tovey, C.A. When is the classroom assignment problem hard? Operations Research40(1),S28-S39(1992) and Review Model is Killing Research and Wasting Your Moneythe Current Publication 9. Casati, F, Giunchiglia, F, Marchese, M. Publish and perish: Why http://www.acmorg/ubiquity/views/v8103_fabio.html 10. Cohen, P R, Kjeldsen, R. Information retrieval by constrained spreading activation in se- mantic network Information Processing Management 23(4), 255-268(1987) w: Web-collaborative Filtering: Recommending Music by Crawling the Web. Computer Networks 33(1-6), 685-698(2000) 12. Cook, W.D., Golany, B, Kress, M, Penn, M., Raviv, T: Optimal Allocation of P to Reviewers to Facilitate Effective Ranking. Management Science 51(4), 655-661 13. Dell'Amico, M, Martello, S: The k-cardinality assignment problem. Discrete mathematics76(1-3),103-121(1997) 14. Dumais, S, Nielsen, J: Automating the assignment of submitted manuscripts to reviewers. Research and Development in information Retrieval, 233-244( 1992) 15. Geller, J: How JCAI 1999 can prove the value of Al by using Al. In: 15th International Joint Conference on Artificial Intelligence, pp 55-58(1997) 16. Geller, J, Scherl, R: Challenge: Technology for Automated Reviewer Selection(1997), http://njit.edu/-geller/finalchall.ps1997 17. Goldsmith, J, Solan, R.H.: The Al Conference Paper Assignment Problem http://www.cs.uic.edu/-sloan/my-papers/godlsmithsloanpaperassignment.pdf 18. Gupta, D, Digiovanni, M, Narita, H, Goldberg, K. Jester 2.0: A new Lineartime Col- Jokes. In: Wo at ACM SIGIR 1999(1999 19. Hansen, P, Wendell, R.E.: A note on airline commuting Interfaces 11(12),85-87(1982) 20. Hartvigsen, D, Wei, J.C., Czuchlewski, R: The Conference Paper-Reviewer Assignment Problem. Decision Sciences 30(3), 865-876(1999) 1. Hettich, S, Pazzani, M.J.: Mining for Proposal Reviewers: Lessons Learned at the Na- tional science foundation ln: 12th acm sigKdd international conference on Knowl- Ige discovery and data mining, pp. 862-871. ACM Press, New York(2006) 22. Hill, W, Stead, L, Rosenstein, M., Furnas, G. Recommending and evaluating Choices in a Virtual Community of Use. In: SIGCHI conference on Human factors in computing sys- ems, pp 23. Hofmann, T: Probabilistic Latent Semantic Analysis. In: 15th Conference on Uncertaint in Artificial Intelligence, pp 289-296(1999) 24. Janak, S.L., Taylor, M.s., Floudas, C.A., Burka, M, Mountzizris, T.J.: Novel and Effec tive Integer Optimization Approach for the NSF Panel-Assignment Problem: A Multire source and Preference-Constrained Generalized Assignment Problem. Ind. Eng. Chem. Res.45(1),258-265(2006) 25. Klingman, D, Phillips, N: Topological and computational aspects of preemptive multicri- teria military personnel assignment problems. Manage. Sci. 30(1), 1362-1375(1984

726 F. Wang, B. Chen, and Z. Miao 5. Biswas, H.K., Hasan, M.M.: Using Publications and Domain Knowledge to Build Re￾search Profiles: an Application in Automatic Reviewer Assignment. In: 2007 International Conference on Information and Communication Technology, pp. 82–86 (2007) 6. Cameron, D., Aleman-Meza, B., Arpinar, B.: Collecting Expertise of Researchers for Find￾ing Relevant Experts in a Peer-Review Setting. In: First International ExpertFinder Work￾shop (2007) 7. Caron, G., Hansen, P., Jaumard, B.: The assignment problem with seniority and job prior￾ity constraints. Operations Research 47(3), 449–454 (1999) 8. Carter, M.W., Tovey, C.A.: When is the classroom assignment problem hard? Operations Research 40(1), S28–S39 (1992) 9. Casati, F., Giunchiglia, F., Marchese, M.: Publish and perish: Why the Current Publication and Review Model is Killing Research and Wasting Your Money, http://www.acm.org/ubiquity/views/v8i03_fabio.html 10. Cohen, P.R., Kjeldsen, R.: Information retrieval by constrained spreading activation in se￾mantic network. Information Processing & Management 23(4), 255–268 (1987) 11. Cohen, W., Fan, W.: Web-collaborative Filtering: Recommending Music by Crawling the Web. Computer Networks 33(1-6), 685–698 (2000) 12. Cook, W.D., Golany, B., Kress, M., Penn, M., Raviv, T.: Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking. Management Science 51(4), 655–661 (2005) 13. Dell’Amico, M., Martello, S.: The k-cardinality assignment problem. Discrete Applied Mathematics 76(1-3), 103–121 (1997) 14. Dumais, S., Nielsen, J.: Automating the assignment of submitted manuscripts to reviewers. Research and Development in information Retrieval, 233–244 (1992) 15. Geller, J.: How IJCAI 1999 can prove the value of AI by using AI. In: 15th International Joint Conference on Artificial Intelligence, pp. 55–58 (1997) 16. Geller, J., Scherl, R.: Challenge: Technology for Automated Reviewer Selection (1997), http://njit.edu/~geller/finalchall.ps1997 17. Goldsmith, J., Solan, R.H.: The AI Conference Paper Assignment Problem, http://www.cs.uic.edu/~sloan/my-papers/GodlsmithSloanPaperAssignment.pdf 18. Gupta, D., Digiovanni, M., Narita, H., Goldberg, K.: Jester 2.0: A new Lineartime Col￾laborative Filtering Algorithm Applied to Jokes. In: Workshop on Recommender Systems at ACM SIGIR 1999 (1999) 19. Hansen, P., Wendell, R.E.: A note on airline commuting. Interfaces 11(12), 85–87 (1982) 20. Hartvigsen, D., Wei, J.C., Czuchlewski, R.: The Conference Paper-Reviewer Assignment Problem. Decision Sciences 30(3), 865–876 (1999) 21. Hettich, S., Pazzani, M.J.: Mining for Proposal Reviewers: Lessons Learned at the Na￾tional Science Foundation. In: 12th ACM SIGKDD international conference on Knowl￾edge discovery and data mining, pp. 862–871. ACM Press, New York (2006) 22. Hill, W., Stead, L., Rosenstein, M., Furnas, G.: Recommending and Evaluating Choices in a Virtual Community of Use. In: SIGCHI conference on Human factors in computing sys￾tems, pp. 195–201. ACM Press/Addison-Wesley Publishing Co., New York (1995) 23. Hofmann, T.: Probabilistic Latent Semantic Analysis. In: 15th Conference on Uncertainty in Artificial Intelligence, pp. 289–296 (1999) 24. Janak, S.L., Taylor, M.S., Floudas, C.A., Burka, M., Mountzizris, T.J.: Novel and Effec￾tive Integer Optimization Approach for the NSF Panel-Assignment Problem: A Multire￾source and Preference-Constrained Generalized Assignment Problem. Ind. Eng. Chem. Res. 45(1), 258–265 (2006) 25. Klingman, D., Phillips, N.: Topological and computational aspects of preemptive multicri￾teria military personnel assignment problems. Manage. Sci. 30(1), 1362–1375 (1984)

A Survey on Reviewer Assignment Problem 727 26. Konstan, J, Miller, B, Maltz, D, Herlocker, L, Gordon, L, Riedl, J: Grouplens: Apply Collaborative Filtering to Usenet News. Communications of the ACM 40(3).77-87 27. LeBlanc, L.J., Randels, D, Swann, T K: Heery Internationals Spreadsheet Optimization Model for Assigning Managers to Construction Projects. INTERFACE 30(6), 95-106 28. Merelo-Guervos, J.J., Castillo-Valdivieso, P. Conference Paper Assignment Using a Combined Greedy/ Evolutionary Algorithm. In: Yao, X, Burke, E.K., Lozano, J.A Smith, J, Merelo-Guervos, JJ. Bullinaria, J.A.. Rowe, J E, Tino, P, Kaban, A. Schwe- fel, H.-P.(eds )PPSN 2004. LNCS, vol. 3242, pp. 602-611. Springer, Heidelberg(2004) 9. Merelo-Guervos. JJ. Garcia-Castellano, F. Castillo. PA. Arenas. M. G. How Evolu tionary Computation and Perl saved my conference. In: Sanchez, L(ed ) Segundo Con greso Espanol sobre Metaheuristicas, Algoritmos Evolutivos y Bioinspirados, pp 93-99 (2003) 30. Popescul, A, Ungar, L.H. Pennock, D M, Lawrence, S: Probabilistic Models for Unified Collaborative and Content-base Recommendation in Sparse-Data Environments. In: 17th Conference on Uncertainty in Artificial Intelligence, pp. 437-444. Morgan Kaufmann Pub lishers Inc, San Francisco(2001) 31. Rodriguez, M.A., Bollen, J, Van de Sompel, H: Mapping the Bid Behavior of Conference Referees. Journal of Informetrics 1(1), 62-82(2007) 32. Rodriguez, M.A., Bollen, J: An Algorithm to Determine Peer-Reviewers. Arxiv preprint csDL0605112(2006 33. Schirmer, A, Doerner, K F, Hartl, R F. Reviewer Assignment for Scientific Articles using Memetic Algorithms OR/CS Interfaces Series 39, 113-134 (2007) 34. Scott, A Peer review and the relevance of science. Futures 39, 827-845(2007) 35. Shardanand, U. Maes. P. Social Information Filtering: Algorithms for Automating"Word of Mouth". In: SIGCHI conference on Human factors in computing systems, pp 210-217 ACM Press/Addison-Wesley Publishing Co, New York(1995) 36. Sun, Y H, Ma, J, Fan, Z.P., Wang, J. A Hybrid Knowledge and Model Approach for Re viewer Assignment. In: 40th Annual Hawaii International Conference on System Sciences, p 47. IEEE Computer Society, Washington(2007) 37. Tian, Q, Ma, J, Liu, O: A Hybrid Knowledge and Model System for r&D Project Selec- on Expert Systems with Applications 23(3), 265-271(2002) 8. Veronika, A, Riantini, L.S., Trigunarsyah, B: Corrective Action Recommendation For Project Cost Variance in Construction Material Management. In: Kanok-Nukulchai, W. lunasinghe S, Anwar, N.(eds 10th East Asia-Pacific Conference on Structural engi- eering and Construction 2005, pp 23-28 (2006) 39. Watanabe, S, Ito, T, Ozono, T, Shintani, T: A Paper Recommendation Mechanism for the Research Support System Papits In: International Workshop on Data Engineering Is sues in E-Commerce, pp 71-80(2005) 40. Weber, R. The Journal Review Process: a Manifesto for Change. Communications of the Association for Information Systems 2(2-3)(1999) 41. Yarowsky, D, Florian, R: Taking the load off the conference chairs: towards a digital pa- per-routing assistant In: 1999 Joint SIGDAT Conference on Empirical Methods in NLP and Very-Large Corpora(1999

A Survey on Reviewer Assignment Problem 727 26. Konstan, J., Miller, B., Maltz, D., Herlocker, L., Gordon, L., Riedl, J.: Grouplens: Apply￾ing Collaborative Filtering to Usenet News. Communications of the ACM 40(3), 77–87 (1997) 27. LeBlanc, L.J., Randels, D., Swann, T.K.: Heery International’s Spreadsheet Optimization Model for Assigning Managers to Construction Projects. INTERFACE 30(6), 95–106 (2000) 28. Merelo-Guervós, J.J., Castillo-Valdivieso, P.: Conference Paper Assignment Using a Combined Greedy/ Evolutionary Algorithm. In: Yao, X., Burke, E.K., Lozano, J.A., Smith, J., Merelo-Guervós, J.J., Bullinaria, J.A., Rowe, J.E., Tiňo, P., Kabán, A., Schwe￾fel, H.-P. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 602–611. Springer, Heidelberg (2004) 29. Merelo-Guervós, J.J., García-Castellano, F.J., Castillo, P.A., Arenas, M.G.: How Evolu￾tionary Computation and Perl saved my conference. In: Sánchez, L. (ed.) Segundo Con￾greso Español sobre Metaheurísticas, Algoritmos Evolutivos y Bioinspirados, pp. 93–99 (2003) 30. Popescul, A., Ungar, L.H., Pennock, D.M., Lawrence, S.: Probabilistic Models for Unified Collaborative and Content-base Recommendation in Sparse-Data Environments. In: 17th Conference on Uncertainty in Artificial Intelligence, pp. 437–444. Morgan Kaufmann Pub￾lishers Inc., San Francisco (2001) 31. Rodriguez, M.A., Bollen, J., Van de Sompel, H.: Mapping the Bid Behavior of Conference Referees. Journal of Informetrics 1(1), 62–82 (2007) 32. Rodriguez, M.A., Bollen, J.: An Algorithm to Determine Peer-Reviewers. Arxiv preprint cs.DL/0605112 (2006) 33. Schirrer, A., Doerner, K.F., Hartl, R.F.: Reviewer Assignment for Scientific Articles using Memetic Algorithms. OR/CS Interfaces Series 39, 113–134 (2007) 34. Scott, A.: Peer review and the relevance of science. Futures 39, 827–845 (2007) 35. Shardanand, U., Maes, P.: Social Information Filtering: Algorithms for Automating “Word of Mouth”. In: SIGCHI conference on Human factors in computing systems, pp. 210–217. ACM Press/Addison-Wesley Publishing Co., New York (1995) 36. Sun, Y.H., Ma, J., Fan, Z.P., Wang, J.: A Hybrid Knowledge and Model Approach for Re￾viewer Assignment. In: 40th Annual Hawaii International Conference on System Sciences, p. 47. IEEE Computer Society, Washington (2007) 37. Tian, Q., Ma, J., Liu, O.: A Hybrid Knowledge and Model System for R&D Project Selec￾tion. Expert Systems with Applications 23(3), 265–271 (2002) 38. Veronika, A., Riantini, L.S., Trigunarsyah, B.: Corrective Action Recommendation For Project Cost Variance in Construction Material Management. In: Kanok-Nukulchai, W., Munasinghe, S., Anwar, N. (eds.) 10th East Asia-Pacific Conference on Structural Engi￾neering and Construction 2005, pp. 23–28 (2006) 39. Watanabe, S., Ito, T., Ozono, T., Shintani, T.: A Paper Recommendation Mechanism for the Research Support System Papits. In: International Workshop on Data Engineering Is￾sues in E-Commerce, pp. 71–80 (2005) 40. Weber, R.: The Journal Review Process: a Manifesto for Change. Communications of the Association for Information Systems 2(2-3) (1999) 41. Yarowsky, D., Florian, R.: Taking the load off the conference chairs: towards a digital pa￾per-routing assistant. In: 1999 Joint SIGDAT Conference on Empirical Methods in NLP and Very-Large Corpora (1999)