Recommender Systems for the Conference Paper Assignment Problem Don col Naren Ramakrishnan Virginia TE ahoo! Research Blacksburg, VA, USA Haifa Israel Blacksburg, VA, USA dconry @cs. vt. edu yehuda@yahoo-inc.com ABSTRACT The primary input to the conference paper assignment We present a recommender problem(CPAP)is a papers x reviewers matrix of"bids' paper assignment, i.e., the expressing interest or disinterest of reviewers to review spe- ons to reviewers. We addres cific papers. The goal is to construct a set of re signments taking into account reviewer capacity constraints, lem)and the optimization of reviewing assignments to sat- adequate numbers of reviews for papers, expertise modeling. isfy global conference criteria(which can be viewed as con straint satisfaction). Due to the paucity of preference data There are three key differences between traditional recom per reviewer or per paper (relative to other recommende mender applications and the CPAP problem. (i)In a tradi ystems applications) we show how we can integrate mul- tional recommender recommendations that meet the needs tiple sources of information to learn reviewer-paper prefer of one user do not affect the satisfaction of other users. In ence models. Our models are evaluated not just in terms of CPAP, on the other hand, multiple users(reviewers )are bid prediction accuracy but in terms of end-assignment quality ding to review the same papers and hence there is the pos- Using a linear program based assignment optin tion sibility of one user's recommendations(assignments) affect we show how our approach better explores the space of un- ing the satisfaction levels(negatively)of other users.Hence supplied assignments to maximize the overall affinities of the design of reviewer preference models must be posed and papers assigned to reviewers. We demonstrate our result studied in an overall optimization framework. (ii)In a con- n real reviewer bidding data from the IEEE ICDm 2007 ventional recommender, the goal is often to recommend new entities that are likely to be of interest, whereas in CPAP, the goal is to ensure that reviewers are predominantly assigned Categories and subject Descriptors their(most)preferred papers. Nevertheless, preference mod- eling is still crucial because it gives the assignment algorithm Systems-Decision support; J 4[Computer Applications): some degree of latitude in aiming to satisfy multiple users Social and Behavioral Sciences sparse data but the amount of 'signal available to model General terms preferences in the CpaP domain is exceedingly small; hence Algorithms, Human Factors we must integrate multiple sources of information to build strong preference models We organize our framework into two stages: 'growing the Recommender systems, collaborative filtering, conference pa- given bids by adapting recommendation techniques to pre- per assignment, linear programming dict unknown reviewer-paper preferences, and identifying a good assignment by optimizing conference criteria. Other pproaches to CPaP (e.g,I) are surveyed elsewhere 1. INTRODUCTION We apply our framework on bids and auxiliary information Modern conferences are beset with excessively high num- (see Fig. 1)gathered from the 7th IEEE Intl. Conf on Data bers of paper submissions. Assigning these papers to ap- Mining(ICDM07) for which the third author was a pro- oropriate reviewers in the program committee(which can gram chair. Similar scope datasets from other conferences constitute a few hundred members) is a daunting task and are not publicly available(also acknowledged in [5])and we hence motivates the use of recommender systems hope our research will spur greater availability.(The Cyber- chair system used by the ICDM series has expressed inter- est in implementing our approach and we plan to approach Easychair and other CMSs as well. )We emphasize that all Permission to make digital or hard copies of all or part of this work for datasets were anonymized before the modeling and analysis personal or classroom use is granted without fee provided that copies are steps conducted here. not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specifi 2. MODELING REVIEW PREFERENCES ReeSys09. October 23-25. 2009. New York, New York, USA We are given ratings(henceforth, interchangeable with Copyright2009ACM978-1-60558-4355/09/1051000. preferences) between m reviewers and n papers.(Recall thatRecommender Systems for the Conference Paper Assignment Problem Don Conry Virginia Tech Blacksburg, VA, USA dconry@cs.vt.edu Yehuda Koren Yahoo! Research Haifa, Israel yehuda@yahoo-inc.com Naren Ramakrishnan Virginia Tech Blacksburg, VA, USA naren@cs.vt.edu ABSTRACT We present a recommender systems approach to conference paper assignment, i.e., the task of assigning paper submis￾sions to reviewers. We address both the modeling of reviewer￾paper preferences (which can be cast as a learning prob￾lem) and the optimization of reviewing assignments to sat￾isfy global conference criteria (which can be viewed as con￾straint satisfaction). Due to the paucity of preference data per reviewer or per paper (relative to other recommender systems applications) we show how we can integrate mul￾tiple sources of information to learn reviewer-paper prefer￾ence models. Our models are evaluated not just in terms of prediction accuracy but in terms of end-assignment quality. Using a linear programming-based assignment optimization, we show how our approach better explores the space of un￾supplied assignments to maximize the overall affinities of papers assigned to reviewers. We demonstrate our results on real reviewer bidding data from the IEEE ICDM 2007 conference. Categories and Subject Descriptors H.4.2 [Information Systems Applications]: Types of Systems—Decision support; J.4 [Computer Applications]: Social and Behavioral Sciences General Terms Algorithms, Human Factors Keywords Recommender systems, collaborative filtering, conference pa￾per assignment, linear programming 1. INTRODUCTION Modern conferences are beset with excessively high num￾bers of paper submissions. Assigning these papers to ap￾propriate reviewers in the program committee (which can constitute a few hundred members) is a daunting task and hence motivates the use of recommender systems. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. RecSys’09, October 23–25, 2009, New York, New York, USA. Copyright 2009 ACM 978-1-60558-435-5/09/10 ...$10.00. The primary input to the conference paper assignment problem (CPAP) is a papers × reviewers matrix of ‘bids’, expressing interest or disinterest of reviewers to review spe￾cific papers. The goal is to construct a set of reviewing as￾signments taking into account reviewer capacity constraints, adequate numbers of reviews for papers, expertise modeling, conflicts of interest, and other global conference criteria. There are three key differences between traditional recom￾mender applications and the CPAP problem. (i) In a tradi￾tional recommender, recommendations that meet the needs of one user do not affect the satisfaction of other users. In CPAP, on the other hand, multiple users (reviewers) are bid￾ding to review the same papers and hence there is the pos￾sibility of one user’s recommendations (assignments) affect￾ing the satisfaction levels (negatively) of other users. Hence the design of reviewer preference models must be posed and studied in an overall optimization framework. (ii) In a con￾ventional recommender, the goal is often to recommend new entities that are likely to be of interest, whereas in CPAP, the goal is to ensure that reviewers are predominantly assigned their (most) preferred papers. Nevertheless, preference mod￾eling is still crucial because it gives the assignment algorithm some degree of latitude in aiming to satisfy multiple users. Finally, (iii) recommender systems are used to working with sparse data but the amount of ‘signal’ available to model preferences in the CPAP domain is exceedingly small; hence we must integrate multiple sources of information to build strong preference models. We organize our framework into two stages: ‘growing’ the given bids by adapting recommendation techniques to pre￾dict unknown reviewer-paper preferences, and identifying a good assignment by optimizing conference criteria. Other approaches to CPAP (e.g., [1]) are surveyed elsewhere [2]. We apply our framework on bids and auxiliary information (see Fig. 1) gathered from the 7th IEEE Intl. Conf on Data Mining (ICDM’07) for which the third author was a pro￾gram chair. Similar scope datasets from other conferences are not publicly available (also acknowledged in [5]) and we hope our research will spur greater availability. (The Cyber￾chair system used by the ICDM series has expressed inter￾est in implementing our approach and we plan to approach Easychair and other CMSs as well.) We emphasize that all datasets were anonymized before the modeling and analysis steps conducted here. 2. MODELING REVIEW PREFERENCES We are given ratings (henceforth, interchangeable with preferences) between m reviewers and n papers. (Recall that