Meeting User Information Needs in Recommender Systems A DISSERTATION SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF MINNESOTA BY Sean Michael McNee IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Joseph A Konstan, Advisor June 2006 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
UMI Number 3230139 Copyright 2006 by McNee. Sean Michael All rights reserved NFORMATION TO USERS The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed-through, substandard margins, and improper alignment can adversely affect reproduction In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion UMI UMI Microform 3230139 Copyright 2006 by ProQuest Information and Learning Company All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code ProQuest Information and Learning Company 300 North Zeeb road P O. Box 1346 Ann Arbor. MI 48106-1346 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Copyright Sean Michael McNee 2006 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
UNIVERSITY OF MINNESOTA This is to certify that i have examined this copy of a doctoral dissertation by Sean Michael McNee and have found that it is complete and satisfactory in all respects, and that any and all revisions required by the final examining committee have been made Joseph A Konstan Name of Faculty Advisor signature of Faculty Advisor d June 2o0 Date GRADUATE SCHOOL Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
ACKNOWLEDGEMENTS Many people helped make this dissertation a reality. First, I would like to thank my hesis committee, John Riedl, John Carlis, Paul Garrett, and Joseph Konstan. I would especially like to thank Joe for being a fantastic advisor; I have learned more from you than you realize. Thank you to my paper writing colleagues, especially Roberto Torres, Jr and Cai-Nicholas Ziegler, it was wonderful working with you. Thank you to everyone in GroupLens Research, especially Dan Cosley, Al Mamunur(Mamun) Rashid, and Shyong(Tony) Lam. Because of your support, encouragement, and friendship, GroupLens Research is a special place. Thanks also those who've already gotten their Ph. D. from Minnesota for their support and encouragement, including Jon Herlocker, Brad Miller, J. Ben Schafer, Brian Bailey, Harini Veeraraghavan, and Doug Perrin. I would also like to thank Ed H Chi for his support and advice during my graduate school career. I have been blessed with many great teachers and professors, especially Jack Goldfeather and Jeff Ondich from Carleton College who showed me the love of computer science, and my sophomore year high school English teacher, Mrs. Skoy. As far as I know, I still have extra credit"in her heart". Finally, I want to thank my wonderful group of friends, my family, and everyone who helped and supported me through this process. I could not have done it otherwise. In a different set of thanks, i want to thank the University of Minnesota Computer Science department Systems Staff for keeping all of the machines running. Thanks also to nec Research for the ResearchIndex data, thanks to Book Crossing. com for allowing us to collect data and run experiments, and thanks to the ACM for providing us with data from the ACM Digital Library Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
For my parents, who always believed Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
ABSTRACT In order to build relevant, useful, and effective recommender systems, researchers need to understand why users come to these systems and how users judge recommendation lists Today, researchers use accuracy-based metrics for judging goodness. Yet these metrics cannot capture users'criteria for judging recommendation usefulness. We need to rethink recommenders from a user's perspective: they help users find new information Thus, not only do we need to know about the user, we need to know what the user is looking for. In this dissertation, we explore how to tailor recommendation lists not just to a user, but to the users current information seeking task. We argue that each recommender algorithm has specific strengths and weaknesses, different from other algorithms. Thus, different recommender algorithms are better suited for specific users and their information seeking tasks. A recommender system should, then, select and tun the appropriate recommender algorithm(or algorithms )for a given user/information eeking task combination. To support this, we present results in three areas. First, we apply recommender systems in the domain of peer-reviewed computer science research pap domain where users have external criteria for selecting items to consume. The effectiveness of our approach is validated through several sets of experiments. Second we argue that current recommender systems research in not focused on user needs, but rather on algorithm design and performance. To bring users back into focus, we reflect on how users perceive recommenders and the recommendation process, and present Human-Recommender Interaction theory, a framework and language for describing recommenders and the recommendation lists they generate. Third, we look to different ways of evaluating recommender systems algorithms. To this end, we propose a new set of recommender metrics, run experiments on several recommender algorithms using these metrics, and categorize the differences we discovered. Through Human Recommender Interaction and these new metrics, we can bridge users and their needs with recommender algorithms to generate more useful recommendation lists Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
TABLE OF CONTENTS CHAPTER 1 INTRODUCTION THESIS STATEMENT..,... External validation The Curse of Accuracy BUILDING BRIDGES RESEARCH APPROACH AND CONTRIBUTIONS…,… Recommending research Papers…… A User-Centric Approach to the Recommendation Process Understanding Recommender algorithms DISSERTATION ORGANIZATION 12 CHAPTER 2 RELATED AND PREVIOUS WORK RECOMMENDER SYSTEMS AND PERSONALIZATION... 14 Content-Based Recommenders Collaborative Filtering 8 Case-Based Reasoning and Conversational Recommenders A SUMMARY OF COMMON RECOMMENDER PROBLEMS en Evaluating Recon Problems with Collaborative Recommender algorithms. Problems with Content-based Recommender algorithms Problems with Knowledge-based Recommender Algorithms CITATION INDEXING AND RECOMMENDING RESEARCH PAPERS THEORIES OF INFORMATION SEEKING SEARCHSEARCH ENGINES, AND DIGITAL LIBRARIES CONCLUSION 32 CHAPTER 3 CONCERNING RECOMMENDING, RECOMMENDER ALGORITHMS, AND RECOMMENDER METRICS…… BACKGROUND AND DEFINITIONS he Ratings Matrix The recommendation Process USER-BASED COLLABORATIVE FILTERING EM-BASED COLLABORATIVE FILTERING EXTENSIONS TO COLLABORATIVE FILTERING ALGORITHMS Denser Collaborative Filtering Symmetric Collaborative Filtering NAIVE BAYES CLASSIFIER PROBABILISTIC LATENT SEMANTIC ANALYSIS TF/IDF CONTENT-BASED FILTERING PREDICTIVE ACCURACY AND DECISION SUPPORT METRICS CHAPTER 4 RECOMMENDATION LISTS AND INTRA- LIST SIMILARITY. INTRA-LIST SIMILARITY EXPERIMENTS Offline Experiments Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Online experiments DISCUSSION AND IMPLICATIONS CHAPTER 5 RECOMMENDING CITATIONS FOR RESEARCH PAPERS INTEGRATING RECOMMENDERS INTO THE DOMAIN OF RESEARCH PAPERS PURE RECOMMENDER ALGORITHMS FOR RESEARCH PAPERS Collaborative algorithms Baseline algorithms. PURE ALGORITHM EXPERIMENTS…… Offline Experiment… 8888 O Experiment Pure Algorithm Discussion COMBINING CONTENT AND COLLABORATIVE FILTERING HYBRID RECOMMENDER ALGORITHMS IN THIS DOMAIN HYBRID RECOMMENDER EXPERIMENTS iline Experiment. 09 Online Experiment 12 Hybrid Algorithm Discussion 22 CONCLUSION(A FUNNY THING HAPPENED.) CHAPTER 6 HUMAN-RECOMMENDER INTERACTION THEORY 126 THE GULF OF INTENTIOI HUMAN RECOMMENDER INTERACTION: A USER-CENTRIC PERSPECTIVE 128 THE PILLARS OF HRI The Recommendation dialog-…… 130 The re nder personality he User Information Seeking Task 132 THE ASPECTS OF HRI 133 Aspects of the Recommendation dialog……… 134 Aspects of the Recommender Personality 139 Aspects of the User Information Seeking Task THE HRI ANALYTIC PROCESS MODEL 148 APPLYING HRI AND THE PROCESS MODEL TO RECOMMENDER DESIGN LIMITATIONS OF HRI AND THE PROCESS MODEL 151 153 CHAPTER 7 RECOMMENDER TASKS IN A DIGITAL LIBRARY 154 FOUR KINDS OF USERS.. USER TASKS RELATIONSHIPS BETWEEN TASKS The large set Phenomenon The Fluidity of" Find More References"…… 160 PERSONAS IN THIS DOMAⅣN APPLYING HRI TO THE DOMAIN OF DIGITAL LIBRARIES CONCLUSIONS CHAPTER 8 UNDERSTANDING RECOMMENDER ALGORITHMS, PARTI: DESIGNING METRICS. RUNNING BENCHMARKS PREVIOUSLY EXISTING RECOMMENDER METRICS METRIC DISCUSSION BENCHMARKING RECOMMENDER ALGORITHMS 178 Research questions 178 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
179 Experiment Results HRI AND RECOMMENDER ALGORITHMS 195 APPLYING HRI TO THE DOMAIN OF DIGITAL LIBRARIES, REVISITED Discussion and Limitations. 198 CONCLUSION CHAPTER 9 UNDERSTANDING RECOMMENDER ALGORITHMS, PART II: A USER EVALUATION OUR RECOMMENDER ALGORITHMS 199 USER STUDY 204 Experimental Design Experiment Walkthrough 207 210 Analysis and Discussion 2l3 IMPLICATIONS AND FUTURE WORK..... 217 CONCLUSION 219 CHAPTER 10 IMPLICATIONS, FUTURE WORK, AND CONCLUSIONS 221 SUMMARY OF CONTRIBUTIONS AND IMPLICATIONS Recommending research Papers Re-examining the Recommendation Process 222 Understanding Recommender Algorithms 223 FUTURE WORK CONCLUSION Reproduced with permission of the copyright owner. Further reproduction prohibited without permission
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission