FULL PAPER Proceedings of the ACM RecSys 2010 Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces(UCERSTI) Barcelona Spain, Sep 30, 2010 Published by CEUR-WS. org. ISSN 1613-0073, online ceur-ws.org/ol-612/paper3.pdf Our overall motivation for this research was to understand the crucial factors that influence the user adoption of recommenders : he eo mmernsted gave m e ms suggest nded to me(reverse Another motivation is to come up with a subjective evaluation questionnaire that other researchers and practitioners can employ However, it is unlikely that a 60-item questionnaire can be A.1.2 Relative Accuracy administered for a quick and easy evaluation. This has motivated The recommendation I received better fits my interests than us in proposing a simplified model based on our past research. hat I may receive from a friend Between 2005 and 2010, we have administered Il subjective A recommendation from my friends better suits my interests questionnaires on a total of 807 subjects [4, 5, 6, 12, 13, 14, 23, 24] than the recommendation from this system(reverse scale) tial questionnaires covered some of the four categories lentified in the ResQue. As we conducted more experiments, we A.1.3 Familiaris ecame more convinced of the four categories and used all of them in recent studies. On average, between 12 and 15 questions Some of the recommended items are familiar to me were used. Based this previous work, we have synthesized and I am not familiar with the items that were recommended to me organized a total of 15 questions as a simplified model for the (reverse scale) purpose of performing a quick and easy usability and adoption evaluation of a recommender(see questions with' sign) A. 4 Attractiveness 5. CONCLUSION AND FUTURE WORK User evaluation of recommender systems is a crucial subject of 4.1.5 Enjoyability study that requires a deep understanding, development and testing I enjoyed the items recommended to me of the right dimensions(or constructs) and the standardization of the questions used. The framework described in thi A 1.6 Novelty presents the first attempt to develop a complete and valuation framework that measures users'subjective The items recommended to me are novel and interesting.. based on their experience towards a recommender The recommender system is educational The recommender system helps me discover new pr ResQue consists of a set of 13 constructs and 60 questions for a I could not find new items through the recommender(reverse high-quality recommender system from the user point of view and can be used as a standard guideline for a user evaluation. It can also be adapted to a custom-made user evaluation by tailoring it in 4.1.6 Diversin an individual research context. Researchers and practitioners can use these questionnaires with eas The items recommended to me are diverse. The items recommended to me are similar to each other satisfaction with recommenders, their readiness to adopt the technology, and their intention to purchase recommended items (reverse scale). and return to the site in the future A.1.7 Context Compatibility After ResQue was finalized, we asked several expert researchers I was only provided with general recommendations in the community of recommender systems to review the model Their feedback and comments were then incorporated into the The items recommended to me took my personal context final version of the model. This method, known as the Delphi The recommendations are timely the work was submitted, we have started conducting a survey to further validate the models reliability, validity and sensitivity A2 Interaction Adequacy using factor analysis, structural equation modeling (SEM), and The recommender provides an adequate way for me to express other techniques described in [21]- Initial results based on 150 my preferences. participants indicate how the model can be interpreted and show The recommender provides an adequate way for me to revise factors that correspond to the original model. At the same time, my preferences. analysis also gives some indications on how to refine the model The recommender explains why the products are More users are expected to participate in the survey and the final outcome will be soon reported APPENDIX A3 Interface Adequacy A. Constructs and Questions of ResQue The recommenders interface provides sufficient information. The following contains the questionnaire statements that can be The information provided for the recommended items is used in a survey. They are developed based on the resQue model sufficient for me described in this paper. Users should be asked to indicate their The labels of the recommender interface are clear and answers to each of the questions using the 1-5 Likert scales, where. The layout of the recommender interface is attractive and I indicates"strongly disagree"and 5 is"strongly agree adequate.* Al. Quality of Recommended Items A II Accuracy A4 Pereeived Ease of Use The items recommended to me matched my interests. A.4.1 Ease of Initial Learning Copynight e 2010 for the individual papers by the papers authors. Copying permitted only for private and academic purposes his volume is published and copyrighted by its editors: Kniinenburg, B P, Schmidt- Thieme, L- Bollen. DOur overall motivation for this research was to understand the crucial factors that influence the user adoption of recommenders. Another motivation is to come up with a subjective evaluation questionnaire that other researchers and practitioners can employ. However, it is unlikely that a 60-item questionnaire can be administered for a quick and easy evaluation. This has motivated us in proposing a simplified model based on our past research. Between 2005 and 2010, we have administered 11 subjective questionnaires on a total of 807 subjects [4,5,6,12,13,14,23,24]. Initial questionnaires covered some of the four categories identified in the ResQue. As we conducted more experiments, we became more convinced of the four categories and used all of them in recent studies. On average, between 12 and 15 questions were used. Based this previous work, we have synthesized and organized a total of 15 questions as a simplified model for the purpose of performing a quick and easy usability and adoption evaluation of a recommender (see questions with * sign). 5. CONCLUSION AND FUTURE WORK User evaluation of recommender systems is a crucial subject of study that requires a deep understanding, development and testing of the right dimensions (or constructs) and the standardization of the questions used. The framework described in this paper presents the first attempt to develop a complete and balanced evaluation framework that measures users’ subjective attitudes based on their experience towards a recommender. ResQue consists of a set of 13 constructs and 60 questions for a high-quality recommender system from the user point of view and can be used as a standard guideline for a user evaluation. It can also be adapted to a custom-made user evaluation by tailoring it in an individual research context. Researchers and practitioners can use these questionnaires with ease to measure users’ general satisfaction with recommenders, their readiness to adopt the technology, and their intention to purchase recommended items and return to the site in the future. After ResQue was finalized, we asked several expert researchers in the community of recommender systems to review the model. Their feedback and comments were then incorporated into the final version of the model. This method, known as the Delphi method, is one of the first validation attempts on the model. Since the work was submitted, we have started conducting a survey to further validate the model’s reliability, validity and sensitivity using factor analysis, structural equation modeling (SEM), and other techniques described in [21]. Initial results based on 150 participants indicate how the model can be interpreted and show factors that correspond to the original model. At the same time, analysis also gives some indications on how to refine the model. More users are expected to participate in the survey and the final outcome will be soon reported. APPENDIX A. Constructs and Questions of ResQue The following contains the questionnaire statements that can be used in a survey. They are developed based on the ResQue model described in this paper. Users should be asked to indicate their answers to each of the questions using the 1-5 Likert scales, where 1 indicates “strongly disagree” and 5 is “strongly agree.” A1. Quality of Recommended Items A.1.1 Accuracy  The items recommended to me matched my interests.*  The recommender gave me good suggestions.  I am not interested in the items recommended to me (reverse scale). A.1.2 Relative Accuracy  The recommendation I received better fits my interests than what I may receive from a friend.  A recommendation from my friends better suits my interests than the recommendation from this system (reverse scale). A.1.3 Familiarity  Some of the recommended items are familiar to me.  I am not familiar with the items that were recommended to me (reverse scale). A.1.4 Attractiveness  The items recommended to me are attractive. A.1.5 Enjoyability  I enjoyed the items recommended to me. A.1.6 Novelty  The items recommended to me are novel and interesting.*  The recommender system is educational.  The recommender system helps me discover new products.  I could not find new items through the recommender (reverse scale). A.1.6 Diversity  The items recommended to me are diverse.*  The items recommended to me are similar to each other (reverse scale).* A.1.7 Context Compatibility  I was only provided with general recommendations.  The items recommended to me took my personal context requirements into consideration.  The recommendations are timely. A2. Interaction Adequacy  The recommender provides an adequate way for me to express my preferences.  The recommender provides an adequate way for me to revise my preferences.  The recommender explains why the products are recommended to me.* A3. Interface Adequacy  The recommender’s interface provides sufficient information.  The information provided for the recommended items is sufficient for me.  The labels of the recommender interface are clear and adequate.  The layout of the recommender interface is attractive and adequate.* A4. Perceived Ease of Use A.4.1 Ease of Initial Learning 19 FULL PAPER Proceedings of the ACM RecSys 2010 Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI), Barcelona, Spain, Sep 30, 2010 Published by CEUR-WS.org, ISSN 1613-0073, online ceur-ws.org/Vol-612/paper3.pdf Copyright © 2010 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors: Knijnenburg, B.P., Schmidt-Thieme, L., Bollen, D