eeper rating scale Amazon rating sca Your Rating Rated Don't like it s I love it! CCCC C Figure 3: Input Rating Scales for Sleeper Amazon(Music) How many Items to Rate? found themselves stumped. With only one opportunity to A few systems ask the user to enter only I piece of provide input to the system, they felt pressure to choose with nformation to receive recommendations while others extreme caution.(b)Ratings on Likert Scale: Users were require a minimum commitment of at least 30 ratings. Our asked to rate items on 5-10 point scale ranging from Like to quantitative and qualitative results indicate that users do not Dislike. This could become repetitive and boring. At mind giving a little more input to the system in order to Song Explorer, Movie Critic, and Rating Zone users expressed receive more accurate suggestions. Across both of c irritation at having to page through lists of items in order to studies 39% of the users felt that the input required by provide the requisite number of ratings. Another systems was not enough, in contrast to only 9.4% of our manifestation of a Likert scale was a continuous rating bar users who thought that the input required was too much. ranging from Like to Dislike. Users liked the rating bar since Table I shows users' opinions regarding the amount of input they could click anywhere to indicate degree of liking for an for music recommender systems(Study 2). Even for a item. The Sleeper system used such a scale(see Figure 3) system like MediaUnbound that required (c)Binary Liking For this type of question, users were questions, only 8% of users regarded this as too much. Users simply asked to check a box if they liked an item. This was indicated that their opinion of required input was influenced simple to do, but could become repetitive and boring as well by the kind of recommendations they received. For systems (d)Hybrid Rating Process: Such systems incorporated commendations were perceived as too simplistic features from all the above types of questions as appropriate (Amazon), or inaccurate(SongExplorer), most (50%)users MediaUnbound used such a process and also provided thought that input was not enoug continuous feedback to the user, keeping him /her engaged. No of Input How users felt about number of Another aspect of the input process was the set of items that Rating was rated. Often users had little or no experience of the item, 4-ad dust Right Foo Much leading to frustration with the rating process. one user commented at rating zone "I'm worried because I haven 3390.094 read many of these-I don 't know what I'm going to get 45% 0.0% back, while at Song Explorer, another user observed"The sonoexplorer 58% 39 Items to be rated] are all so obvious. I feel like I'm more 34 179 75% 8 39 sophisticated than the system is going to give me credit for Design Suggestion. It is important to design an easy and engaging process that keeps users from getting bored or Design Suggestion: Designers of recommender systems are frustrated. A mix of different types of questions, and often faced with a choice between enhancing ease of use(by continuous feedback during the input phase can help achieve asking users to rate fewer items)or enhancing the accuracy this goal of the algorithms(by asking users to provide more ratings) Our suggestion is that it is fine to ask the users for a few Filtering by Genre more ratings if that leads to substantial increases in Several recommender systems ask users whether they want accuracy. Users dislike bad recommendations more than they dislike providing a few additional ratings. Movie Critic allows users to set a variety of genre filters Without being asked, almost all of the users volunteered What kind of rating process? favorable comments on these filtersthey liked being able In the systems we studied, there were four types of rating to quickly set the "include and"exclude options on a list input formats: (a)Open-ended Name an artist/ writer you of about 20 genres. However, we discovered two possible ike. When asked to name one "favorite" artist, some users oblems with genre filtering. Several users commented thatHow many Items to Rate? A few systems ask the user to enter only 1 piece of information to receive recommendations, while others require a minimum commitment of at least 30 ratings. Our quantitative and qualitative results indicate that users do not mind giving a little more input to the system in order to receive more accurate suggestions. Across both of our studies 39% of the users felt that the input required by systems was not enough, in contrast to only 9.4 % of our users who thought that the input required was too much. Table 1 shows users’ opinions regarding the amount of input for music recommender systems (Study 2). Even for a system like MediaUnbound that required answers to 34 questions, only 8% of users regarded this as too much. Users indicated that their opinion of required input was influenced by the kind of recommendations they received. For systems whose recommendations were perceived as too simplistic (Amazon), or inaccurate (SongExplorer), most (>50%) users thought that input was not enough. Design Suggestion: Designers of recommender systems are often faced with a choice between enhancing ease of use (by asking users to rate fewer items) or enhancing the accuracy of the algorithms (by asking users to provide more ratings). Our suggestion is that it is fine to ask the users for a few more ratings if that leads to substantial increases in accuracy. Users dislike bad recommendations more than they dislike providing a few additional ratings. What kind of rating process? In the systems we studied, there were four types of rating input formats: (a) Open-ended: Name an artist / writer you like. When asked to name one “favorite” artist, some users found themselves stumped. With only one opportunity to provide input to the system, they felt pressure to choose with extreme caution. (b) Ratings on Likert Scale: Users were asked to rate items on 5-10 point scale ranging from Like to Dislike. This could become repetitive and boring. At SongExplorer, MovieCritic, and RatingZone users expressed irritation at having to page through lists of items in order to provide the requisite number of ratings. Another manifestation of a Likert scale was a continuous rating bar ranging from Like to Dislike. Users liked the rating bar since they could click anywhere to indicate degree of liking for an item. The Sleeper system used such a scale (see Figure 3). (c) Binary Liking: For this type of question, users were simply asked to check a box if they liked an item. This was simple to do, but could become repetitive and boring as well. (d) Hybrid Rating Process: Such systems incorporated features from all the above types of questions as appropriate. MediaUnbound used such a process and also provided continuous feedback to the user, keeping him / her engaged. Another aspect of the input process was the set of items that was rated. Often users had little or no experience of the item, leading to frustration with the rating process. One user commented at RatingZone “I’m worried because I haven’t read many of these—I don’t know what I’m going to get back,” while at SongExplorer, another user observed “The [items to be rated] are all so obvious. I feel like I’m more sophisticated than the system is going to give me credit for.” Design Suggestion: It is important to design an easy and engaging process that keeps users from getting bored or frustrated. A mix of different types of questions, and continuous feedback during the input phase can help achieve this goal. Filtering by Genre Several recommender systems ask users whether they want recommendations from a particular genre. For example, MovieCritic allows users to set a variety of genre filters. Without being asked, almost all of the users volunteered favorable comments on these filters—they liked being able to quickly set the “include” and “exclude” options on a list of about 20 genres. However, we discovered two possible problems with genre filtering. Several users commented that No. of Input Ratings System Not Enough Just Right Too Much Amazon 4-20 67% 33% 0.0% CDNow 3 67% 33% 0.0% MoodLogic ~4 45% 55% 0.0% SongExplorer 20 58% 25% 8.3% MediaUnbound 34 17% 75% 8.3% Table 1: Input Ratings (From Study 2) How users felt about number of ratings Sleeper Rating Scale Amazon Rating Scale Figure 3: Input Rating Scales for Sleeper & Amazon (Music)