3.3 Additional Ranking approaches We here introduce four additional ranking approaches that can be used as alternatives to ranksrandard to aprove recommendation diversity, and the formal definitions of each ranking approach as well as standard and item popularity-based ranking approaches are provided in Figure 2. As seen from the empirical analysis on the positive relationships between average item popularity and predicted ratings in ( Adomavicius and Kwon 2008), we also consistently observed that popular items, on average, are likel to have higher predicted ratings than less popular items, using different traditional recommendation techniques. Therefore, it can be suggested that recommending not as highly predicted items(but still predicted to be above Th) likely implies recommending, on average, less popular items, potentially leading to diversity improvements; following this observation, we propose to use predicted rating value itself as an item ranking criterion. Based on similar empirical observations, we propose a number of alternative ranking approaches, including the ones based on average rating, absolute likeability, and relative likeability, as defined in Figure 2 4. Empirical Results The proinetflixprize. com) data sets. We pre-processed each dataset to include users and movies with The proposed recommendation ranking approaches were tested with MovieLens (grouplens. org) and significant rating history, which makes it possible to have sufficient number of highly-predicted items for recommendations to each user(in the test data). For each dataset, we randomly chose 60% of the ratings as training data and used them to predict the remaining 40%(i.e, test data 4.1 Performance of Proposed ranking Approaches All five proposed ranking approaches were used in conjunction with one of three widely popular recommendation techniques for rating prediction, including two heuristic-based (user-based and item- based "nearest neighbor" )and one model-based(matrix factorization) collaborative filtering(CF) techniques(Sarwar et al. 2001, Funk 2006), to generate top-N(N=1, 5, 10)recommendations to each user. We set predicted rating threshold as TH=3.5 (out of 5)to ensure that only relevant items are recommended to users, and ranking threshold TR varied from 3.5 to 5.0. The performance of each ranking approach was measured in terms of precision-in-top-N and diversity-in-top-N and, for comparison purposes, its diversity gain and precision loss with respect to the standard ranking approach was calculated. Consistently with the accuracy-diversity tradeoff discussed in the introduction, all the proposed ranking approaches improved the diversity of recommendations by sacrificing the accuracy However, with each ranking approach, as ranking threshold TR increases, the accuracy loss is significantly minimized(smaller precision loss)while still exhibiting substantial diversity improvement. Therefore, with different ranking thresholds, one can obtain different diversity gains for different levels of tolerable precision loss, as compared to the standard ranking approach. Following this idea, in our experiments compare the effectiveness (i.e, diversity gain) of different ranking techniques for various precision lo levels(0.1-10%). Standard, i.e., ranking the candidate(highly. Item Average Rating, i.e., ranking by an average of predicted) items by their predicted rating value. known ratings for each item from highest to lowest rankStandard(O=R(u, i) rankAvgRating(=R(D=>R(u,D)/U() Item Popularity, i.e, ranking by item popularity, Item Ab solute Lik ability, i.e, ranking by how from lowest to highest, where popularity is many users liked the item(i.e, rated it above TH) each item has rankitemPop(D)=U(D, U()=uEUT3R(u, i)) Item Rela tive Lik ability, i.e., ranking by the Reverse Pred icted Rating Value, i.e, ranking by percentage of the users who liked an item(among predicted rating value, from lowest to highest all users who rated it) rankRevPred(i=R (u, i) rankRelLike(D)=JUHDI/JU(Dl Figure 2. Various ranking functions 19th Workshop on Information Technologies and Systems3.3 Additional Ranking Approaches We here introduce four additional ranking approaches that can be used as alternatives to rankStandard to improve recommendation diversity, and the formal definitions of each ranking approach as well as standard and item popularity-based ranking approaches are provided in Figure 2. As seen from the empirical analysis on the positive relationships between average item popularity and predicted ratings in (Adomavicius and Kwon 2008), we also consistently observed that popular items, on average, are likely to have higher predicted ratings than less popular items, using different traditional recommendation techniques. Therefore, it can be suggested that recommending not as highly predicted items (but still predicted to be above TH) likely implies recommending, on average, less popular items, potentially leading to diversity improvements; following this observation, we propose to use predicted rating value itself as an item ranking criterion. Based on similar empirical observations, we propose a number of alternative ranking approaches, including the ones based on average rating , absolute likeability , and relative likeability, as defined in Figure 2. 4. Empirical Results The proposed recommendation ranking approaches were tested with MovieLens (grouplens.org) and Netflix (netflixprize.com) data sets. We pre-processed each dataset to include users and movies with significant rating history, which makes it possible to have sufficient number of highly-predicted items for recommendations to each user (in the test data). For each dataset, we randomly chose 60% of the ratings as training data and used them to predict the remaining 40% (i.e., test data). 4.1 Performance of Proposed Ranking Approaches All five proposed ranking approaches were used in conjunction with one of three widely popular recommendation techniques for rating prediction, including two heuristic-based (user-based and item￾based “nearest neighbor”) and one model-based (matrix factorization) collaborative filtering (CF) techniques (Sarwar et al. 2001, Funk 2006), to generate top-N (N=1, 5, 10) recommendations to each user. We set predicted rating threshold as TH = 3.5 (out of 5) to ensure that only relevant items are recommended to users, and ranking threshold TR varied from 3.5 to 5.0. The performance of each ranking approach was measured in terms of precision-in-top-N and diversity-in-top-N and, for comparison purposes, its diversity gain and precision loss with respect to the standard ranking approach was calculated. Consistently with the accuracy-diversity tradeoff discussed in the introduction, all the proposed ranking approaches improved the diversity of recommendations by sacrificing the accuracy. However, with each ranking approach, as ranking threshold TR increases, the accuracy loss is significantly minimized (smaller precision loss) while still exhibiting substantial diversity improvement. Therefore, with different ranking thresholds, one can obtain different diversity gains for different levels of tolerable precision loss, as compared to the standard ranking approach. Following this idea, in our experiments we compare the effectiveness (i.e., diversity gain) of different ranking techniques for various precision loss levels (0.1-10%). • Standard, i.e., ranking the candidate (highly predicted) items by their predicted rating value, from highest to lowest. rankStandard(i)=R* (u, i) -1. • Item Popul arity, i.e., ranking by item popularity, from lowest to highest, where popularity is represented by the number of known ratings that each item has . rankItemPop(i)= |U(i)|, U(i) = {u∈U | ∃R(u, i)}. • Reverse Pred icted Ra ting Va lue, i.e., ranking by predicted rating value, from lowest to highest. rankRevPred(i) = R*(u,i). • Item Average Rating, i.e., ranking by an average of all known ratings for each item: rankAvgRating(i) = |)(| ),()( )( iUiuRiR iUu ∑∈ = . • Item Ab solute Lik eability, i.e., ranking by how many users liked the item (i.e., rated it above TH): rankAbsLike(i) = |UH(i)|, UH(i)={u∈U(i)| R(u,i) ≥ TH}. • Item Rela tive Lik eability, i.e., ranking by the percentage of the users who liked an item (among all users who rated it). rankRelLike(i) = |UH(i)| / |U(i)|. Figure 2. Various ranking functions 82 19th Workshop on Information Technologies and Systems