J Bobadilla et al Expert Systems with Applications 38(2011)14609-14623 recall measure obtained by making the sameN recommendations Table 12 to user u Values of precision and recall using N=4. 0=4. Assuming that all the users accept N test recommendations h{i∈Zui∈Y} 33+1)1(1+1)4(4+1)3/3+0)3/3+0)x=0.81 #{i∈zu∈Y} #{i∈Zui∈Y}+#{i∈Zi∈Y} Number of items that both x and y have voted for (rx≠·Arya≠) in relation to the total number of items voted for by both ∑l In order to include time in our model, we extend formulas(4) and(5)to contain a time value in timestamp format. 2.10.2. Running example irstly, we find the number of votes received for each item. Ru={(i.U,D)i∈l,v∈v,t∈·l which we represent in the last row of Table 13: later we establish Tui =vALui=t a threshold of novelty(y=3). The set of items belonging to the We define Ex, y as the group of items that both x and y have voted novelty set is as follows: for most recently. Most recently means within a period of B days as Y={2.3.56,7,8.9,11,12,14 regards the current time(t) Table14 shows the recommendations made to each of the users Exy={i∈lx≠·Nry≠·^t ng N=4 and 0=4(Zu): the items belonging to y (in the first We define Su as the group of votes of user u which have beer ow)are framed. Table 15 shows the novelty-precision and nov- made in the time interval B as regards the current time Ity-recall results obtained by each of the users and the total nov- elty-precision and novelty-recall obtained in the example. S={(,v,D)∈l,v∈V,t∈·.t-tu≤B If the votes' time information is not available, Eq (51)can be 2.10.3. Case of study mplified in the following way Firstly, in order to be able to adjust parameter y to a suitable value in the rs used, it is valuable to know the distribution of Exy=Axy={i∈|xi≠·^yi≠·h the votes regarding the items. As an example, Fig 9 shows this data From the group of items defined in(51), or failing that, in(53). obtained in MovieLens 100 K. Thus, we can determine, for instance, we use the similarity measure which each user will intuitively use that 600 items of the rs have been voted for by 13 or less users. Figs.10 and 11, respectively, show the novelty-precision and to compare their votes with those of each of their neighbors: the novelty-recall results obtained using MovieLens 100 K with values Mean Absolute Difference(MaD) of 7: 13, 17, 21 and 25. a general increase in the precision may be MAD(x, y B)=MAD,X, B) ted as we take higher values of 7, due to the gradual increase that this implies in the number of relevant recommended elements. E∑-列E≠中 (54) 2.11. Quality of trust: trust-precision and trust-recall As a list of common votes among users, as regards the total, we use Jaccard 2.11.1. Formalization As follows from actual results obtained in an experiment carried Jaccard(x, y, B)=Jaccard(y, x, B) out on a group of users of the filmaffinity. com website, the trust of s, OS ser x towards another user y could be based on the following 3 Willing to obtain similar importance to metrics(54)and(55). aspects we place the MAD results on the scale [0. 1. where 1 represents the greatest possible similitude and 0 the least possible. We com- votes bine both metrics by multiplying them, so that when either of Greater importance to the last items voted. them is low the total similitude is highly affected. 11 nt recommended: values with diagonal lines, relevant not recommended: values with horizontal lines. 33 234 3.33 555445 444454 3recall measure obtained by making the sameN recommendations to user u. Assuming that all the users accept N test recommendations: nu ¼ #fi 2 Zuji 2 Yg N ð45Þ lu ¼ #fi 2 Zuji 2 Yg #fi 2 Zuji 2 Yg þ #fi 2 Zc uji 2 Yg ð46Þ n ¼ 1 #U X u2U nu ð47Þ l ¼ 1 #U X u2U lu ð48Þ 2.10.2. Running example Firstly, we find the number of votes received for each item, which we represent in the last row of Table 13; later we establish a threshold of novelty (c = 3). The set of items belonging to the novelty set is as follows: Y ¼ f2; 3; 5; 6; 7; 8; 9; 11; 12; 14g Table 14 shows the recommendations made to each of the users using N = 4 and h = 4(Zu); the items belonging to Y (in the first row) are framed. Table 15 shows the novelty-precision and nov￾elty-recall results obtained by each of the users and the total nov￾elty-precision and novelty-recall obtained in the example. 2.10.3. Case of study Firstly, in order to be able to adjust parameter c to a suitable value in the RS used, it is valuable to know the distribution of the votes regarding the items. As an example, Fig. 9 shows this data obtained in MovieLens 100 K. Thus, we can determine, for instance, that 600 items of the RS have been voted for by 13 or less users. Figs. 10 and 11, respectively, show the novelty-precision and novelty-recall results obtained using MovieLens 100 K with values of c:13, 17, 21 and 25. A general increase in the precision may be noted as we take higher values of c, due to the gradual increase that this implies in the number of relevant recommended elements. 2.11. Quality of trust: trust-precision and trust-recall 2.11.1. Formalization As follows from actual results obtained in an experiment carried out on a group of users of the filmaffinity.com website, the trust of user x towards another user y could be based on the following 3 aspects: Similarity in the votes. Greater importance to the last items voted. Number of items that both x and y have voted for (rx,i – ^ ry,i – ) in relation to the total number of items voted for by both. In order to include time in our model, we extend formulas (4) and (5) to contain a time value in timestamp format. Ru ¼ fði; v;tÞji 2 I; v 2 V;t 2 g ð49Þ ru;i ¼ v ^ tu;i ¼ t ð50Þ We define Ex,y as the group of items that both x and y have voted for most recently. Most recently means within a period of b days as regards the current time (tc) Ex;y ¼ fi 2 Ijrx;i – ^ry;i – ^tc tx;i 6 b ^ tc ty;i 6 bg ð51Þ We define Su as the group of votes of user u which have been made in the time interval b as regards the current time. Su ¼ fði; v;tÞji 2 I; v 2 V;t 2 ;tc tu;i 6 bg ð52Þ If the votes’ time information is not available, Eq. (51) can be simplified in the following way: Ex;y ¼ Ax;y ¼ fi 2 Ijrx;i – ^ry;i – g ð53Þ From the group of items defined in (51), or failing that, in (53), we use the similarity measure which each user will intuitively use to compare their votes with those of each of their neighbors: the Mean Absolute Difference (MAD): MADðx; y; bÞ ¼ MADðy; x; bÞ ¼ 1 #Ex;y X i2Ex;y jrx;i ry;ij () Ex;y – / ð54Þ As a list of common votes among users, as regards the total, we use Jaccard: Jaccardðx; y; bÞ ¼ Jaccardðy; x; bÞ ¼ Sx \ Sy Sx [ Sy ð55Þ Willing to obtain similar importance to metrics (54) and (55), we place the MAD results on the scale [0, 1], where 1 represents the greatest possible similitude and 0 the least possible. We com￾bine both metrics by multiplying them, so that when either of them is low the total similitude is highly affected. Table 11 Relevant recommended: values with diagonal lines; relevant not recommended: values with horizontal lines. I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 U1 r1,i 5 3 4 1 4 2 4 eZ1 4.5 3.5 4.33 4.5 U2 r2,i 1 2 41 4 1 eZ2 4.5 3 4 4.5 U3 r3,i 5 2 4 35 4 4 eZ3 3.33 5 4 4 U4 r4,i 4 3 5 4 eZ4 5 3.5 4.5 4.33 U5 r5,i 334 5 5 eZ5 35 4 4 Table 12 Values of precision and recall using N = 4, h = 4. U1 U2 U3 U4 U5 Average tu 3/4 1/4 4/4 3/4 3/4 t = 0.70 xu 3/(3 + 1) 1/(1 + 1) 4/(4 + 1) 3/(3 + 0) 3/(3 + 0) x = 0.81 J. Bobadilla et al. / Expert Systems with Applications 38 (2011) 14609–14623 14617