behaviour data has been accumulated. The quicker the profiles tended to miss current interests. This is because publications are move to this state the quicker they will have overcome the cold. generally not available for up-to-date interests As we would expect, once the weekly behaviour logs become Week 7 was chosen as the cut-off point of our analysis since after available to the system the profiles adjust accordingly, moving about 7 weeks of use the behaviour data gathered by Quickstep away from the initial bootstrapping. On week 7 the profiles ll dominate the user profiles. The effects of bootstrapping yond this point would not be significant. If we were to run the system beyond week 7 we would simply see the profiles The new-user algorithm result show a more dramatic increase in precision to 0.84, but comes at the price of a significant error rate continually adjusting to the behaviour logged each week of 0.55. The profiles produced by the new-user algorithm tended 8.2 Experimental results to be very inclusive, taking the set of similar user interests and Two measurements were preformed when comparing profiles to producing a union of these interests. While this captures many of the benchmark week 7 profile. The first, profile precision, interests not relevant to the new user but which were interesting to measures how many topics were mentioned in both the current the people similar to the new user. profile and benchmark profile. Profile precision is an indication how quickly the profile is converging to the final state, and thus Profile precision relative to benchmark profile how quickly the effects of the cold-start are overcome. The second, profile error rate, measures how many topics appeared in the current profile that did not appear within the benchmark 0.75 profile. Profile error rate is an indication of the errors introduced by the two bootstrapping algorithms. Figure 11 describes these metrIcs It should be noted that we are not measuring the precision and error rate of the profiles -only the relative and error rate compared to the week 7 steady state Measuring absolute profile accuracy subjective matter ve do not attempt it here, we ar interested in ho ly profiles reach their stead Figure 12. Profile precision evaluation of Quicksteps overall profiling and recommendatio performance can be found in [18]. Profile error rate relative to benchmark profile Noted+ N new-system bootstrap user bootstrap profile error rate of user topics that appear in current and benchmark profile Week of user topics that appear in benchmark Figure 13. Profile error rate profile but not in current profile Nineorreet Number of user topics that appear in current profile but not in benchmark profile Since error rate is measured relative to the final benchmark profile Total number of users of week 7, all the topics seen in the behaviour logs will be present Figure 11. Evaluation metrics within the benchmark profile. Incorrect topics must thus come from another source- in this case bootstrapping on week 0. This causes error rates to be constant over the 7 weeks the The results of our experimental runs are detailed in figures 12 and incorrect topics introduced on week 0 remain for all subsequent 13. The new-user results consist of a single iteration, so appear on weeks the graphs as a single point 9. DISCUSSION At the start, week 0, no browsing behaviour log data is available Cold-starts in recommender systems and interest acquisition in to the system so the profiles without bootstrapping are empty. The ontologies are serious problems. If initial recommendations are new-system algorithm, however, can bootstrap the initial us inaccurate, user confidence in the recommender system may drop profiles and achieves a reasonable precision of 0.35 and a low with the result that not enough usage data is gathered to overcome error rate of 0.06. We found that the new-system profiles the cold-start. In regards to ontologies, up-to-date interests are not accurately captured interests users had a year or so ago, butbehaviour data has been accumulated. The quicker the profiles move to this state the quicker they will have overcome the cold￾start. Week 7 was chosen as the cut-off point of our analysis since after about 7 weeks of use the behaviour data gathered by Quickstep will dominate the user profiles. The effects of bootstrapping beyond this point would not be significant. If we were to run the system beyond week 7 we would simply see the profiles continually adjusting to the behaviour logged each week. 8.2 Experimental results Two measurements were preformed when comparing profiles to the benchmark week 7 profile. The first, profile precision, measures how many topics were mentioned in both the current profile and benchmark profile. Profile precision is an indication of how quickly the profile is converging to the final state, and thus how quickly the effects of the cold-start are overcome. The second, profile error rate, measures how many topics appeared in the current profile that did not appear within the benchmark profile. Profile error rate is an indication of the errors introduced by the two bootstrapping algorithms. Figure 11 describes these metrics. It should be noted that we are not measuring the absolute precision and error rate of the profiles – only the relative precision and error rate compared to the week 7 steady state profiles. Measuring absolute profile accuracy is a very subjective matter, and we do not attempt it here; we are only interested in how quickly profiles reach their steady states. A more complete evaluation of Quickstep’s overall profiling and recommendation performance can be found in [18]. Ncorrect Number of user topics that appear in current profile and benchmark profile Nmissing Number of user topics that appear in benchmark profile but not in current profile Nincorrect Number of user topics that appear in current profile but not in benchmark profile Nusers Total number of users profile error rate = ______________________ Ncorrect + Nincorrect + Nmissing Nincorrect profile precision = Ncorrect + Nmissing Ncorrect _____________ Nusers 1 _____ ˇ 1.. Nusers user Nusers 1 _____ ˇ 1.. Nusers user Ncorrect Number of user topics that appear in current profile and benchmark profile Nmissing Number of user topics that appear in benchmark profile but not in current profile Nincorrect Number of user topics that appear in current profile but not in benchmark profile Nusers Total number of users profile error rate = ______________________ Ncorrect + Nincorrect + Nmissing Nincorrect ______________________ Ncorrect + Nincorrect + Nmissing Nincorrect profile precision = Ncorrect + Nmissing Ncorrect _____________ Ncorrect + Nmissing Ncorrect Ncorrect + Nmissing Ncorrect _____________ Nusers 1 _____ ˇ 1.. Nusers user Nusers 1 _____ ˇ 1.. Nusers user Figure 11. Evaluation metrics The results of our experimental runs are detailed in figures 12 and 13. The new-user results consist of a single iteration, so appear on the graphs as a single point. At the start, week 0, no browsing behaviour log data is available to the system so the profiles without bootstrapping are empty. The new-system algorithm, however, can bootstrap the initial user profiles and achieves a reasonable precision of 0.35 and a low error rate of 0.06. We found that the new-system profiles accurately captured interests users had a year or so ago, but tended to miss current interests. This is because publications are generally not available for up-to-date interests. As we would expect, once the weekly behaviour logs become available to the system the profiles adjust accordingly, moving away from the initial bootstrapping. On week 7 the profiles converge to the benchmark profile. The new-user algorithm result show a more dramatic increase in precision to 0.84, but comes at the price of a significant error rate of 0.55. The profiles produced by the new-user algorithm tended to be very inclusive, taking the set of similar user interests and producing a union of these interests. While this captures many of the new users real interests, it also included a large number of interests not relevant to the new user but which were interesting to the people similar to the new user. Profile precision relative to benchmark profile 0 0.25 0.5 0.75 1 01234567 Week Precision new-system bootstrap no bootstrap new-user bootstrap Figure 12. Profile precision Profile error rate relative to benchmark profile 0 0.25 0.5 0.75 1 01234567 Week Error rate new-system bootstrap no bootstrap new-user bootstrap Figure 13. Profile error rate Since error rate is measured relative to the final benchmark profile of week 7, all the topics seen in the behaviour logs will be present within the benchmark profile. Incorrect topics must thus come from another source – in this case bootstrapping on week 0. This causes error rates to be constant over the 7 weeks, since the incorrect topics introduced on week 0 remain for all subsequent weeks. 9. DISCUSSION Cold-starts in recommender systems and interest acquisition in ontologies are serious problems. If initial recommendations are inaccurate, user confidence in the recommender system may drop with the result that not enough usage data is gathered to overcome the cold-start. In regards to ontologies, up-to-date interests are not