Real Tag TF TFIDF LDA Tag LDA Prob. AR Tag AR Conf. 0.0906 LCLO.US web bookmarks.06950.1721delicious reference 0. 005460.1468 tools reference.0521 0.0407 0.0223 internet social 0509 language 0.642 folksonomy.044701306 bookmarks0166 delicio.t 004090.1166 0.0090 software 0.03970.0271 tool 0.0090 0.515 008002cice 0.0085 0.467 0.03470.0714 0.0065 interesting.417 0.02480.0722 dictionary0.004 0.398 0.02360.0770 bookmark information academic0.02230.0780 english 0.0049 search 0.393 search 0. 0.0402environment 0.0040 0.391 web 0.018 0.0084 0.0039 bookmarking 0717 astronomy 0.0037 0.0161 0.0496 marketing 0.0033 socialsoftware0.01490.0387 0.0033 internet 0012400124 0.0032 academia.01110.0780 startup 0.0031 collaboration.011100322 0.0028 Table 8: Actual tags with tag frequency and recommended tags with computed probablity for URL #BM # LDA topics I0U25050 two lists are compared with the list based on all original se second uses the tags recommended by our algorithm. Thes 0.3130.3100.2970.268 assigned to the test set. For the ranking of the results in each 210.3530.3600.3510.328 list, we implemented a simple baseline algorithm based on 0.3670.3810.37 eyword search. The resources are weighted according 4|0.3710.3920.397 0356 FIDF score of the query tag. E.g. a search for the 50.3780.4010.4140.403 "web" gives a list with resources annotated with the tag"web". The list is ranked according to tag frequency Table 7: F-measure for different sized test set and i.e., how high is the number of"web"-tags compared to the different number of LDA topics(threshold 0.001) overall number of tags assigned to a resource The testset without recommended tags is also ranked by TFIDF, whereas the testset with recommended tags is ranked Table 8 shows the actual tag distribution for a randomly by the probability assigned by LDa. To compare the three selectedresource(http://www.connotea.org),thetoptags anked lists. we need to first decide which of the baseline recommended by LDa with aggregated probabilities, and results are considered relevant. We report scores for taking (all) the tags recommended by association rules based on the top 10 and the top 20 resources as relevant results. A five known bookmarks. The tags available in the known well known measure for comparing rankings in information bookmarks( first column)and the correctly recommended retrieval is Mean Average precision(MAP)(22), computed tags( forth and sixth column)are marked in bold. As the as follows actual tags indicate, Connotea is a tagging site focusing on scientists and scientific resources. The tags recommended by LDa come from five latent topics, comprising social sys- tems, tagging, science, business, and language. These tags MAP(Q)=∑m∑Pn( characterize Connotea quite well, and accordingly among the nine most likely recommended tags, there is only one with Rik the set of ranked results from the top of the list rather general tag(business) that is not among the actual down to item k in the list. where the set of relevant items is tags. In contrast, the tags recommended by association rules ii.im, If no relevant document is retrieved, precisio hardly characterize the site, but are rather non descriptive is taken to be O Table 9 shows the map values based on the number of known bookmarks. When considering the top 10 TFIDF 3.2.3 Tag search ranked results levant. extension of the resources with To evaluate the effectiveness of our recommended tags for our recommended tags increases MAP by more than 300% tag search we compared three result lists: The first is based for one known bookmark. When considering the top 20 re- on the testset with only 1-5 bookmarks per resource, the sults of the baseline algorithm as relevant, the MAP score for the Lda probabilities weighted ranked list increases by SThe first five bookmarks contain three more tags with more then 400% in the one bookmark case. rather low TF: webware. management and social software.Real Tag TF TFIDF LDA Tag LDA Prob. AR Tag AR Conf. science 0.0906 0.2281 del.icio.us 0.1001 web 0.912 bookmarks 0.0695 0.1721 delicious 0.0478 reference 0.760 tags 0.0546 0.1468 tools 0.0356 tools 0.664 reference 0.0521 0.0407 business 0.0223 internet 0.657 social 0.0509 0.1068 language 0.0204 cool 0.642 folksonomy 0.0447 0.1306 bookmarks 0.0166 tech 0.585 del.icio.us 0.0409 0.1166 web 0.0090 software 0.541 tools 0.0397 0.0271 tool 0.0090 toread 0.515 tagging 0.0360 0.1062 science 0.0085 technology 0.467 research 0.0347 0.0714 space 0.0065 interesting 0.417 delicious 0.0248 0.0722 dictionary 0.0064 design 0.398 bookmark 0.0236 0.0770 bookmark 0.0059 information 0.395 academic 0.0223 0.0780 english 0.0049 search 0.393 search 0.0223 0.0402 environment 0.0040 blog 0.391 web 0.0186 0.0084 reference 0.0039 – – bookmarking 0.0173 0.0717 astronomy 0.0037 – – tag 0.0161 0.0496 marketing 0.0033 – – socialsoftware 0.0149 0.0387 tags 0.0033 – – internet 0.0124 0.0124 cool 0.0032 – – academia 0.0111 0.0780 startup 0.0031 – – collaboration 0.0111 0.0322 words 0.0028 – – Table 8: Actual tags with tag frequency and recommended tags with computed probablity for URL www.connotea.org #BM # LDA topics 50 100 250 500 1 0.313 0.310 0.297 0.268 2 0.353 0.360 0.351 0.328 3 0.367 0.381 0.378 0.356 4 0.371 0.392 0.397 0.386 5 0.378 0.401 0.414 0.403 Table 7: F-measure for different sized test set and different number of LDA topics (threshold 0.001) Table 8 shows the actual tag distribution for a randomly selected resource (http://www.connotea.org), the top tags recommended by LDA with aggregated probabilities, and (all) the tags recommended by association rules based on five known bookmarks. The tags available in the known bookmarks (first column)5 and the correctly recommended tags (forth and sixth column) are marked in bold. As the actual tags indicate, Connotea is a tagging site focusing on scientists and scientific resources. The tags recommended by LDA come from five latent topics, comprising social sys￾tems, tagging, science, business, and language. These tags characterize Connotea quite well, and accordingly among the nine most likely recommended tags, there is only one rather general tag (business) that is not among the actual tags. In contrast, the tags recommended by association rules hardly characterize the site, but are rather non descriptive and general. 3.2.3 Tag Search To evaluate the effectiveness of our recommended tags for tag search we compared three result lists: The first is based on the testset with only 1 – 5 bookmarks per resource, the 5The first five bookmarks contain three more tags with rather low TF: webware, management, and social software. second uses the tags recommended by our algorithm. These two lists are compared with the list based on all original tags assigned to the test set. For the ranking of the results in each list, we implemented a simple baseline algorithm based on single keyword search. The resources are weighted according to the TFIDF score of the query tag. E.g. a search for the keyword “web” gives a list with resources annotated with the tag “web”. The list is ranked according to tag frequency, i.e., how high is the number of “web”-tags compared to the overall number of tags assigned to a resource. The testset without recommended tags is also ranked by TFIDF, whereas the testset with recommended tags is ranked by the probability assigned by LDA. To compare the three ranked lists, we need to first decide which of the baseline results are considered relevant. We report scores for taking the top 10 and the top 20 resources as relevant results. A well known measure for comparing rankings in information retrieval is Mean Average precision (MAP) [22], computed as follows: MAP(Q) = 1 |Q| X |Q| j=1 1 mj Xmj k=1 Precision(Rjk) (5) with Rjk the set of ranked results from the top of the list down to item k in the list, where the set of relevant items is {i1 . . . imj }. If no relevant document is retrieved, precision is taken to be 0. Table 9 shows the MAP values based on the number of known bookmarks. When considering the top 10 TFIDF ranked results as relevant, extension of the resources with our recommended tags increases MAP by more than 300% for one known bookmark. When considering the top 20 re￾sults of the baseline algorithm as relevant, the MAP score for the LDA probabilities weighted ranked list increases by more then 400% in the one bookmark case