Finding Similar Posts. Auto Tag uses Information Retrieval only the first 10 results are checked because it is assumed measures to estimate the similarity between weblog posts that users are unwilling to examine long result lists. For the n practice, this means a large collection of posts is indexed automated evaluation, we measured recall at 10 as well: the by an IR engine, and a query generated from the original fraction of tags offered by Auto Tag in the top-10 suggestions post is submitted to this engine. The most similar posts are which were also assigned by the blogger out of the total hen taken to be the highest-ranking ones retrieved from the collection, using some retrieval model We experimented with a number of methods for generat. t Evaluation Precision @10 Recall@1o ing queries from a post, including using the entire text of 0.59 the post and using links in it to locate cocitations. The best Manual osts Automated 0.40 results were obtained by using a" distinctive term"query standard corpus comparison methods are first used to de rive the most distinctive terms in the vocabulary of a post Table 1: Auto-tagging accuracy compared to the general vocabulary in the corpus:the Results are shown in Table 1; an example of tags offered by top-ranking terms are in turn used as the query. Auto Tag is given in Table 2. On average, 4 to 6 suggestion out of AutoTag's top-10 suggestions are either considered A Tag Model. Auto Tag uses simple heuristics for compos- useful by the blogger, or were actually used by her for the ing the ranked list of tags from the top-retrieved posts: each given post. Cursory examination of posts for which Auto Tag tag is scored according to its frequency in the top results. scores low shows many non-English posts(for whic Experimenting with more complex ways of scoring the tags, less data exists in the corpus, entailing lower success taking into account the retrieval rank or score, yielded only driven methods), and many tags which are highly minor improvements in accuracy. Filtering and Reranking. One clear source of informa- and used by few bloggers(such as names of family members) ion we have about a blogger is the tags she already used http://www prior to writing the post being analyzed. Therefore, if such previously-used tags appear in the ranked list, AutoTag boosts their score by a constant factor. nments on how to pitch a product to him as a blogger, and suggestions as Lindsay agrees getting distracted 3.EⅤ ALUATION how this For evaluating our method, we used the corpus distributed by Intelliseek for the 3rd Weblogging Workshop. The cor- pus contains 10M weblog posts collected during a period of 3 weeks; of these, 1.8M posts are tagged, with a total of 2. 3M uggested tags: PR, blogging, weblogs, marketing, net, gripes, tags. For indexing and retrieval, we used the open source nail, small business life, Anil Dash, PR pitching ngine Lucene which uses a rather simple vector space re- trieval model; text was stemmed with an English stemmer Two methods were used to evaluate the effectiveness of the Table 2: Sample tags suggested for a post tag suggestion methods. First, we manually examined the 4. CONCLUSIONS lection; for each tag, we decided whether the tag was indee e proposed and evaluated AutoTag, a tool for tagging a relevant label for the post. This is the preferred method weblog posts based on a collaborative filtering approach. of evaluation, but due to its cost it can only be applied to Auto Tag offers suggestions for tags based on tags assign to other, similar posts; the final decision about a tag is left a small number of posts; additionally, it is difficult to use to the blogger. Despite a relatively small corpus for this type non automated methods to tune and im a system. Be- of task, Auto Tag shows good results, and has the potential ause of this, we used an automated method to evaluate a much larger subset of posts: AutoTag was used to tag 6000 to benefit both the bloggers and others making use of tags f the " tagged posts"in our corpus- the posts which were assigned to weblog posts. assigned tags by their authors (we used only posts with 3 The different components of Auto Tag provide fertile ground or more tags ). Then, AutoTag's output was compared to for further work: identifying effective ways to generate queries he actual post tags. To account for minor differences in from a post and successful retrieval models to use; improving tags ("blogs"and"blogging"), we used string distance to the aggregation of tags from the retrieved posts; and vari- compare the tags rather than exact string match. Even so, by Auto Tag. In addition to the collaborative approaches since tags which are useful for a post but were not originally cal"approach to tag suggestion, in which suggestions for sed by its author are mistakenly taken to be incorrect. To demonstrate this. we evaluated the small test set with the tags are made without access to the entire blogosphere as automated method as well, resulting in substantially lower is the case with AutoTag, but using deeper analysis of the res: this indicates that the actual performance of Auto- contents of the post and the blog it belongs to. Tag on the large set is likely to be much better than reported 5. REFERENCES by the automated evaluation 1S. Golder and B. A H an. The structure of collaborative For the manual evaluation, we measured precision at 10 the fraction of tags out of the top-10 suggestions by AutoTag [2 A. Mathes Folksonomies: Cooperative classification and which were judged as appropriate for the post by a human Communication. 2004. [3] P Resnick and H. R. Varian. Recommender systems(spe http://www.blogpulse.com/www2006-workshop section). Comm. of the ACM, 40(3), 1997Finding Similar Posts. AutoTag uses Information Retrieval measures to estimate the similarity between weblog posts. In practice, this means a large collection of posts is indexed by an IR engine, and a query generated from the original post is submitted to this engine. The most similar posts are then taken to be the highest-ranking ones retrieved from the collection, using some retrieval model. We experimented with a number of methods for generat￾ing queries from a post, including using the entire text of the post and using links in it to locate cocitations. The best results were obtained by using a “distinctive term” query: standard corpus comparison methods are first used to de￾rive the most distinctive terms in the vocabulary of a post (compared to the general vocabulary in the corpus); the top-ranking terms are in turn used as the query. A Tag Model. AutoTag uses simple heuristics for compos￾ing the ranked list of tags from the top-retrieved posts: each tag is scored according to its frequency in the top results. Experimenting with more complex ways of scoring the tags, taking into account the retrieval rank or score, yielded only minor improvements in accuracy. Filtering and Reranking. One clear source of informa￾tion we have about a blogger is the tags she already used prior to writing the post being analyzed. Therefore, if such previously-used tags appear in the ranked list, AutoTag boosts their score by a constant factor. 3. EVALUATION For evaluating our method, we used the corpus distributed by Intelliseek for the 3rd Weblogging Workshop.1 The cor￾pus contains 10M weblog posts collected during a period of 3 weeks; of these, 1.8M posts are tagged, with a total of 2.3M tags. For indexing and retrieval, we used the open source engine Lucene which uses a rather simple vector space re￾trieval model; text was stemmed with an English stemmer. Two methods were used to evaluate the effectiveness of the tag suggestion methods. First, we manually examined the tags assigned to a random subset of 30 posts from our col￾lection; for each tag, we decided whether the tag was indeed a relevant label for the post. This is the preferred method of evaluation, but due to its cost it can only be applied to a small number of posts; additionally, it is difficult to use non automated methods to tune and improve a system. Be￾cause of this, we used an automated method to evaluate a much larger subset of posts: AutoTag was used to tag 6000 of the “tagged posts” in our corpus - the posts which were assigned tags by their authors (we used only posts with 3 or more tags). Then, AutoTag’s output was compared to the actual post tags. To account for minor differences in tags (“blogs” and “blogging”), we used string distance to compare the tags rather than exact string match. Even so, the automated precision scores are lower than manual ones, since tags which are useful for a post but were not originally used by its author are mistakenly taken to be incorrect. To demonstrate this, we evaluated the small test set with the automated method as well, resulting in substantially lower scores; this indicates that the actual performance of Auto￾Tag on the large set is likely to be much better than reported by the automated evaluation. For the manual evaluation, we measured precision at 10: the fraction of tags out of the top-10 suggestions by AutoTag which were judged as appropriate for the post by a human; 1http://www.blogpulse.com/www2006-workshop only the first 10 results are checked because it is assumed that users are unwilling to examine long result lists. For the automated evaluation, we measured recall at 10 as well: the fraction of tags offered by AutoTag in the top-10 suggestions which were also assigned by the blogger out of the total number of tags assigned by her. Test Set Evaluation Precision@10 Recall@10 30 posts Automated 0.38 0.47 30 posts Manual 0.59 — 6000 posts Automated 0.40 0.49 Table 1: Auto-tagging accuracy Results are shown in Table 1; an example of tags offered by AutoTag is given in Table 2. On average, 4 to 6 suggestions out of AutoTag’s top-10 suggestions are either considered useful by the blogger, or were actually used by her for the given post. Cursory examination of posts for which AutoTag scores low shows many non-English posts (for which much less data exists in the corpus, entailing lower success of data￾driven methods), and many tags which are highly personal and used by few bloggers (such as names of family members). http://www.stillhq.com/diary/000959.html On pitching products to bloggers Anil comments on how to pitch a product to him as a blogger, and makes good suggestions as Lindsay agrees before getting distracted by how this applies to press releases. I have to wonder though how much of this promotional pitching actually happens. I certainly haven’t ever had a product pitched to me for this site. I’ve had people pitch advertising, and other spammy things, but not products. Does it really happen to us normal people bloggers? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Suggested tags: PR, blogging, weblogs, marketing, net, gripes, email, small business life, Anil Dash, PR pitching Original tags: blog, pitch, product, marketing, communications Table 2: Sample tags suggested for a post 4. CONCLUSIONS We proposed and evaluated AutoTag, a tool for tagging weblog posts based on a collaborative filtering approach. AutoTag offers suggestions for tags based on tags assigned to other, similar posts; the final decision about a tag is left to the blogger. Despite a relatively small corpus for this type of task, AutoTag shows good results, and has the potential to benefit both the bloggers and others making use of tags assigned to weblog posts. The different components of AutoTag provide fertile ground for further work: identifying effective ways to generate queries from a post and successful retrieval models to use; improving the aggregation of tags from the retrieved posts; and vari￾ous methods for filtering and reranking the lists produced by AutoTag. In addition to the collaborative approaches described in this paper, we are currently investigating a “lo￾cal” approach to tag suggestion, in which suggestions for tags are made without access to the entire blogosphere as is the case with AutoTag, but using deeper analysis of the contents of the post and the blog it belongs to. 5. REFERENCES [1] S. Golder and B. A. Huberman. The structure of collaborative tagging systems. J. Inf. Science, 2006. [2] A. Mathes. Folksonomies: Cooperative classification and communication through shared metadata. Computer Mediated Communication, 2004. [3] P. Resnick and H. R. Varian. Recommender systems (special section). Comm. of the ACM, 40(3), 1997