009 Eighth International Symposium on Natural Language Processing Improving research Paper Searching with Social Tagging A Preliminary Investigation P. Jomsri. S. Sanguansintukul and W. Choochaiwattana Abstract-The www provides an efficient way to store The tags can be useful for tasks such as search, navigatio and share information. Search engines and social or information extraction. Therefore. it is interesting to okmarking systems are important tools for web resource investigate how well a set of tags for the link to academic discovery. This study investigated three different indexing research papers on CiteULike contribute to search results approaches applied to CiteULike- a social bookmarking when they are used as research paper indexes system for tagging academie research papers. The indexing approaches here are known as: Tag only; Title with Abstract; In this paper, we investigated the use of social tagging to and Tag, Title with Abstract. These three indexing improve research paper indexing. We proposed an indexing approaches were evaluated using mean values of Normalized method using tagging information together with a title and Discount Cumulative Gain(NDCG). The preliminary results abstract of the paper(TTA). We refer to it as a"Tag Title illustrated that indexing using "Tag, Title, with with Abstract"indexing method. To evaluate the proposed performed the best. The initial evaluation indexing method, it was compared with two indexing implementation implied that these designs might im approaches: tagging information only indexing method and ccuracy and efficiency of web resouree searching mprove the title with abstract indexing method. We refer to them as bookmarking system, not only in academics but also in other domains "Tag Only' indexing method (T) andTitle with Abstraction"indexing method (TA)respectively The paper is structured as follows. First, we discuss L INTRODUCTION related work in Section i. we then describe framework There are an increasing number of people using the Sect son ai taghensectisnd re is re splen d discussion. documents on the internet. A social bookmarking system is wor Section V contains the Conclusion and Future is one important tool that allows people to search for also an important tool that allows people to share interesting web resources. It not only provides web IL. RELATED WORK resource sharing functions but also allows people to create Researchers who studied CiteULike include: Capocci a set of tags attached with the web resource (2007) analyzed the small-world properties of the Citeulike(wWw. cIteulike. org is a fusion of Web CiteULike folksonomy [2] and Santos-Neto(2007) based social bookmarking services and traditional explore presentin bibliographic management tools. It helps scientists characterizations of CiteULike and Bibsonomy that target researchers and academics store, organize, share and the management of scientific literature [15]. Toine Bogers discover links to academic research papers. It has been (2008)employed CiteULike to generate reading lists for publicly available to use since November 2004. Like many scientific articles based on the user's online reference successful tools, CiteULike has a flexible filing system library. They applied three different CF algorithms and ased on tags. These tags provide an open, quick and user- found that user-based filtering performs the best[14].Noel defined classification model that can produce interesting (2008)looked at the tagging behavior of people who were user-defined categorizations describing four frequently entered references[12]. The While the primary goal of these applications is to serve techniques from CiteULike have been applied to other the needs of individual users, the tags of each web academic searching, such as Farooq(2007)where four resource,links to academic research papers for each novel implications for designing the CiteSeer[4][5],[71 particular case, should also help other users to categorize, browse, and find items. The tags can also be used for Researchers who studied and improved social information discovery, sharing, and community ranking Suchanek(2008)found that tags are "meaningful Manuscript received June 15, 2009 >gging process is influenced by tag suggestions[12] while P. Jomsri, Department of Mathematics, Faculty of Scienc Chulalongkorn Univers communication in these systems in social tagging [13] PijitraJ@Student chula ac th Gelernter(2008)compares the information retrieval value S. Sanguansintukul, Department of Mathematics, Faculty of Science, of the cloud format tags and the tag words themselves as Chulalongk Thailand. (e-m found in the Library Thing catalog. Results also show that whether searchers are working toward research or personal Pundit University, Bangkok, Thailand (e-mail: worasit cha @dpu ac th) ends, high recall matters [6].A. Budura(2008 )present 978-1-4244-4139609525.00c2009IEEE
Abstract— The WWW provides an efficient way to store and share information. Search engines and social bookmarking systems are important tools for web resource discovery. This study investigated three different indexing approaches applied to CiteULike – a social bookmarking system for tagging academic research papers. The indexing approaches here are known as: Tag only; Title with Abstract; and Tag, Title with Abstract. These three indexing approaches were evaluated using mean values of Normalized Discount Cumulative Gain (NDCG). The preliminary results illustrated that indexing using “Tag, Title, with Abstract” performed the best. The initial evaluation on our implementation implied that these designs might improve the accuracy and efficiency of web resource searching on social bookmarking system, not only in academics but also in other domains. I. INTRODUCTION here are an increasing number of people using the internet to exchange information. Thus, a search engine is one important tool that allows people to search for documents on the internet. A social bookmarking system is also an important tool that allows people to share interesting web resources. It not only provides web resource sharing functions but also allows people to create a set of tags attached with the web resource. CiteULike (www.CiteULike.org) is a fusion of Webbased social bookmarking services and traditional bibliographic management tools. It helps scientists, researchers and academics store, organize, share and discover links to academic research papers. It has been publicly available to use since November 2004. Like many successful tools, CiteULike has a flexible filing system based on tags. These tags provide an open, quick and userdefined classification model that can produce interesting user-defined categorizations. While the primary goal of these applications is to serve the needs of individual users, the tags of each web resource, links to academic research papers for each particular case, should also help other users to categorize, browse, and find items. The tags can also be used for information discovery, sharing, and community ranking. Manuscript received June 15, 2009. P. Jomsri, Department of Mathematics, Faculty of Science, Chulalongkorn University, Bangkok, Thailand. (e-mail: Pijitra.J@Student.chula.ac.th). S. Sanguansintukul, Department of Mathematics, Faculty of Science, Chulalongkorn University, Bangkok, Thailand. (e-mail: siripun.s@chula.ac.th). W. Choochaiwattana, Faculty of Information Technology,Dhurakij Pundit University, Bangkok,Thailand. (e-mail: worasit.cha@dpu.ac.th). The tags can be useful for tasks such as search, navigation or information extraction. Therefore, it is interesting to investigate how well a set of tags for the link to academic research papers on CiteULike contribute to search results when they are used as research paper indexes. In this paper, we investigated the use of social tagging to improve research paper indexing. We proposed an indexing method using tagging information together with a title and abstract of the paper (TTA). We refer to it as a “Tag Title with Abstract” indexing method. To evaluate the proposed indexing method, it was compared with two indexing approaches: tagging information only indexing method and title with abstract indexing method. We refer to them as “Tag Only” indexing method (T) and “Title with Abstraction” indexing method (TA) respectively. The paper is structured as follows. First, we discuss related work in Section II. We then describe Framework for social tagging based research paper searching in Section III. The Section IV is Result and Discussion. Finally, Section V contains the Conclusion and Future work. II. RELATED WORK Researchers who studied CiteULike include: Capocci (2007) analyzed the small-world properties of the CiteULike folksonomy [2] and Santos-Neto (2007) explored three main directions for presenting characterizations of CiteULike and Bibsonomy that target the management of scientific literature [15]. Toine Bogers (2008) employed CiteULike to generate reading lists for scientific articles based on the user’s online reference library. They applied three different CF algorithms and found that user-based filtering performs the best [14]. Noël (2008) looked at the tagging behavior of people who were describing four frequently entered references [12]. The techniques from CiteULike have been applied to other academic searching, such as Farooq (2007) where four novel implications for designing the CiteSeer [4], [5], [7] were presented. Researchers who studied and improved social tagging: Suchanek (2008) found that tags are “meaningful” and the tagging process is influenced by tag suggestions [12] while Thom-Santelli (2008) explored the use of tags for communication in these systems in social tagging [13]. Gelernter (2008) compares the information retrieval value of the cloud format tags and the tag words themselves as found in the LibraryThing catalog. Results also show that, whether searchers are working toward research or personal ends, high recall matters [6].A. Budura (2008) present Improving Research Paper Searching with Social Tagging – A Preliminary Investigation P. Jomsri, S. Sanguansintukul, and W. Choochaiwattana T 978-1-4244-4139-6/09/$25.00 ©2009 IEEE 152 2009 Eighth International Symposium on Natural Language Processing
HAMLET to promote an efficient and precise reuse of shared metadata in dynamic where tags are scarce [1]. J. Gelernter(2008)offers a method of evaluating user fidic= n,d:t,∈d tag preference and the relative strength of social tag LCSH string retrieval performance. Choochaiwattana (2009)examined the use of social annotations to improve the quality of web searches [3]. Li (2008) use the self fdf,=n×log ∑n,°(d:t1∈d organizing characteristics of SoM neural networks to classify the popular tags in"Del icio us"website [101 tfid/, lo (3) IIL. FRAMEWORK FOR SOCIAL TAGGING BASED RESEARCH n t.∈d In this section, we discuss the experimental design and Let ny be the number of occurrences of the considered evaluation method. The experiment was divided into five term in document d, I T I is total number of"Tag Only steps follow to Fig. I documents in the corpus, I TA I is total number of“" Tag and Abstract"documents in the corpus, and I TTA I is total number of"Tag, Title with Abstract"documents in the corpus TTA. d: t,edy is number of documents where the term t, appears(that is ny +0). If the term is not in the corpus, Searchpaper WWw this will lead to a division-by-zero. It is therefore common Cite C. Research Paper Searching Three search engines based on the three indexers were Research paper search developed. Fig. 2 shows an interface of the first search engine. Fig 3 shows an interface of search result. Subjects can see: titled of the document. title name that can link for Crawler ind link for obtaining data from CiteULike Fl Edt wew Farvurtes Toots Hak Index ⊙=0沿P如m合,回,回 Result tser Result Welcome to Research Paper Search Engine Fig 1. A framework for social tagging based research paper searching 20 Resuts Per Page [Search J A. Research Paper Crawling Copyright 2009 Research papers were crawled from CiteULike between March and May 2009. The final set consisted of 62. 192 research papers from 103 research communities related Fig 2 Shows Research Paper Search Engine web page. computer science. To compare a query with research paper indexes, we B. Research Paper Indexing used a cosine similarity measurement to retrieve and rank In our experiment, three different indexers were research paper search results developed. The equation(1),(2), and (3)show a modified D. Experimental Setting Term Frequency/Inverse Document Frequency (tf/idf Fifteen subjects who were lecturers and Ph. D. students formula for the different indexers, where T is"Tag only", from Chulalongkorn University were recruited to be TA is"Title with Abstract, and TTA is"Tag. Title with participants In the experiments, each subject was assigned abstract to find research papers using our search engines. Each
HAMLET to promote an efficient and precise reuse of shared metadata in highly dynamic where tags are scarce [1]. J. Gelernter (2008) offers a method of evaluating user tag preference and the relative strength of social tag vs. LCSH string retrieval performance. Choochaiwattana (2009) examined the use of social annotations to improve the quality of web searches [3]. LI (2008) use the self organizing characteristics of SOM neural networks to classify the popular tags in "Del.icio.us" website [10]. III. FRAMEWORK FOR SOCIAL TAGGING BASED RESEARCH PAPER SEARCHING In this section, we discuss the experimental design and evaluation method. The experiment was divided into five steps follow to Fig.1. Fig.1. A framework for social tagging based research paper searching A. Research Paper Crawling Research papers were crawled from CiteULike between March and May 2009. The final set consisted of 62,192 research papers from 103 research communities related to computer science. B. Research Paper Indexing In our experiment, three different indexers were developed. The equation (1), (2), and (3) show a modified Term Frequency/Inverse Document Frequency (tf/idf) formula for the different indexers, where T is “Tag only”, TA is “Title with Abstract”, and TTA is “Tag, Title with Abstract”: ^ ` d t d T n n tfidf k k j i i j i j u ¦ : log , , , ^ ` d t d TA n n tfidf k k j i i j i j u ¦ : log , , , ^ ` d t d TTA n n tfidf k k j i i j i j u ¦ : log , , , Let ni,j be the number of occurrences of the considered term in document dj , | T | is total number of “Tag Only” documents in the corpus, | TA | is total number of “Tag and Abstract” documents in the corpus, and | TTA | is total number of “Tag, Title with Abstract” documents in the corpus TTA. ^ ` d t d : i is number of documents where the term ti appears (that is nij z 0 ). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to use ^ ` d t d 1 : i . C. Research Paper Searching Three search engines based on the three indexers were developed. Fig. 2 shows an interface of the first search engine. Fig. 3 shows an interface of search result. Subjects can see: titleID of the document, title name that can link for download full paper and link for obtaining data from CiteULike. Fig.2. Shows Research Paper Search Engine web page. To compare a query with research paper indexes, we used a cosine similarity measurement to retrieve and rank research paper search results. D. Experimental Setting Fifteen subjects who were lecturers and Ph.D. students from Chulalongkorn University were recruited to be participants. In the experiments, each subject was assigned to find research papers using our search engines. Each (3) (2) (1) 153
subject was given three questions to find related research and TTA is the "Tag, Title with Abstract" indexing method papers. They formulated their own queries according to the as shown in Table I given questions. They were asked to use same query for each search engine. Then, they were asked to rate the TABLE I TOP 10 RANKS AVERAGE NDCG FOR THREE DIFFERENT relevancy of the search result set on a five-point scale SEARCH ENGINES Score o is not relevant at all Average NDCG Score I is probably not relevant. TTA Score 2 is less relevant 0.4476 0.6106 Score 3 is probably relevant Score 4 is extremely relevant 0.4733 0.6305 0.6748 0490906240 0.517 0.5282 0.6344 0.6225 0.6367 0.522 0.6218 0.5214 0.6225 0.6189 0.6222 0.5331 0.6149 0.6267 Fig. 4 compares the average of NDCG score on three different search engines. The x-axis denotes the first 10 ranks of the search results, whereas the y-axis represents adage the average ndcg score Fig 3. Example of search results It suggests that the"Tag, Title with Abstract" indexing method provide a better set of search results compared with The top 20 search results of each search en "Tag only "indexing method and the"Title with Abstract displayed for relevancy judgment. The subject indexing method experiment were considered experts in the 0. science field; their relevancy ratings for each query are considered to be perfect 中Tag TItle and Astac E. Evaluation metric NDCG(Normalized Discounted Cumulative Gain)as originally proposed by Jarvelin and Kekalainen [9], was so used to evaluate the performance of each search engine This metric is a retrieval measurement devised specifically for web search evaluation. The NDCG is computed as in the equation(4) ADC=M.12(+ Where k is a truncation or threshold level, r() is an integer representing the relevancy given by the subject, and Fig 4 Comparison of the average NDCG for three indexing methods. M is a normalization constant calculated so that the perfect ordering would obtain a ndcg of 1. ndcg rewards Table ll. result of ftest relevant documents appearing in the top ranked search Rank sults and punishes irrelevant documents by reducing their F-test contributions to ndcg 0.008 IV RESULT AND DISCUSSION 0.000 We present the top ten ranks of average NDCG scores 12165 0.000 where k is level of the rank. T is"Tag only"indexing 13.971 0.000 method. TA is the "Title with abstract" indexing method 15.533 0.000
subject was given three questions to find related research papers. They formulated their own queries according to the given questions. They were asked to use same query for each search engine. Then, they were asked to rate the relevancy of the search result set on a five-point scale: Score 0 is not relevant at all. Score 1 is probably not relevant. Score 2 is less relevant. Score 3 is probably relevant. Score 4 is extremely relevant. Fig.3. Example of search results The top 20 search results of each search engine were displayed for relevancy judgment. The subjects in this experiment were considered experts in the computer science field; their relevancy ratings for each query are considered to be perfect. E. Evaluation Metric NDCG (Normalized Discounted Cumulative Gain) as originally proposed by Jarvelin and Kekalainen [9], was used to evaluate the performance of each search engine. This metric is a retrieval measurement devised specifically for web search evaluation. The NDCG is computed as in the equation (4). ¦ k j r j q q j NDCG M 1 log 1 2 1 Where k is a truncation or threshold level, r(j) is an integer representing the relevancy given by the subject, and Mq is a normalization constant calculated so that the perfect ordering would obtain a NDCG of 1. NDCG rewards relevant documents appearing in the top ranked search results and punishes irrelevant documents by reducing their contributions to NDCG IV. RESULT AND DISCUSSION We present the top ten ranks of average NDCG scores where k is level of the rank. T is “Tag only” indexing method, TA is the “Title with Abstract” indexing method, and TTA is the “Tag, Title with Abstract” indexing method as shown in Table I. TABLE I. TOP 10 RANKS AVERAGE NDCG FOR THREE DIFFERENT SEARCH ENGINES K Average NDCG T TA TTA 1 0.4476 0.6106 0.7187 2 0.4733 0.6305 0.6748 3 0.4909 0.6240 0.6515 4 0.5171 0.6334 0.6329 5 0.5282 0.6274 0.6344 6 0.5211 0.6225 0.6367 7 0.5227 0.6218 0.6281 8 0.5214 0.6197 0.6225 9 0.5279 0.6189 0.6222 10 0.5331 0.6149 0.6267 Fig. 4 compares the average of NDCG score on three different search engines. The x-axis denotes the first 10 ranks of the search results, whereas the y-axis represents the average NDCG score. It suggests that the “Tag, Title with Abstract” indexing method provide a better set of search results compared with “Tag only” indexing method and the “Title with Abstract” indexing method. Fig.4 Comparison of the average NDCG for three indexing methods. Table II. Result of F-test Rank (K) N F-test sig (2-tailed) 1 45 5.071 0.008 1-2 90 9.155 0.000 1-3 135 12.165 0.000 1-4 180 13.971 0.000 1-5 225 15.533 0.000 (4) 154
We applied One Way ANOVA on NDCG at K=l, 1-2, the"Tag only" indexing approach. In order to confirm the 1-3, 1-4 and 1-5 respectively to test whether there is a result of the experiment, additional experiments should be difference among the mean NDCG from three different conducted. Improving research paper indexing approaches dexing approaches. As shown in Table Il, we found that not only enhance the performance of academic paper there was evidence that not all of the means of NDCG of searches, but also all document searches in general the three indexing approaches were equal at a=0.05 levels We plan to integrate social tagging information to of significance. In other words, the difference in the set of improve a ranking of search results. Future research in search results returned from three different indexing the area consists of extending the scale of experiments approaches were statistically significant. comparing better index with CiteULike, der ranking as well as optimizing the parameters Table Ill Result of mul of top five ranks Rank Difference Error(2-tailed) The authors would like to thank suan sunandha 62936 Rajabhat University for scholarship support. The study TA TTA not possible without the data from CiteULike 0857099 TTA 2711ll 005 10814 REfERENCES 160087 [1]T. Budura, S. Michel, P. Cudre-Mauroux, and K. Aberer, "To .076224 236311 .0563720 Systems', SIGIS08, Singapore, 20-24 July 2008 TTA [2] A Capocci and G. Caldarelli, "Folksonomies and Clustering 076224 in the Collaborative System CiteULike, eprint ar Xiv: 0710.2835 151072 TA 3] w. Choochaiwattana, and M. B. Spring, Applying Social TTA 059981 annotations to Retrieve 0440961 211053 Proceedings of 2009 International Conference on Information TTA Management and Engineering(ICIME 2009), Kuala Lumpur TA 059981 Malaysia 3-5 April 2009 14 U. Farooq, C H. Ganoe, J M. Carroll, and C TA "Supporting distributed scientific collaboration: Im 036986 fordesigning the CiteSeer collaborator" Proceedings T 187231 awaii Int'/ Conference on System Sciences), IEEE c TTA Society, Waikoloa, Hawaii, 3-6 January 2007, 26c. 5]U. Farooq, T.G. Kannampallil, Y Song, C H. Ganoe John M. Carroll 133654 nd C. Lee Giles, "Evalating Tagging Behavior in Social Bookman TA Systems: Metrics and design heuristics", Proceedings of the 2007 037276 0322510 T 170930 (GROUP"07Sanibel Island, Florida, USA, 4-7 November 2007, pp. 351-360 TTA [6]J. Gelernter,"A Quantitative Analysis of Collaborative Tags 037276 ternational Conference on Collaborative Computing: Nenvorking applications and worksharing. 2007. Collaborate Com 2007., 12-15 Nov We then preformed a multiple comparisons to find the 2007. New York. ny. Dp 376-38 differences among the three indexing approaches table Im 7 C L. Giles, K. Bollacker, and S. Lawrence, CiteSeer: An shows the result of the multiple comparisons of the three Automatic Citation Indexing System", Proceedings of the difference indexing approaches. Although Fig. 4 suggests Conference on Digital Libraries, ACM Press, Pittsburg, that the " Tag, Title with Abstract" indexing approach Pennsylvania, 23-26 June 1998, pp. 89-98 provides a better set of search results compared with the 8]K. Jarvelin, and J. Kekalainen, "IR evaluat for retrieving highly relevant documents, Proceeding of the orld Wide other two approaches, the results from the multiple Web Conference(ww2006), May 2006 comparisons show that a set of search results provided by 9R. aschke, L. B. Marinho, A. Hotho, L. Schmidt-Thieme, and G. the"Tag, Title with Abstract" indexing approach is not Tag Recommendations in Folksonomies", In Proceedings of statistically different from a set of search results provided 4702 of Lecture Notes in Computer Science, by the"Title with Abstract indexing approach Verlag,2007,pp.506-514 [10B. Ll, and Q. Zhu, " The Determination of Semantic Dimension in V. CONCLUSION AND FUTURE WORK 2008(,08),20-22 Dec2008, Shanghai,pp909913 Appdioonal [11]S. Noel, and R. Beale, " Sharing vocabularies: Tag Usage in This paper proposes to use an interesting method of CiteULike", Proceedings of the 22 Anmual Conference of Interaction,a indexing of research papers on CiteULike. The experiment Specialist group of the British Computer Societ(HCl shows that the"Title with Abstract"indexing approach and 12]F. M. Suchanek, M. Vojnovic, and D Gunawardena, ""Social Tags the Tag, Title with Abstract" indexing approach provides Meaning and Suggestions", CIKM08, Napa Valley, California, USA. 26 a better research paper search result set as compared with 0 October 2008
We applied One Way ANOVA on NDCG at K=1, 1-2, 1-3, 1-4 and 1-5 respectively to test whether there is a difference among the mean NDCG from three different indexing approaches. As shown in Table II, we found that there was evidence that not all of the means of NDCG of the three indexing approaches were equal at Į=0.05 levels of significance. In other words, the difference in the set of search results returned from three different indexing approaches were statistically significant. Table III Result of Multiple Comparisons of top five ranks Rank (K) Indexing Mean Difference (I-J) Std. Error sig (I) (J) (2-tailed) 1 TA T .162936 .0857099 .142 TTA -.108148 .419 TTA T .271111 .005 TA .108148 .419 1-2 TA T .160087 .0563720 0.13 TTA -.076224 .038 TTA T .236311 .000 TA .076224 .368 1-3 TA T .151072 .0440961 .002 TTA -.059981 .363 TTA T .211053 .000 TA .059981 .363 1-4 TA T .142373 .0369865 .000 TTA -.044857 .446 TTA T .187231 .000 TA .044857 .446 1-5 TA T .133654 .0322510 .000 TTA -.037276 .480 TTA T .170930 .000 TA .037276 .480 We then preformed a multiple comparisons to find the differences among the three indexing approaches. Table III shows the result of the multiple comparisons of the three difference indexing approaches. Although Fig. 4 suggests that the “Tag, Title with Abstract” indexing approach provides a better set of search results compared with the other two approaches, the results from the multiple comparisons show that a set of search results provided by the “Tag, Title with Abstract” indexing approach is not statistically different from a set of search results provided by the “Title with Abstract” indexing approach. V. CONCLUSION AND FUTURE WORK This paper proposes to use an interesting method of indexing of research papers on CiteULike. The experiment shows that the “Title with Abstract” indexing approach and the “Tag, Title with Abstract” indexing approach provides a better research paper search result set as compared with the “Tag only” indexing approach. In order to confirm the result of the experiment, additional experiments should be conducted. Improving research paper indexing approaches not only enhance the performance of academic paper searches, but also all document searches in general. We plan to integrate social tagging information to improve a ranking of search results. Future research in the area consists of extending the scale of experiments, comparing better index with CiteULike, developing ranking as well as optimizing the parameters. ACKNOWLEDGMENT The authors would like to thank Suan Sunandha Rajabhat University for scholarship support. The study is not possible without the data from CiteULike. REFERENCES [1] T. Budura, S. Michel, P. Cudre-Mauroux, and K. Aberer, “To Tag or Not to tag-Harvesting Adjacent Metadata in Large-Scale Tagging Systems”, SIGIS’08, Singapore, 20-24 July 2008. [2] A. Capocci and G. Caldarelli, “Folksonomies and Clustering in the Collaborative System CiteULike”, eprint arXiv: 0710.2835, 2007. [3] W. Choochaiwattana, and M.B. Spring, “Applying Social Annotations to Retrieve and Re-rank Web Resources”, Proceedings of 2009 International Conference on Information Management and Engineering (ICIME 2009), Kuala Lumpur, Malaysia 3 – 5 April 2009. [4] U. Farooq, C.H. Ganoe, J.M. Carroll, and C.L. Giles, “Supporting distributed scientific collaboration: Implications fordesigning the CiteSeer collaborator” Proceedings of the Hawaii Int’l Conference on System Sciences), IEEE Compute Society,Waikoloa, Hawaii, 3-6 January 2007, 26c. [5] U. Farooq, T.G. Kannampallil, Y. Song ,C.H. Ganoe ,John M. Carroll ,and C. Lee Giles, “Evalating Tagging Behavior in Social Bookmarking Systems: Metrics and design heuristics”, Proceedings of the 2007 international ACM conference on Supporting group work (GROUP’07)Sanibel Island,Florida, USA,4-7 November 2007, pp.351-360. [6] J. Gelernter, “A Quantitative Analysis of Collaborative Tags: Evaluation for Information Retrieval—a Preliminary Study”, International Conference on Collaborative Computing: Networking, Applications and Worksharing, 2007. CollaborateCom 2007. ,12-15 Nov. 2007, New York, NY. pp.376-381 [7] C.L. Giles, K. Bollacker, and S. Lawrence, “CiteSeer: An Automatic Citation Indexing System”, Proceedings of the Conference on Digital Libraries,ACM Press, Pittsburg, Pennsylvania, 23-26 June 1998, pp. 89-98. [8] K. Jarvelin, and J. Kekalainen, “IR evaluation methods for retrieving highly relevant documents”, Proceeding of the International World Wide Web Conference (WWW2006), May 2006. [9] R. J¨aschke, L. B. Marinho, A. Hotho, L. Schmidt-Thieme, and G. Stumme, “Tag Recommendations in Folksonomies”, In Proceedings of PKDD 2007, volume 4702 of Lecture Notes in Computer Science, Springer Verlag, 2007, pp. 506–514. [10] B. LI, and Q. Zhu, “The Determination of Semantic Dimension in Social Tagging System Based on SOM Model”, Second International Symposium on Intelligent Information Technology Application 2008(IITA’08), 20-22 Dec. 2008, Shanghai, pp. 909-913 [11] S. Noel, and R. Beale, “Sharing vocabularies : Tag Usage in CiteULike”, Proceedings of the 22nd Annual Conference of Interaction, a Specialist group of the British Computer Society(HCI2008), Liverpool, UK, 1-5 September. [12] F. M. Suchanek, M. Vojnovi´c, and D. Gunawardena, “Social Tags: Meaning and Suggestions”, CIKM’08, Napa Valley, California, USA. 26– 30 October 2008. 155
[13]J. Thom-Santelli, and M. J. Muller, David R. Millen, "Social Tagging Roles: Publishers, Evangelists, Leaders", CHI 2008, Florence, Italy, 5 10Aprl2008 [14]T. Toine Bogers, and A. van den Bosch, "Recommending Scientific Articles Using CiteULike" RecSys 08, Switzerland, 23-25 October 2008. [15 E Santos-Neto, M. Ripeanu, and A lamnitchi, "Tracking usage [oNline).available:http://www.csee.usf.edu/-anda/papers/camao7_read [16]E. Hatcher, and O. Gospodnetic, Lucene in Action, Mannin Publications Co., United States of America, 2005
[13] J. Thom-Santelli, and M. J. Muller, David R. Millen, “Social Tagging Roles: Publishers, Evangelists, Leaders”, CHI 2008, Florence, Italy, 5- 10April 2008. [14] T. Toine Bogers, and A. van den Bosch, “Recommending Scientific Articles Using CiteULike”, RecSys’08, Switzerland, 23-25 October 2008. [15] E. Santos-Neto, M. Ripeanu, and A. Iamnitchi, “Tracking usage in collaborative tagging communities” [Online].Available:http://www.csee.usf.edu/~anda/papers/CAMA07_read y_v2.pdf [16] E. Hatcher , and O. Gospodnetic, Lucene in Action, Manning Publications Co., United States of America, 2005. 156