Susan T.Dumais is a senior researcher in the Decision Theory and Adap tive Systems Group at Microsoft Research.Her research interests include algorithms and interfaces for improved information retrieval and classifica- tion,human-computer interaction,combining search and navigation,user method consists of solving a linear modeling,individual differences,collaborative filtering,and organizational eigenvalue problem for a matrix whose impacts of new technology.She received a BA in mathematics and psychol- ogy from Bates College and a PhD in cognitive psychology from Indiana elements are computed using the kernel University.She is a member of the ACM,the ASIS,the Human Factors and function.The resulting feature extrac- Ergonomic Society,and the Psychonomic Society.Contact her at Microsoft tors have the same architecture as SV Research,1 Microsoft Way,Redmond,WA 98052;sdumais@microsoft. machines(see Figure 4).A number of com;http://research.microsoft.com/-sdumais. researchers have since started to"ker- Edgar Osuna has just returned to his native Venezuela after receiving his nelize"various other linear algorithms. PhD in operations research from the Massachusetts Institute of Technology His research interests include the study of different aspects and properties of Vapnik's SVM.He received his BS in computer engineering from the References Universidad Simon Bolivar,Caracas,Venezuela.Contact him at IESA, 1.B.E.Boser,I.M.Guyon,and V.N.Vapnik, POBA International #646,PO Box 02-5255.Miami,FL 33102-5255: "A Training Algorithm for Optimal Margin eosuna/@usb.ve. Classifiers,"Proc.Fifth Ann.Workshop Computational Learning Theory,ACM Press,New York,1992,pp.144-152. 2. V.Vapnik,The Nature of Statistical Learn- John Platt is a senior researcher in the Signal Processing Group at Micro- ing Theory,Springer-Verlag,New York, 1995. soft Research.Recently,he has concentrated on fast general-purpose ma- chine-learing algorithms for use in processing signals.More generally,his 3. B.Scholkopf,A.Smola,and K.-R.Muller, research interests include neural networks,machine learning,computer "Nonlinear Component Analysis as a Ker- nel Eigenvalue Problem,"Neural Computa- vision,and signal processing.He received his PhD in computer science at Caltech in 1989,where he studied computer graphics and neural networks. tion,Vol.10,1998,pp.1299-1319. His received his BS in chemistry from California State University at Long 4. B.Scholkopf,C.J.C.Burges,and A.J. Beach.Contact him at Microsoft Research,I Microsoft Way,Redmond, Smola,Advances in Kernel Methods-Sup- WA 98005;jplatt @microsoft.com;http://research.microsoft.com/-jplatt. port Vector Learning,to appear,MIT Press, Cambridge,Mass,1998. Bernhard Scholkopf is a researcher at GMD First,Berlin.His research 5. B.Scholkopfetal,"Support Vector Regres- interests are in machine leaming and perception,in particular using SVMs sion with Automatic Accuracy Control,"to be and Kemel PCA.He has an MSc in mathematics from the University of published in Proc.Eighth Int'I Conf.Artifi- London,and a Diploma in physics and a PhD in computer science,both cial Neural Networks,Perspectives in Neural from the Max Planck Institute.Contact him at GMD-First,Rm.208, Computing,Springer-Verlag,Berlin,1998. Rudower Chaussee 5,D-12489 Berlin;bs @first.gmd.de:http://www.first. 6. C.J.C.Burges,"Simplified Support Vector gmd.de/-bs. Decision Rules,"Proc.13th Int'l Conf. Machine Learning,Morgan Kaufmann,San Francisco,1996,pp.71-77. 1. A.Smola and B.Scholkopf,"From Regu- larization Operators to Support Vector Ker- nels,"Advances in Neural Information Pro- cessing Systems 10,M.Jordan,M.Kearns, methods,including SVMs,have tremendous bility-especially for large or rapidly and S.Solla,eds.,MIT Press,1998. potential for helping people more effectively changing collections.Consequently,inter- 8.F.Girosi.An Equivalence benveen Sparse organize electronic resources. est is growing in developing technologies Approximation and Support Vector Machines, Today,most text categorization is done by for(semi)automatic text categorization. AI Memo No.1606,MIT,Cambridge,Mass. 1997. people.We all save hundreds of files,e-mail Rule-based approaches similar to those 9. J.Weston et al.,Density Estimation Using messages,and URLs in folders every day. employed in expert systems have been used Support Vector Machines.Tech.Report We are often asked to choose keywords but they generally require manual construc- CSD-TR-97-23,Royal Holloway,Univ.of from an approved set of indexing terms for tion of the rules.make rigid binary deci- London,1997. describing our technical publications.On a sions about category membership,and are much larger scale,trained specialists assign typically difficult to modify.Another strat- new items to categories in large taxonomies egy is to use inductive-learning techniques Using SVMs for text categorization such as the Dewey Decimal or Library of to automatically construct classifiers using Congress subject headings,Medical Subject labeled training data.Researchers have ap- Susan Dumais.Decision Theory and Adap- Headings(MeSH),or Yahoo!'s Internet di- plied a growing number of learning tech- tive Systems Group,Microsoft Research rectory.Between these two extremes,people niques to text categorization,including As the volume of electronic information organize objects into categories to support a multivariate regression,nearest-neighbor increases,there is growing interest in devel- wide variety of information-management classifiers,probabilistic Bayesian models, oping tools to help people better find,filter, tasks,including information routing/filter- decision trees,and neural networks.2Re- and manage these resources.Text categoriza- ing/push,identification of objectionable cently,my colleagues and I and others have tion-the assignment of natural-language materials or junk mail,structured search and used SVMs for text categorization with texts to one or more predefined categories browsing,and topic identification for topic- very promising results.3.4 In this essay,I based on their content-is an important com- specific processing operations. briefly describe the results of experiments ponent in many information organization Human categorization is very time-con- in which we use SVMs to classify newswire and management tasks.Machine-leaming suming and costly,thus limiting its applica- stories from Reuters.4 JULY/AUGUST 1998 21JULY/AUGUST 1998 21 method consists of solving a linear eigenvalue problem for a matrix whose elements are computed using the kernel function. The resulting feature extrac￾tors have the same architecture as SV machines (see Figure 4). A number of researchers have since started to “ker￾nelize” various other linear algorithms. References 1. B.E. Boser, I.M. Guyon, and V.N. Vapnik, “A Training Algorithm for Optimal Margin Classifiers,” Proc. Fifth Ann. Workshop Computational Learning Theory, ACM Press, New York, 1992, pp. 144–152. 2. V. Vapnik, The Nature of Statistical Learn￾ing Theory, Springer-Verlag, New York, 1995. 3. B. Schölkopf, A. Smola, and K.-R. Müller, “Nonlinear Component Analysis as a Ker￾nel Eigenvalue Problem,” Neural Computa￾tion, Vol. 10, 1998, pp. 1299–1319. 4. B. Schölkopf, C.J.C. Burges, and A.J. Smola, Advances in Kernel Methods—Sup￾port Vector Learning, to appear, MIT Press, Cambridge, Mass, 1998. 5. B. Schölkopf et al., “Support Vector Regres￾sion with Automatic Accuracy Control,” to be published in Proc. Eighth Int’l Conf. Artifi￾cial Neural Networks, Perspectives in Neural Computing, Springer-Verlag, Berlin, 1998. 6. C.J.C. Burges, “Simplified Support Vector Decision Rules,” Proc. 13th Int’l Conf. Machine Learning, Morgan Kaufmann, San Francisco, 1996, pp. 71–77. 7. A. Smola and B. Schölkopf, “From Regu￾larization Operators to Support Vector Ker￾nels,” Advances in Neural Information Pro￾cessing Systems 10, M. Jordan, M. Kearns, and S. Solla, eds., MIT Press, 1998. 8. F. Girosi, An Equivalence between Sparse Approximation and Support Vector Machines, AI Memo No. 1606, MIT, Cambridge, Mass., 1997. 9. J. Weston et al., Density Estimation Using Support Vector Machines, Tech. Report CSD-TR-97-23, Royal Holloway, Univ. of London, 1997. Using SVMs for text categorization Susan Dumais, Decision Theory and Adap￾tive Systems Group, Microsoft Research As the volume of electronic information increases, there is growing interest in devel￾oping tools to help people better find, filter, and manage these resources. Text categoriza￾tion—the assignment of natural-language texts to one or more predefined categories based on their content—is an important com￾ponent in many information organization and management tasks. Machine-learning methods, including SVMs, have tremendous potential for helping people more effectively organize electronic resources. Today, most text categorization is done by people. We all save hundreds of files, e-mail messages, and URLs in folders every day. We are often asked to choose keywords from an approved set of indexing terms for describing our technical publications. On a much larger scale, trained specialists assign new items to categories in large taxonomies such as the Dewey Decimal or Library of Congress subject headings, Medical Subject Headings (MeSH), or Yahoo!’s Internet di￾rectory. Between these two extremes, people organize objects into categories to support a wide variety of information-management tasks, including information routing/filter￾ing/push, identification of objectionable materials or junk mail, structured search and browsing, and topic identification for topic￾specific processing operations. Human categorization is very time-con￾suming and costly, thus limiting its applica￾bility—especially for large or rapidly changing collections. Consequently, inter￾est is growing in developing technologies for (semi)automatic text categorization. Rule-based approaches similar to those employed in expert systems have been used, but they generally require manual construc￾tion of the rules, make rigid binary deci￾sions about category membership, and are typically difficult to modify. Another strat￾egy is to use inductive-learning techniques to automatically construct classifiers using labeled training data. Researchers have ap￾plied a growing number of learning tech￾niques to text categorization, including multivariate regression, nearest-neighbor classifiers, probabilistic Bayesian models, decision trees, and neural networks.1,2 Re￾cently, my colleagues and I and others have used SVMs for text categorization with very promising results.3,4 In this essay, I briefly describe the results of experiments in which we use SVMs to classify newswire stories from Reuters.4 Susan T. Dumais is a senior researcher in the Decision Theory and Adap￾tive Systems Group at Microsoft Research. Her research interests include algorithms and interfaces for improved information retrieval and classifica￾tion, human-computer interaction, combining search and navigation, user modeling, individual differences, collaborative filtering, and organizational impacts of new technology. She received a BA in mathematics and psychol￾ogy from Bates College and a PhD in cognitive psychology from Indiana University. She is a member of the ACM, the ASIS, the Human Factors and Ergonomic Society, and the Psychonomic Society. Contact her at Microsoft Research, 1 Microsoft Way, Redmond, WA 98052; sdumais@microsoft. com; http://research.microsoft.com/~sdumais. Edgar Osuna has just returned to his native Venezuela after receiving his PhD in operations research from the Massachusetts Institute of Technology. His research interests include the study of different aspects and properties of Vapnik’s SVM. He received his BS in computer engineering from the Universidad Simon Bolivar, Caracas, Venezuela. Contact him at IESA, POBA International #646, PO Box 02-5255, Miami, FL 33102-5255; eosuna@usb.ve. John Platt is a senior researcher in the Signal Processing Group at Micro￾soft Research. Recently, he has concentrated on fast general-purpose ma￾chine-learning algorithms for use in processing signals. More generally, his research interests include neural networks, machine learning, computer vision, and signal processing. He received his PhD in computer science at Caltech in 1989, where he studied computer graphics and neural networks. His received his BS in chemistry from California State University at Long Beach. Contact him at Microsoft Research, 1 Microsoft Way, Redmond, WA 98005; jplatt@microsoft.com; http://research.microsoft.com/~jplatt. Bernhard Schölkopf is a researcher at GMD First, Berlin. His research interests are in machine learning and perception, in particular using SVMs and Kernel PCA. He has an MSc in mathematics from the University of London, and a Diploma in physics and a PhD in computer science, both from the Max Planck Institute. Contact him at GMD-First, Rm. 208, Rudower Chaussee 5, D-12489 Berlin; bs@first.gmd.de; http://www.first. gmd.de/~bs.