《电子商务 E-business》阅读文献：Expertise recommender a flexible recommendation system and architecture

Expertise Recommender: A Flexible Recommendation System and Architecture David w. McDonald and mark s. Ackerman Department of Information and Computer Science University of California, Irvine Irvine CA 92697-3425 ddmcdonal, ackerman @ics. uci. edu ABSTRACT Recommendation systems are one possible technology that Locating the expertise necessary to solve difficult problems can augment and assist the natural expertise locating behav- a nuanced social and collaborative problem. In organiza- ior in organizations. A recommendation system that su tions, some people assist others in locating expertise by gests people who have some expertise with a problem holds making referrals. People who make referrals fill key organ the promise to provide, in a very small way, a service simi izational roles that have been identified by CSCW and af- lar to that provided by key people. Expertise recommenda filiated research. Expertise locating systems are not de- tion systems can reduce the load on people in these roles signed to replace people who fill these key organizational and provide alternative recommendations when these peo- roles. Instead, expertise locating systems attempt to de crease workload and support people who have no other Th resents an options. Recommendation systems are collaborative soft the Expertise Recommender system(ER). ER offers several ware that can be applied to expertise locating. This work advances over previously existing systems describes a general recommendation architecture that is grounded in a field study of expertise locating. Our exper- The architecture is open and flexible enough to address tise recommendation system details the work necessary to different organizational environments. If CSCW sys fit expertise recommendation to a work setting. The archi tems, and specifically recommender systems, are si tecture and implementation begin to tease apart the techni ated in their activity, organizationally specific compo cal aspects of providing good recommendations from social nents will be necessary within any general-purpose nd collaborative concerns framework. Recommender systems, to date, have been “ one size fits all Keywords Recommendation systems, collaborative filtering, expert .Organizationally specific implementations are more locators, expertise location, expertise finding, information robust by teasing apart technical aspects of making seeking, CSCW, computer-supported cooperative work. good recommendations from the social and collabora tive aspects of matching individuals. We do this based NTRODUCTION veryday, people face difficult problems that they cannot on findings from an earlier field study solve alone. In these situations the right people are the ones The work proposes an alternative approach to the crea who have the expertise to answer a specific question or in ion and maintenance of user profiles than ratings. This some other way move the problem toward resolution approach uses organizationally relevant data sources to In organizational settings, people fill key roles that assist in create profiles that are more suited for automated solving problems or initiating important collaborations pertise location. Managers, senior employees, gurus, information mediators Each of these claims will be covered below. [4], and expertise concierges [13] are sought because of We begin this paper with an overview of prior systems that their ability to solve problems directly or make well in- support expertise locating. Next, we consider what it means ned referrals. The ability to fill this role makes these to seek expertise socially, in order to firmly ground the individuals valuable to the organization and makes it ap- requirements of our system. We follow with a description parent when they are unavailable of our system, Expertise Recommender(Er). A small sce nario demonstrates how a user interacts with our system. Our last sections include the details of the architecture and Permission to make digital or hard are not made or distributed for profit or commercial advantage and that EXPERTISE LOCATING SYSTEMS AND CSCW copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers CSCW and adjacent literatures provide examples of sys ion and/or a fee tems that help people find others with suitable expertise. CSCW00, December 2-6, 2000, Philadelphia, PA Systems that assist with expertise location are similar to a Copyright2000ACM1-58113-222000012S5.00. broad class of systems known as recommendation systems

Expertise Recommender: A Flexible Recommendation System and Architecture David W. McDonald and Mark S. Ackerman Department of Information and Computer Science University of California, Irvine Irvine, CA 92697-3425 {dmcdonal, ackerman}@ics.uci.edu ABSTRACT Locating the expertise necessary to solve difficult problems is a nuanced social and collaborative problem. In organiza￾tions, some people assist others in locating expertise by making referrals. People who make referrals fill key organ￾izational roles that have been identified by CSCW and af￾filiated research. Expertise locating systems are not de￾signed to replace people who fill these key organizational roles. Instead, expertise locating systems attempt to de￾crease workload and support people who have no other options. Recommendation systems are collaborative soft￾ware that can be applied to expertise locating. This work describes a general recommendation architecture that is grounded in a field study of expertise locating. Our exper￾tise recommendation system details the work necessary to fit expertise recommendation to a work setting. The archi￾tecture and implementation begin to tease apart the techni￾cal aspects of providing good recommendations from social and collaborative concerns. Keywords Recommendation systems, collaborative filtering, expert locators, expertise location, expertise finding, information seeking, CSCW, computer-supported cooperative work. INTRODUCTION Everyday, people face difficult problems that they cannot solve alone. In these situations the right people are the ones who have the expertise to answer a specific question or in some other way move the problem toward resolution. In organizational settings, people fill key roles that assist in solving problems or initiating important collaborations. Managers, senior employees, gurus, information mediators [4], and expertise concierges [13] are sought because of their ability to solve problems directly or make well in￾formed referrals. The ability to fill this role makes these individuals valuable to the organization and makes it ap￾parent when they are unavailable. Recommendation systems are one possible technology that can augment and assist the natural expertise locating behav￾ior in organizations. A recommendation system that sug￾gests people who have some expertise with a problem holds the promise to provide, in a very small way, a service simi￾lar to that provided by key people. Expertise recommenda￾tion systems can reduce the load on people in these roles and provide alternative recommendations when these peo￾ple are unavailable. This work presents an architecture and implementation of the Expertise Recommender system (ER). ER offers several advances over previously existing systems: • The architecture is open and flexible enough to address different organizational environments. If CSCW sys￾tems, and specifically recommender systems, are situ￾ated in their activity, organizationally specific compo￾nents will be necessary within any general-purpose framework. Recommender systems, to date, have been “one size fits all”. • Organizationally specific implementations are more robust by teasing apart technical aspects of making good recommendations from the social and collabora￾tive aspects of matching individuals. We do this based on findings from an earlier field study. • The work proposes an alternative approach to the crea￾tion and maintenance of user profiles than ratings. This approach uses organizationally relevant data sources to create profiles that are more suited for automated ex￾pertise location. Each of these claims will be covered below. We begin this paper with an overview of prior systems that support expertise locating. Next, we consider what it means to seek expertise socially, in order to firmly ground the requirements of our system. We follow with a description of our system, Expertise Recommender (ER). A small sce￾nario demonstrates how a user interacts with our system. Our last sections include the details of the architecture and of an implementation. EXPERTISE LOCATING SYSTEMS AND CSCW CSCW and adjacent literatures provide examples of sys￾tems that help people find others with suitable expertise. Systems that assist with expertise location are similar to a broad class of systems known as recommendation systems. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CSCW’00, December 2-6, 2000, Philadelphia, PA. Copyright 2000 ACM 1-58113-222-0/00/0012…$5.00. 231

Recommendation systems are commonly used to recom- rithms, implement a single collaborative model that can be nend documents based on user preferences [17]. This defi- described as Birds of a Feather Flock Together(BOF) nition distinguishes recommendation systems from systems These systems tightly couple the architecture and the col that are more properly characterized as information re- laborative model. trieval (IR)or information filtering In the case of PHOAKS, Referral Web, and active collabo- There are a wide variety of recommender systems. In that rative filtering [12], different algorithms and different col context, the term"documents"should be broadly con- laborative models were chosen because the BOF model and strued. Recommendation systems have been used to rec- clustering algorithms did not effectively fit the recommen- ommend a variety of things including Usenet news [11], dation situation. In expertise recommendation, a BOF web pages [2, 91, videos [8], movies, music [18] and books. model that clusters profiles may not effectively distinguish Some systems refer a user to other people, which is essen- individuals who have expertise from those who have only a tially making recommendations about people Who Knows small amount of knowledge. Alternatively, a BOF model and Referral Web are typical examples of systems that that was effective at clustering the profiles of individuals make referrals. Who Knows [6, 20] found people whe who have expertise would likely not be effective at distin- knew something about a topic, based on a profile con- guishing in which topics their expertise lay structed from exemplary work documents. In a sense, This previous work suggests two critical improvements documents were seen as surrogates for people's interests in First, any expertise recommendation solution should sup a twist on standard IR models. On the other hand, Referral port a range of collaborative recommendation models and Web [10] used social networks to assist expertise location. behaviors. Below, we will show the social requirements for Referral Web identified relevant individuals by their par- this, but technically, an open architecture would permit a icipation in co-authoring relationships and presented users range of potential solutions. Second, one of the most diffi with a chain of relationships that needed to be traversed cult and perhaps most interesting problems from a CSCW from the person seeking expertise to the person who might perspective is how to begin untying the technical aspects of have the desired knowledge. Yenta [5] is a hybrid of these making recommendations from the social and collaborative two models, using profiles constructed from personal data aspects of making a recommendation to a specific user. s well as routing messages along a network of inferred Current recommendation approaches, especially BOF ap- shared interests proaches, do not provide this flexi In our view, recommendation systems are not completely The architecture that we will present is capable of support synonymous with collaborative filtering (CF) systems. ing a range of collaborative recommendation models. An Many CF systems rely on explicit statements of user opin- organizationally specific implementation will demonstrate ion, such as ratings, to create user profiles By relying on the flexibility in the architecture ratings, CF systems often have difficulty generating the However befor ore turning to the architecture and the subse initial user profile and, as the profiles develop, must deal quent implementation we must describe what people want with a sparseness of ratings relative to the total number of when they seek other people for their expertise. We need to items. These are two active areas of cf research show how the recommendation of expertise is in the nu- A few successful recommendation systems rely on implicit anced details of collaborative interaction. These details are opinions, rather than explicit ratings. Tapestry [7] the basis for our approach to the technical work. GroupLens [16] and PHOAKs [9, 21] have all used indi- rect activity. Tapestry made recommendations based on Our prior work [13] describes a field study to examine ex- who read or responded to a particular bulletin board mes- pertise location. The field study was motivated by two de sage. One version of Group Lens used time-spent-reading a frequency-of-mention in a stream of discussion as a type of ing of the intricacies of expertise and expertise locating in a voting mechanism for web pages. Explicit rating schemes rich work setting. Secondly, we hoped to learn how exper are unlikely to work for expertise recommendation, because augment the natural expertise locating behavior that people people are reticent to explicitly state opinions of co-workers with out an appropriate context. Ratings may not sufficiently informative or have not always been ori- not work, but it might be possible to collect feedback. ented toward the objective of informing a system architec- CF systems have another weakness for expertise recom- ture and implementation. In short, we hoped to firmly situ- mendation. Many recommendation systems have very spe- ate systems development in a complex nuanced social set cific architectures that are tailored to recommend their spe. ting as well as implement something useful to a group of cific type of artifact. As well, these architectures commonly participants implement a single clustering algorithm(e. g. Pearson Cor- relation, Single-Value Decomposition, Bayesian classifier The following briefly recounts our field study and the re- for all participants and all artifacts. These clustering algo- sults [13] rithms, also called neighborhood-based predictive algo- 232

Recommendation systems are commonly used to recom￾mend documents based on user preferences [17]. This defi￾nition distinguishes recommendation systems from systems that are more properly characterized as information re￾trieval (IR) or information filtering. There are a wide variety of recommender systems. In that context, the term “documents” should be broadly con￾strued. Recommendation systems have been used to rec￾ommend a variety of things including Usenet news [11], web pages [2, 9], videos [8], movies, music [18] and books. Some systems refer a user to other people, which is essen￾tially making recommendations about people. Who Knows and Referral Web are typical examples of systems that make referrals. Who Knows [6, 20] found people who knew something about a topic, based on a profile con￾structed from exemplary work documents. In a sense, documents were seen as surrogates for people’s interests in a twist on standard IR models. On the other hand, Referral Web [10] used social networks to assist expertise location. Referral Web identified relevant individuals by their par￾ticipation in co-authoring relationships and presented users with a chain of relationships that needed to be traversed from the person seeking expertise to the person who might have the desired knowledge. Yenta [5] is a hybrid of these two models, using profiles constructed from personal data as well as routing messages along a network of inferred shared interests. In our view, recommendation systems are not completely synonymous with collaborative filtering (CF) systems. Many CF systems rely on explicit statements of user opin￾ion, such as ratings, to create user profiles. By relying on ratings, CF systems often have difficulty generating the initial user profile and, as the profiles develop, must deal with a sparseness of ratings relative to the total number of items. These are two active areas of CF research. A few successful recommendation systems rely on implicit opinions, rather than explicit ratings. Tapestry [7], GroupLens [16] and PHOAKS [9, 21] have all used indi￾rect activity. Tapestry made recommendations based on who read or responded to a particular bulletin board mes￾sage. One version of GroupLens used time-spent-reading a message as an implicit opinion. And PHOAKS relies on frequency-of-mention in a stream of discussion as a type of voting mechanism for web pages. Explicit rating schemes are unlikely to work for expertise recommendation, because people are reticent to explicitly state opinions of co-workers with out an appropriate context. Ratings may not work, but it might be possible to collect feedback. CF systems have another weakness for expertise recom￾mendation. Many recommendation systems have very spe￾cific architectures that are tailored to recommend their spe￾cific type of artifact. As well, these architectures commonly implement a single clustering algorithm (e.g. Pearson Cor￾relation, Single-Value Decomposition, Bayesian classifier) for all participants and all artifacts. These clustering algo￾rithms, also called neighborhood-based predictive algo￾rithms, implement a single collaborative model that can be described as Birds of a Feather Flock Together (BOF). These systems tightly couple the architecture and the col￾laborative model. In the case of PHOAKS, Referral Web, and active collabo￾rative filtering [12], different algorithms and different col￾laborative models were chosen because the BOF model and clustering algorithms did not effectively fit the recommen￾dation situation. In expertise recommendation, a BOF model that clusters profiles may not effectively distinguish individuals who have expertise from those who have only a small amount of knowledge. Alternatively, a BOF model that was effective at clustering the profiles of individuals who have expertise would likely not be effective at distin￾guishing in which topics their expertise lay. This previous work suggests two critical improvements. First, any expertise recommendation solution should sup￾port a range of collaborative recommendation models and behaviors. Below, we will show the social requirements for this, but technically, an open architecture would permit a range of potential solutions. Second, one of the most diffi￾cult and perhaps most interesting problems from a CSCW perspective is how to begin untying the technical aspects of making recommendations from the social and collaborative aspects of making a recommendation to a specific user. Current recommendation approaches, especially BOF ap￾proaches, do not provide this flexibility. The architecture that we will present is capable of support￾ing a range of collaborative recommendation models. An organizationally specific implementation will demonstrate the flexibility in the architecture. However, before turning to the architecture and the subse￾quent implementation we must describe what people want when they seek other people for their expertise. We need to show how the recommendation of expertise is in the nu￾anced details of collaborative interaction. These details are the basis for our approach to the technical work. GROUNDED REQUIREMENTS Our prior work [13] describes a field study to examine ex￾pertise location. The field study was motivated by two de￾sires. First, we hoped to contribute to a better understand￾ing of the intricacies of expertise and expertise locating in a rich work setting. Secondly, we hoped to learn how exper￾tise locating systems could be improved to better assist and augment the natural expertise locating behavior that people exhibit. The issues and findings raised in the literature were not sufficiently informative or have not always been ori￾ented toward the objective of informing a system architec￾ture and implementation. In short, we hoped to firmly situ￾ate systems development in a complex nuanced social set￾ting as well as implement something useful to a group of participants. The following briefly recounts our field study and the re￾sults [13]. 232

The MSC Field Study status and can then constrain who is pursued for expertise. The field site was a medium sized software company that Lastly, Orr's [ 14] work with repair technicians reveals how builds, sells, and supports medical and dental practice man- narrative exchange(war stories)serves to demonstrate ex agement software. Medical Software Corporation(MSC) pertise and notions of competency which then guide re has about 100 employees at its headquarters where the quests for assistance with difficult problems study took place. The participants in the study worked in Escalation is the mechanism that fixes breakdowns in iden- Communications. These three departments are central to the development and support of MSC's products. These identification and selection because it requires fine judge ments often based on incomplete and imperfect informa- departments comprise a little more than one third of the tion. Escalation is an iteration of identification and selec- employees at headquarters tion given additional knowledge and information gained The following findings are based on an initial 5 months in from prior iterations. During escalation the individual who the field. Data was collected using qualitative methods in- initially engaged the problem maintains problem owner cluding participant observation, semi-structured formal ship. Escalation results in a wider, spreading activation interviews and informal interviews. The data was analyzed through the organization that involves more people in the sing standard ethnographic techniques eventual solution to a difficult problem. Our study found that people at MSC commonly engage in The architecture that we describe later in this paper pro- three behaviors that support expertise locating: expertise vides specific support for this model of expertise location identification, expertise selection, and escalation. We note An implementation of this architecture, however, requires that identification, selection, and escalation are analytical specifics of how people perform these behaviors. The next distinctions useful for understanding the fine social and section describes two specific identification heuristics that cognitive decisions which are made during expertise locat were used by participants at MSC ing and that will later be useful for technical design. It is Expertise Locating Heuristics clear that only the most self-reflective of the participants The MSC field study provides a detailed understanding of made much if any distinction among these three behaviors the following two identification heuristics. We found this Expertise identification is the problem of finding a set of understanding essential to building a nuanced system. candidates who are likely to have the desired expertise. These heuristics, Change History and Tech Support, are Expertise identification relies on prior experience with oth behaviors and rules about interpreting implied meaning ers, key people like an expertise concierge, and historica using specific historical artifacts at MSC when identifying artifacts. The expertise concierge routes people to others potential expertise at MSC. These historical artifacts have with the necessary knowledge and expertise. The role has many, potentially crucial, features, but the participants at much in common with Ehrlich and Cash's information me- tend to only a few through specific rules of thumb. The diator [4],Paepke's contact broker [15], and Allen's tech- details are quite important to the behavior of the partici ological gatekeeper [l]. Each of these roles is specialized pants and the eventual behavior of a system to the organizational context. Historical artifacts are prod ucts and byproducts of the work that other MSC employees Change History Heuristic have produced. The composition and form of historical Change History is a single heuristic designed to augment an artifacts range from unstructured to highly structured things expertise locating behavior common to MSC's developers that can exist on-line or off-line Expertise selection is the way participants picked one per The"Line 10 Rule"is a function of the organization that son (or a small number of people)to approach for help MSC and the work practices that are enforced there. The Expertise selection is guided by organizational criteria, the developers follow a common work practice. For each soft ad on the potential source, and how a source reacts and ware change, they check the appropriate module out of the interacts during a request for help. Social, organizational, version control system, make the changes, test the and individual preference assist in narrowing a set of iden and then check the changes back into the version tified candidates to one or a small number of people who system. Each change is annotated with the module name, can be contacted the module version, the developer responsible for the ge, the check-in date, and In some cases, prior research has intertwined elements of the change. Since many changes are the result of a contrac expertise identification and selection. Allens[l] work with tual obligation, an administrative assistant cross-validates engineers demonstrated that information seeking is heavily the changes in the version control system, the work-orders or uenced by social networks. Cicourel [3] observed that that divide up multiple changes, the time spent on changes ganizational and institutional factors reinforce expertise and the total time and accounts for the contract Developers at MsC do not specialize in any part of the sys Names and individual identifiers have been changed to tem. In practice, however, there is some attempt to assign a protect the privacy of the corporation and the participants developer work in an area of the code where she ha

The MSC Field Study The field site was a medium sized software company that builds, sells, and supports medical and dental practice man￾agement software. Medical Software Corporation1 (MSC) has about 100 employees at its headquarters where the study took place. The participants in the study worked in Technical Development, Technical Support and Technical Communications. These three departments are central to the development and support of MSC’s products. These departments comprise a little more than one third of the employees at headquarters. The following findings are based on an initial 5 months in the field. Data was collected using qualitative methods in￾cluding participant observation, semi-structured formal interviews and informal interviews. The data was analyzed using standard ethnographic techniques. Our study found that people at MSC commonly engage in three behaviors that support expertise locating: expertise identification, expertise selection, and escalation. We note that identification, selection, and escalation are analytical distinctions useful for understanding the fine social and cognitive decisions which are made during expertise locat￾ing and that will later be useful for technical design. It is clear that only the most self-reflective of the participants made much if any distinction among these three behaviors. Expertise identification is the problem of finding a set of candidates who are likely to have the desired expertise. Expertise identification relies on prior experience with oth￾ers, key people like an expertise concierge, and historical artifacts. The expertise concierge routes people to others with the necessary knowledge and expertise. The role has much in common with Ehrlich and Cash’s information me￾diator [4], Paepke’s contact broker [15], and Allen’s tech￾nological gatekeeper [1]. Each of these roles is specialized to the organizational context. Historical artifacts are prod￾ucts and byproducts of the work that other MSC employees have produced. The composition and form of historical artifacts range from unstructured to highly structured things that can exist on-line or off-line. Expertise selection is the way participants picked one per￾son (or a small number of people) to approach for help. Expertise selection is guided by organizational criteria, the load on the potential source, and how a source reacts and interacts during a request for help. Social, organizational, and individual preference assist in narrowing a set of iden￾tified candidates to one or a small number of people who can be contacted. In some cases, prior research has intertwined elements of expertise identification and selection. Allen’s [1] work with engineers demonstrated that information seeking is heavily influenced by social networks. Cicourel [3] observed that organizational and institutional factors reinforce expertise 1 Names and individual identifiers have been changed to protect the privacy of the corporation and the participants. status and can then constrain who is pursued for expertise. Lastly, Orr’s [14] work with repair technicians reveals how narrative exchange (war stories) serves to demonstrate ex￾pertise and notions of competency which then guide re￾quests for assistance with difficult problems. Escalation is the mechanism that fixes breakdowns in iden￾tification and selection. Participants had difficulty with identification and selection because it requires fine judge￾ments often based on incomplete and imperfect informa￾tion. Escalation is an iteration of identification and selec￾tion given additional knowledge and information gained from prior iterations. During escalation the individual who initially engaged the problem maintains problem owner￾ship. Escalation results in a wider, spreading activation through the organization that involves more people in the eventual solution to a difficult problem. The architecture that we describe later in this paper pro￾vides specific support for this model of expertise location. An implementation of this architecture, however, requires specifics of how people perform these behaviors. The next section describes two specific identification heuristics that were used by participants at MSC. Expertise Locating Heuristics The MSC field study provides a detailed understanding of the following two identification heuristics. We found this understanding essential to building a nuanced system. These heuristics, Change History and Tech Support, are behaviors and rules about interpreting implied meaning using specific historical artifacts at MSC when identifying potential expertise at MSC. These historical artifacts have many, potentially crucial, features, but the participants at￾tend to only a few through specific rules of thumb. The details are quite important to the behavior of the partici￾pants and the eventual behavior of a system. Change History Heuristic Change History is a single heuristic designed to augment an expertise locating behavior common to MSC’s developers. Developers called this behavior the “Line 10 Rule.” The “Line 10 Rule” is a function of the organization that is MSC and the work practices that are enforced there. The developers follow a common work practice. For each soft￾ware change, they check the appropriate module out of the version control system, make the changes, test the changes, and then check the changes back into the version control system. Each change is annotated with the module name, the module version, the developer responsible for the change, the check-in date, and a short text description of the change. Since many changes are the result of a contrac￾tual obligation, an administrative assistant cross-validates the changes in the version control system, the work-orders that divide up multiple changes, the time spent on changes and the total time and accounts for the contract. Developers at MSC do not specialize in any part of the sys￾tem. In practice, however, there is some attempt to assign a developer work in an area of the code where she has 233

worked in the past. Yet, the expanse of the system, the Detailed heuristics and social interaction guide an ex number of developers, and developer turnover will result in pertise seeker to identify candidates who are likely to a developer being assigned work in portions of the code have the required expertise where she has not worked before. Developers use the"Line 10 Rule"in an attempt to overcome the problems that arise Social and organizational norms and factors guide an when working with unfamiliar code expertise seeker to pick a small number of candidates to pursue for help On the surface the"Line 10 Rule"is a simple heuristic Given a problem with a module a developer looks into the The details matter. The heuristics described here are version control system to see who last modified the code extremely bound to the organizational environment. nd then approaches that person for help. Developers say Systems that augment expertise locating must be capa that this rule works because the person who last made a ble of handling a large number of details that vary hange has the code"freshest "in mind The version control based on the specific context and problem. system is not specifically designed to support this exact The focus of this paper now shifts from the details of the use. As a result, the rule is not strictly followed, but most requirements(as exemplified by MSC) to the technical developers state that the heuristic is"good enough"to often design of the expertise recommendation system. THE EXPERTISE RECOMMENDER (ER) Tech Support Heuristic The Expertise Recommender(Er) is firmly grounded in The tech support identification heuristic is not known by a the findings from the MSC study. Er is really two things at specific name. The heuristic augments a behavior of tech- once, at two different levels of abstraction. At the most nical support representatives when faced with an unfamiliar abstract, ER is an architecture designed to handle a range of difficult support problem. recommendation problems. However, recommendation The heuristic is applied to the support database. The sup- architectures cannot solve specific recommendation prob- port database mediates all work performed by technical lems at specific organizations; only specific implemer support representatives. New problems ("calls")can be tions of an architecture can do so. Therefore, we also show entered by a support rep or by customers via email how our work addresses this issue by implementing a spe- When faced with a difficult problem, a representative will cific instantiation suitable for the expertise locating beha perform multiple, separate queries over the support data ior found in the field study of MSC base using the symptoms, customer, or program module To avoid confusion, for the remainder of this paper the ar- volved. The support rep then scans the records sequen- chitecture will be referred to as ER-Arch and the imple tially looking for similarities between the current problem mentation will be known simply as er and any past problems as returned by the different queries This section first presents a simple usage scenario that pro- In scanning records a support rep looks to identify people vides an overview of a client and how the user interacts who have previously solved similar problems with the system. We follow the scenario with a description The support database is not designed to enable this type of of ER-Arch, and conclude with details of the Msc imple activity. Each query(symptom, customer or program)must mentation of ER. be completed separately, so finding similarities among the A Brief Usage Scenario three primary characteristics is mostly done in the support This scenario provides a picture of the user interaction with representative's head. At MSC support representatives are the system and several key features of an ER client. The assigned to specific customers. This benefits the customer scenario describes a problem that MSC support representa- because a single support representative can remember more tives might encounter in the course of their day-to-day details about the context of a customers installation. A side work with msc customers effect is that, while the database does not facilitate queries e re Madhu is a junior technical support representative. Support tween an MSC customer and the support rep who worl reps like Madhu are assigned to handle all incoming call with them is relatively simple from a specific set of MSC customers. However, today The tech support identification heuristic consists of attach- tion. Madhu receives a call from a customer. PCL. who g the symptoms, customers, and program modules solved problems to the person who solved them. As a sim states that the system is reporting a file error in patient ple rule of thumb this scanning behavior works fairly well demographics. Even as a relatively new support 1 is fairly knowledgeable about patient demographics. How- profitable when applied to more difficult support problernly ever. Madhu is not familiar with the specific customer, PCL, However, the process can be time consuming and is and why that customers system might be reporting an er In summary, prior work and our field work demonstrates ror. tant requirements of expertise locating: Madhu recognizes that he will need help resolving this problem. He launches the ER client, logs into the system and gets the main window(figure 1). The main window

worked in the past. Yet, the expanse of the system, the number of developers, and developer turnover will result in a developer being assigned work in portions of the code where she has not worked before. Developers use the “Line 10 Rule” in an attempt to overcome the problems that arise when working with unfamiliar code. On the surface the “Line 10 Rule” is a simple heuristic. Given a problem with a module a developer looks into the version control system to see who last modified the code and then approaches that person for help. Developers say that this rule works because the person who last made a change has the code “freshest” in mind. The version control system is not specifically designed to support this exact use. As a result, the rule is not strictly followed, but most developers state that the heuristic is “good enough” to often get the help they need. Tech Support Heuristic The tech support identification heuristic is not known by a specific name. The heuristic augments a behavior of tech￾nical support representatives when faced with an unfamiliar or difficult support problem. The heuristic is applied to the support database. The sup￾port database mediates all work performed by technical support representatives. New problems (“calls”) can be entered by a support rep or by customers via email. When faced with a difficult problem, a representative will perform multiple, separate queries over the support data￾base using the symptoms, customer, or program module involved. The support rep then scans the records sequen￾tially looking for similarities between the current problem and any past problems as returned by the different queries. In scanning records a support rep looks to identify people who have previously solved similar problems. The support database is not designed to enable this type of activity. Each query (symptom, customer or program) must be completed separately, so finding similarities among the three primary characteristics is mostly done in the support representative’s head. At MSC support representatives are assigned to specific customers. This benefits the customer because a single support representative can remember more details about the context of a customer’s installation. A side effect is that, while the database does not facilitate queries across the three indices, establishing the relationship be￾tween an MSC customer and the support rep who works with them is relatively simple. The tech support identification heuristic consists of attach￾ing the symptoms, customers, and program modules of solved problems to the person who solved them. As a sim￾ple rule of thumb this scanning behavior works fairly well. However, the process can be time consuming and is only profitable when applied to more difficult support problems. In summary, prior work and our field work demonstrates several important requirements of expertise locating: • Detailed heuristics and social interaction guide an ex￾pertise seeker to identify candidates who are likely to have the required expertise. • Social and organizational norms and factors guide an expertise seeker to pick a small number of candidates to pursue for help. • The details matter. The heuristics described here are extremely bound to the organizational environment. Systems that augment expertise locating must be capa￾ble of handling a large number of details that vary based on the specific context and problem. The focus of this paper now shifts from the details of the requirements (as exemplified by MSC) to the technical design of the expertise recommendation system. THE EXPERTISE RECOMMENDER (ER) The Expertise Recommender (ER) is firmly grounded in the findings from the MSC study. ER is really two things at once, at two different levels of abstraction. At the most abstract, ER is an architecture designed to handle a range of recommendation problems. However, recommendation architectures cannot solve specific recommendation prob￾lems at specific organizations; only specific implementa￾tions of an architecture can do so. Therefore, we also show how our work addresses this issue by implementing a spe￾cific instantiation suitable for the expertise locating behav￾ior found in the field study of MSC. To avoid confusion, for the remainder of this paper the ar￾chitecture will be referred to as ER-Arch and the imple￾mentation will be known simply as ER. This section first presents a simple usage scenario that pro￾vides an overview of a client and how the user interacts with the system. We follow the scenario with a description of ER-Arch, and conclude with details of the MSC imple￾mentation of ER. A Brief Usage Scenario This scenario provides a picture of the user interaction with the system and several key features of an ER client. The scenario describes a problem that MSC support representa￾tives might encounter in the course of their day-to-day work with MSC customers. Madhu is a junior technical support representative. Support reps like Madhu are assigned to handle all incoming calls from a specific set of MSC customers. However, today Madhu is also covering for a support rep who is on vaca￾tion. Madhu receives a call from a customer, PCI, who states that the system is reporting a file error in patient demographics. Even as a relatively new support rep, Madhu is fairly knowledgeable about patient demographics. How￾ever, Madhu is not familiar with the specific customer, PCI, and why that customer’s system might be reporting an er￾ror. Madhu recognizes that he will need help resolving this problem. He launches the ER client, logs into the system and gets the main window (figure 1). The main window 234

contains a menu bar and a list of recent prior requests that dress. Additionally, each pane contains a""Contact "button Madhu made. Prior requests are live elements(save sets) that is active only when the recommended person is logged that can be revisited. When revisiting a prior req quest, the into the er server. The contact button, when active estab- user can review the people who were recommended or es- lishes a synchronous chat with the recommended person. calate the request to gain additional recommendations File Edit Special Help Topic Area: Tech support Request: Io Error 16 in pregram M013 customer PCI social Networ rk Reporting Labels MOS5 M054 Departme (949)82 change History Social Network insurance file maintenance m086 m08 81363 suite 101 x1255 klashneresupportmsc.com escalate Close Figure 1. Expertise Recommender Main Window Figure 3. Recommendation Response(Escalated Madhu has not made any prior requests for this problem, so Madhu s escalated request returns two additional people he will have to make a new request. He chooses"New Re- one from development and one from training. Madhu rec quest” from the“File” menu and the client displays the expertise request dialog(figure 2). Through conversation ognizes that Er has recommended a friend in development with the person reporting the error, Madhu is able to get the as someone who is likely to have expertise. Madhu walks specific error code and determines that the error is gener- over to the development suite to look for his friend ated in a module of the patient demographics system called This brief overview of a user's interaction with ER demon- M013 strates several key aspects of Er: EDts吧R就 The user can pick the heuristic and the associated data Expertise Request: that will Tech support Filter: social Network rror 16 in program M 013 cus The user can choose the mechanism that tailors the Cancel Recommend Recommendations remain live so that they can be Figure 2. Expertise Request Dialog tickly revisited and escalated should the initial Madhu decides that the problem is most closely associated ommendations prove unfruitful. with the"Tech Support"topic area and picks this topic in Moving on from the scenario, we next turn to a description the new request dialog. Madhu chooses the"Social Net- of the ER-Arch and the implementation details of an ER ork filter because he would like the results filtered based system for MSC on the people whom he knows best. The dialog lists the The architecture- ER-Arch available identification heuristics and selection techniques Like many recommendation systems, ER-Arch is server in the "Topic Area"and"Filter"pop-up menus respec- based, and a client is necessary to access the functionality tively. In the"Request"text field, Madhu enters the error ER-Arch can support simple clients, like a Web based in (0 Error 16), the program module (program M013)and terface, or clients tailored to support specific features of the the customer(customer PCD). (These names would be natu- ER server. Regardless of the client, the important architec- ral to a support rep at MSC )Madhu clicks the"Recom- tural details are in mend"button, sending the request to the server cated, the components and interconnections should be as After a short wait, the server responds with a list of re sumed to exist in the ommendations. Madhu quickly recognizes that the first few At a high level, ER-Arch is a pipe and filter architecture people will not work. The first recommendations include 9]. This architecture is widely used in information re- the support rep whose calls Madhu is now covering, as well trieval and information filtering, and it has great utility in as a person from training who is currently off-site. So, expertise recommendation (as will be discussed below) Madhu immediately escalates the request to get additional Accordingly, ER-Arch is a collection of high-level supervi sors, easily extensible heuristic modules, and their data The response dialog displays the request followed by a list stores. The supervisors provide general services and con- of recommended people. Each person is listed in a single nections that facilitate a specific implementation. They also ane containing contact information that includes office coordinate the underlying heuristic modules to provide re number, phone number, phone extension, and email ad- quired services( such as identification)to the system. The databases provide persistent storage for profiles and various 235

contains a menu bar and a list of recent prior requests that Madhu made. Prior requests are live elements (save sets) that can be revisited. When revisiting a prior request, the user can review the people who were recommended or es￾calate the request to gain additional recommendations. Figure 1. Expertise Recommender Main Window Madhu has not made any prior requests for this problem, so he will have to make a new request. He chooses “New Re￾quest” from the “File” menu and the client displays the expertise request dialog (figure 2). Through conversation with the person reporting the error, Madhu is able to get the specific error code and determines that the error is gener￾ated in a module of the patient demographics system called M.013. Figure 2. Expertise Request Dialog Madhu decides that the problem is most closely associated with the “Tech Support” topic area and picks this topic in the new request dialog. Madhu chooses the “Social Net￾work” filter because he would like the results filtered based on the people whom he knows best. The dialog lists the available identification heuristics and selection techniques in the “Topic Area” and “Filter” pop-up menus respec￾tively. In the “Request” text field, Madhu enters the error (I/O Error 16), the program module (program M.013) and the customer (customer PCI). (These names would be natu￾ral to a support rep at MSC.) Madhu clicks the “Recom￾mend” button, sending the request to the server. After a short wait, the server responds with a list of rec￾ommendations. Madhu quickly recognizes that the first few people will not work. The first recommendations include the support rep whose calls Madhu is now covering, as well as a person from training who is currently off-site. So, Madhu immediately escalates the request to get additional recommendations (figure 3). The response dialog displays the request followed by a list of recommended people. Each person is listed in a single pane containing contact information that includes office number, phone number, phone extension, and email ad￾dress. Additionally, each pane contains a “Contact” button that is active only when the recommended person is logged into the ER server. The contact button, when active, estab￾lishes a synchronous chat with the recommended person. Figure 3. Recommendation Response (Escalated) Madhu’s escalated request returns two additional people, one from development and one from training. Madhu rec￾ognizes that ER has recommended a friend in development as someone who is likely to have expertise. Madhu walks over to the development suite to look for his friend. This brief overview of a user’s interaction with ER demon￾strates several key aspects of ER: • The user can pick the heuristic and the associated data that will be used to generate a recommendation. • The user can choose the mechanism that tailors the recommendations to the user. • Recommendations remain live so that they can be quickly revisited and escalated should the initial rec￾ommendations prove unfruitful. Moving on from the scenario, we next turn to a description of the ER-Arch and the implementation details of an ER system for MSC. The Architecture — ER-Arch Like many recommendation systems, ER-Arch is server based, and a client is necessary to access the functionality. ER-Arch can support simple clients, like a Web based in￾terface, or clients tailored to support specific features of the ER server. Regardless of the client, the important architec￾tural details are in the ER server. Unless otherwise indi￾cated, the components and interconnections should be as￾sumed to exist in the server. At a high level, ER-Arch is a pipe and filter architecture [19]. This architecture is widely used in information re￾trieval and information filtering, and it has great utility in expertise recommendation (as will be discussed below). Accordingly, ER-Arch is a collection of high-level supervi￾sors, easily extensible heuristic modules, and their data stores. The supervisors provide general services and con￾nections that facilitate a specific implementation. They also coordinate the underlying heuristic modules to provide re￾quired services (such as identification) to the system. The databases provide persistent storage for profiles and various 235

preferences. The supervisors in ER-Arch support profiling, more than one type of thing and generate profiles from identification, selection and interaction management. An more than one single source. This is facilitated by a collec overview of ER-Arch is provided in figure 4. tion of modules supervised by the profiling supervisor. Many prior recommendation systems only support one ERClient method of profiling The profiling supervisor coordinates the profiling modules and provides access to the profile database. The profiling database stores the results of profiling and also links profil Supervisor ing to the other supervisors Identification Supervisor Identification picks a set of items or people who are rea- Supervisor sonable candidates for a recommendation. Items are picked Web browse by applying one or more heuristics and adding the identi fied candidates to a set. In the same way that profiling in ER-Arch is designed to support multiple types of profiling, ER-Arch is also designed to support multiple identification heuristics. This is different from many prior recommenda- tion systems that support only one method of picking can- Http didates. The identification supervisor coordinates the appli Server cation of each heuristic. The supervisor provides each heu ristic module with access to the profile database. Figure 4. Expertise Recommender Architectural Overview ser/Client he Er server logically glues together portions of ER-Arch. It also handles the details of managing connec- tions and servicing requests. The ER server implements a Profile d protocol that clients use to request and receive recommen- dations. The supervisors in ER-Arch support profiling, identification, selection and interaction management. The following sections discuss the ER-Arch supervisors, data- Identification Supervisor bases, and the interconnections among them in more detail Feather Profiling Supervisor ating Figure 6. Identification Supervisor Identification is initiated in response to a user request Through an eR client, the user can indicate which heuristic Profile DB she would like to apply and provide parameters that tailor the heuristic to her current needs or desires the result of Figure 5. Profiling Supervisor identification is a set of raw recommendations. Raw rec ng Supervisor ommendations are passed directly to the next stage The profiling supervisor is responsible for creating and maintaining profiles. In many recommendation systems, Selection takes a set of candidate recommendations. and profiles are a list of items which an individual has rated. then reorders and possibly removes items from the set to These ratings profiles are used in two ways. First, profiles generate a refined recommendation. The selection supervi are clustered to create groups of users who have similar sor receives the raw recommendations from identification likes and dislikes. Additionally, profiles are used to identify and applies one or more selection modules(such as organ- items that a user has not yet rated and are therefore good izational criteria). The selection supervisor in ER-Arch is candidates for recommendation designed to support many different methods of selecting In ER-Arch profiling is a periodic activity which is de- and filtering the raw recommendations signed to be run off-line relative to the other activities of The selection supervisor provides selection modules access the server. ER-Arch can be used to maintain profiles of to the preference database that maintains personal and

preferences. The supervisors in ER-Arch support profiling, identification, selection and interaction management. An overview of ER-Arch is provided in figure 4. Network Profile DB ERServer Identification Supervisor Selection Supervisor Interaction Management Prefs DB Profiling Supervisor HTTP Server ERClient Netscape: Expertise Recommender - Madhu's Recent Requests N http://www.msc.com/cgi-bin/experthome/ Madhu's Recent Requests Change History Social Network Reporting Labels M.055 M.054 Tech Support Departmental FQHC Medicare Billing Change History Social Network insurance file maintenance m.086 m.087 New Request Preferences Log Out Web Browser Figure 4. Expertise Recommender Architectural Overview The ER server logically glues together portions of ER-Arch. It also handles the details of managing connec￾tions and servicing requests. The ER server implements a protocol that clients use to request and receive recommen￾dations. The supervisors in ER-Arch support profiling, identification, selection and interaction management. The following sections discuss the ER-Arch supervisors, data￾bases, and the interconnections among them in more detail. Profile DB Explicit Ratings P 1 Hearsay P n User Behavior P 2 Implicit Activity P 3 Profiling Supervisor Figure 5. Profiling Supervisor Profiling Supervisor The profiling supervisor is responsible for creating and maintaining profiles. In many recommendation systems, profiles are a list of items which an individual has rated. These ratings profiles are used in two ways. First, profiles are clustered to create groups of users who have similar likes and dislikes. Additionally, profiles are used to identify items that a user has not yet rated and are therefore good candidates for recommendation. In ER-Arch profiling is a periodic activity which is de￾signed to be run off-line relative to the other activities of the server. ER-Arch can be used to maintain profiles of more than one type of thing and generate profiles from more than one single source. This is facilitated by a collec￾tion of modules supervised by the profiling supervisor. Many prior recommendation systems only support one method of profiling. The profiling supervisor coordinates the profiling modules and provides access to the profile database. The profiling database stores the results of profiling and also links profil￾ing to the other supervisors. Identification Supervisor Identification picks a set of items or people who are rea￾sonable candidates for a recommendation. Items are picked by applying one or more heuristics and adding the identi￾fied candidates to a set. In the same way that profiling in ER-Arch is designed to support multiple types of profiling, ER-Arch is also designed to support multiple identification heuristics. This is different from many prior recommenda￾tion systems that support only one method of picking can￾didates. The identification supervisor coordinates the appli￾cation of each heuristic. The supervisor provides each heu￾ristic module with access to the profile database. Profile DB User/Client Request Raw Recommendation Birds of a Feather H1 Line 10 Rule Hn Opinion Leader H2 Identification Supervisor Figure 6. Identification Supervisor Identification is initiated in response to a user request. Through an ER client, the user can indicate which heuristic she would like to apply and provide parameters that tailor the heuristic to her current needs or desires. The result of identification is a set of raw recommendations. Raw rec￾ommendations are passed directly to the next stage. Selection Supervisor Selection takes a set of candidate recommendations, and then reorders and possibly removes items from the set to generate a refined recommendation. The selection supervi￾sor receives the raw recommendations from identification and applies one or more selection modules (such as organ￾izational criteria). The selection supervisor in ER-Arch is designed to support many different methods of selecting and filtering the raw recommendations. The selection supervisor provides selection modules access to the preference database that maintains personal and or- 236

ganizationally relevant data(e.g, departmental affiliation) need some combination of generic and organizationally that can be used as selection criteria. The result of selection specific mechanisms for finding people. This section details is a refined recommendation. The refined recommendation the work involved in creating an implementation of Er is passed into the interaction management before becoming suitable for MSCs use For MSC, ER implements the two a final recommendation identification heuristics described above as well as three selection techniques. Perhaps more importantly, this section Recommendation JUser& Org also describes the organizationally specific data processing Prets DB required to effectively generate expertise recommendations. To implement the identification heuristics, real data from MSC's change history system and support database were required. MSC provided the most recent eight years of pro- gram change history covering their medical system and Social workload o Filter etwork portion of the shared libraries used by both their medical and dental systems. This corresponds to a total of 7316 individual code modifications. The data from the support database cover the last four years of customer support ac tivity. These data include all calls logged and marked as completed. The support data cover software and hardware issues for all of MSC's products. More than 200,000 calls were handled during the time period covered by the data Profiling The identification heuristics and profiling modules strongly related through the profile database. The identifi- Figure 7. Selection Supervisor and Interaction Management cation heuristics specify which features the profiling mod ules must extract. The intricate details of implementing Interaction Management profiling modules for MSC demonstrate how real, messy Interaction management receives the refined recommenda- data sources are transformed into profile records more tions and processes them before releasing them as a final amenable to the application of identification heuristics. recommendation. Interaction management in ER-Arch The profiling supervisor controls profiling in ER Profiling tracks the users'interaction with the system and tracks the modules are extensions to a base class that provides meth recommendations. Interaction management also collects ods for interacting with the profiling supervisor and the On interact ystem performance. The specific compo- profile database. ER supports a profiling module for each on management depend upon the specific ata source and corresponding identification heuristic. Pre needs of an organization and a given implementation filing modules incrementally update each profile record. Interaction management serves to reintegrate and smooth The change history profiling module crawls through each out discontinuities introduced by separating expertise loca change collecting the developer responsible for the change, tion into two distinct phases. In the real world, communica the module name the version and date For each change tion, escalation, issue tracking, problem context, and how the developer's profile is found and the slot containing individuals manage their accessibility to others are the set- code changes is searched for previous modifications to the ting in which expertise location resides. Interaction man- module. If the developer has no prior changes, then a new agement is the architectural component that provides these entry is made containing the module, the date of the most components, thereby serving to tie together and serve as the recent change, and the total number of changes. If the de setting for identification and selection veloper had previously made a change, then the date of last rent State of change is updated and the total count of changes is incre- R-Arch is instantiated in an implementation of ER for mented. The profiling module es dates and is careful MSC. ER includes a server and a dedicated client. Both to modify profiles only when changes are more recent server and client were coded in Java, and consist of more The profiling module for the tech support area is incre than 20,000 lines in 84 classes. The code includes numer- mental as well. However, the tech support profiling module ous base classes and abstract classes that simplify the ex- assumes that it will be fed unique records from the techni- tension of ER with new identification heuristics and new cal support database. The profiling module considers four selection tech fields key: the support representative, the description of the The MSc Implementation problem(a small amount of free form text), the customer, As mentioned, much of the promise of ER-Arch lay in its and the module that the customer reports is the problem flexibility. If expertise seeking really is situated within spe- These four fields are used to incrementally ild three vec cific organizational or social contexts, presumably we will tor spaces that characterize a specific person's activity in the support database

ganizationally relevant data (e.g., departmental affiliation) that can be used as selection criteria. The result of selection is a refined recommendation. The refined recommendation is passed into the interaction management before becoming a final recommendation. User & Org. Prefs DB Final Recommendation Interaction Management Raw Recommendation Social Network F 1 No Filter F n Workload F 2 Selection Supervisor Figure 7. Selection Supervisor and Interaction Management Interaction Management Interaction management receives the refined recommenda￾tions and processes them before releasing them as a final recommendation. Interaction management in ER-Arch tracks the users’ interaction with the system and tracks the recommendations. Interaction management also collects feedback about system performance. The specific compo￾nents of interaction management depend upon the specific needs of an organization and a given implementation Interaction management serves to reintegrate and smooth out discontinuities introduced by separating expertise loca￾tion into two distinct phases. In the real world, communica￾tion, escalation, issue tracking, problem context, and how individuals manage their accessibility to others are the set￾ting in which expertise location resides. Interaction man￾agement is the architectural component that provides these components, thereby serving to tie together and serve as the setting for identification and selection. Current State of ER ER-Arch is instantiated in an implementation of ER for MSC. ER includes a server and a dedicated client. Both server and client were coded in Java, and consist of more than 20,000 lines in 84 classes. The code includes numer￾ous base classes and abstract classes that simplify the ex￾tension of ER with new identification heuristics and new selection techniques. The MSC Implementation As mentioned, much of the promise of ER-Arch lay in its flexibility. If expertise seeking really is situated within spe￾cific organizational or social contexts, presumably we will need some combination of generic and organizationally specific mechanisms for finding people. This section details the work involved in creating an implementation of ER suitable for MSC’s use. For MSC, ER implements the two identification heuristics described above as well as three selection techniques. Perhaps more importantly, this section also describes the organizationally specific data processing required to effectively generate expertise recommendations. To implement the identification heuristics, real data from MSC’s change history system and support database were required. MSC provided the most recent eight years of pro￾gram change history covering their medical system and a portion of the shared libraries used by both their medical and dental systems. This corresponds to a total of 7316 individual code modifications. The data from the support database cover the last four years of customer support ac￾tivity. These data include all calls logged and marked as completed. The support data cover software and hardware issues for all of MSC’s products. More than 200,000 calls were handled during the time period covered by the data. Profiling The identification heuristics and profiling modules are strongly related through the profile database. The identifi￾cation heuristics specify which features the profiling mod￾ules must extract. The intricate details of implementing profiling modules for MSC demonstrate how real, messy data sources are transformed into profile records more amenable to the application of identification heuristics. The profiling supervisor controls profiling in ER. Profiling modules are extensions to a base class that provides meth￾ods for interacting with the profiling supervisor and the profile database. ER supports a profiling module for each data source and corresponding identification heuristic. Pro￾filing modules incrementally update each profile record. The change history profiling module crawls through each change collecting the developer responsible for the change, the module name, the version, and date. For each change, the developer’s profile is found and the slot containing code changes is searched for previous modifications to the module. If the developer has no prior changes, then a new entry is made containing the module, the date of the most recent change, and the total number of changes. If the de￾veloper had previously made a change, then the date of last change is updated and the total count of changes is incre￾mented. The profiling module examines dates and is careful to modify profiles only when changes are more recent. The profiling module for the tech support area is incre￾mental as well. However, the tech support profiling module assumes that it will be fed unique records from the techni￾cal support database. The profiling module considers four fields key: the support representative, the description of the problem (a small amount of free form text), the customer, and the module that the customer reports is the problem. These four fields are used to incrementally build three vec￾tor spaces that characterize a specific person’s activity in the support database. 237

Creating profiles with a vector space model required im- For identification, the identification supervisor manages cementing additional functionality to support profiling and modules that have been built to perform identification. An identification. A set of thesauri were needed, one for each identification module base class provides basic functiona f the vector spaces. These determine which terms are in ity for working with the profile database, interaction with the vector space, and therefore should be counted or not recommendation requests, and recommendation lists. Rec- counted. a parser was required to separate terms, phrases, ommendation requests encapsulate the user request, the and important punctuation. This same parser is also used to system state of the request, and the recommendations that parse user querie were made in an attempt to satisfy the request. Recommen- The support database records are exceedingly messy. The dation lists provide methods for ordering and reordering entries have few restrictions and data entry conventions that items that have been recommended. Each recommendation are not formalized. Different support representatives have module is free to determine how escalation is managed and different preferred misspellings and abbreviations. These exactly what additional action is taken on an escalation misspellings and abbreviations created two potential prob- In the case of the change history, the action of the identifi lems that had to be solved. First, by using frequency of cation module is fairly straightforward. The module parses occurrence as a mechanism for determining which terms the request text to identify all of the program modules that are entered into the thesaurus, some of the abbreviations are mentioned in the request. For each program module, a would never be entered because they occur too infre- query is made of the profile database, returning a list of quently. Secondly, with many misspellings and non- individuals who have modified the module. If the request uniform abbreviations, users might find it difficult to garner contains multiple program modules then the resulting lists results when using their personal, quirky language are intersected to find people who touched all modules An ad-hoc approach addressed these problems with the mentioned. For each person identified, a recommendation list item is created that includes the most recent modifica- data. First, a parser was built that recognized punctuation tion date and the total number of touches. The item is then coded to recognize certain phrases which are relevant to the inserted into an unfiltered recommendation list ordered way customers and support reps express problems. The from the most recent touch to least recent touch parser removes common stop words, but does not perform Tech support identification is similar to change any stemming. The parser was used to parse the symptoms, identification. The request text is parsed and three customers, and program fields of all support records gener- vectors are created, one for symptoms, one for customer ating a complete list of terms. For each field, terms were and one for program modules. The profile database is ther sorted. reviewed. and cleaned to create a thesaurus queried using the vector space index. This index groups The tech support profiling module constructs a profile in profiles based on the percentage of terms covered by a per two stages. In the first stage each support database record is sons vector space profile. Escalation level is used to de- examined, and the support rep assigned to the problem is termine the percentage term space that must identified. The symptom, customer, and program module before a person will be considered. As the escalation leve fields are parsed, and terms are validated against the appro- rises, the term space that must be covered drops priate thesaurus. The support reps profile is then updated For each person returned, similarity of that person's profile by incrementing the term counts in the appropriate vector is gauged relative to three query vectors determined by space. Additionally, a master term vector is incremented parsing the request text. The cosine angle of incidence is representing the total number of times a term is used in the calculated for each query vector and the appropriate nor entire database. In the second stage, the profiling module malized vector in the profile. A weighted sum of the angles visits each profile record and normalizes each vector is calculated to create the overall ranking for the match. The profiling supervisor finishes profiling, for all profiling The recommendation list maintains the order from highest techniques, by automatically rebuilding all indices that are to lowest weighted sum. associated with the profile database. The MSC implementa Selection tion requires three indices: one that indexes individuals by When an identification module has completed, the unfil name,one that indexes program modules, and one that in- tered recommendation lists are passed to the selection su lexes the vector space profiles based on the percentage of pervisor. The operation of the selection supervisor is simi- term space coverage. These indices are specifically de- lar to that of the identification supervisor. Selection mod- signed to support identification and their use will be cov- ules transform an unfiltered recommendation list to a fina ered in the next section recommendation list by visiting each recommendation item Identification in the unfiltered list, and then validating or modifying the The details of implementing the change history and tech recommendation. If the item will be part of the final rec- support heuristics provide a clear example of how identifi ommendation it is inserted into the final list. the design cation heuristics can be moved into ER. As well, they dem- of a selection technique decides how escalation level modi- onstrate how different modules can handle escalation in fies or changes each selection technique different ways

Creating profiles with a vector space model required im￾plementing additional functionality to support profiling and identification. A set of thesauri were needed, one for each of the vector spaces. These determine which terms are in the vector space, and therefore should be counted or not counted. A parser was required to separate terms, phrases, and important punctuation. This same parser is also used to parse user queries. The support database records are exceedingly messy. The entries have few restrictions and data entry conventions that are not formalized. Different support representatives have different preferred misspellings and abbreviations. These misspellings and abbreviations created two potential prob￾lems that had to be solved. First, by using frequency of occurrence as a mechanism for determining which terms are entered into the thesaurus, some of the abbreviations would never be entered because they occur too infre￾quently. Secondly, with many misspellings and non￾uniform abbreviations, users might find it difficult to garner results when using their personal, quirky language. An ad-hoc approach addressed these problems with the data. First, a parser was built that recognized punctuation important to how MSC’s systems work. The parser was coded to recognize certain phrases which are relevant to the way customers and support reps express problems. The parser removes common stop words, but does not perform any stemming. The parser was used to parse the symptoms, customers, and program fields of all support records gener￾ating a complete list of terms. For each field, terms were sorted, reviewed, and cleaned to create a thesaurus. The tech support profiling module constructs a profile in two stages. In the first stage each support database record is examined, and the support rep assigned to the problem is identified. The symptom, customer, and program module fields are parsed, and terms are validated against the appro￾priate thesaurus. The support rep’s profile is then updated by incrementing the term counts in the appropriate vector space. Additionally, a master term vector is incremented representing the total number of times a term is used in the entire database. In the second stage, the profiling module visits each profile record and normalizes each vector. The profiling supervisor finishes profiling, for all profiling techniques, by automatically rebuilding all indices that are associated with the profile database. The MSC implementa￾tion requires three indices: one that indexes individuals by name, one that indexes program modules, and one that in￾dexes the vector space profiles based on the percentage of term space coverage. These indices are specifically de￾signed to support identification and their use will be cov￾ered in the next section. Identification The details of implementing the change history and tech support heuristics provide a clear example of how identifi￾cation heuristics can be moved into ER. As well, they dem￾onstrate how different modules can handle escalation in different ways. For identification, the identification supervisor manages modules that have been built to perform identification. An identification module base class provides basic functional￾ity for working with the profile database, interaction with recommendation requests, and recommendation lists. Rec￾ommendation requests encapsulate the user request, the system state of the request, and the recommendations that were made in an attempt to satisfy the request. Recommen￾dation lists provide methods for ordering and reordering items that have been recommended. Each recommendation module is free to determine how escalation is managed and exactly what additional action is taken on an escalation. In the case of the change history, the action of the identifi￾cation module is fairly straightforward. The module parses the request text to identify all of the program modules that are mentioned in the request. For each program module, a query is made of the profile database, returning a list of individuals who have modified the module. If the request contains multiple program modules then the resulting lists are intersected to find people who touched all modules mentioned. For each person identified, a recommendation list item is created that includes the most recent modifica￾tion date and the total number of touches. The item is then inserted into an unfiltered recommendation list ordered from the most recent touch to least recent touch. Tech support identification is similar to change history identification. The request text is parsed and three query vectors are created, one for symptoms, one for customers and one for program modules. The profile database is then queried using the vector space index. This index groups profiles based on the percentage of terms covered by a per￾son’s vector space profile. Escalation level is used to de￾termine the percentage term space that must be covered before a person will be considered. As the escalation level rises, the term space that must be covered drops. For each person returned, similarity of that person’s profile is gauged relative to three query vectors determined by parsing the request text. The cosine angle of incidence is calculated for each query vector and the appropriate nor￾malized vector in the profile. A weighted sum of the angles is calculated to create the overall ranking for the match. The recommendation list maintains the order from highest to lowest weighted sum. Selection When an identification module has completed, the unfil￾tered recommendation lists are passed to the selection su￾pervisor. The operation of the selection supervisor is simi￾lar to that of the identification supervisor. Selection mod￾ules transform an unfiltered recommendation list to a final recommendation list by visiting each recommendation item in the unfiltered list, and then validating or modifying the recommendation. If the item will be part of the final rec￾ommendation it is inserted into the final list. The designer of a selection technique decides how escalation level modi￾fies or changes each selection technique. 238

The MSC implementation includes three selection tech- The escalation tracker manages the expiration of recom niques. The first technique is a simple""No Filter"capabil- mendation requests on the server side. This includes main- y. This technique removes individuals who no longer taining a database of all active requests for each ER user work for MSC. As each recommendation is visited, there is When a client connects, the escalation tracking component a lookup in the user preference and organizational database. looks up the user in the request database and sends the ac- If the person's record indicates that she is still working for tive requests. As requests expire, escalation tracking marks MSC, she is added to the final recommendation list. The the item, removes it from the database and notifies the ap- escalation level does not modify this filter. propriate client. Lastly, the escalation tracker updates the The second technique, "Departmental, "selects based on the user profile and preference data organizational distance between the department of the per The communications tracker is responsible for routing the son making the request and the department of each person chat messages between users. The tracker receives all in recommended. The filter module maintains a small graph coming messages, validates that the recipient is still con- structure that represents the"distance"between each of the nected and then sends the message out. The tracker modi departments at MSC. The departmental technique relies on fies the user preference data on each communications at the current escalation level and the departmental graph to tempt. This implicit preference data contains a list of all of determine who will be added final recommendation the other users with whom contact was attempted and the list. The distance is calculated, and if the distance is less total number of communication attempts. The communica than a threshold, then the person is added to the final rec- tions tracker also provides a simple mechanism for a user to ommendation list state explicit preference for any of the other users known The third selection technique, "Social Network, "filters by the system commendations based on an aggregate social network of Expert face management allows a user to have some con the MSC workplace. Data for the social network were ob- trol over her availability to other users through the system tained during the fieldwork at MSC. Extended interviews Face management allows a user to state her availability on were conducted where the participants performed succes- a four step scale from "available"to"do not disturb. "The ive pile sorts, which resulted in an aggregate social net- default, unless otherwise set, is available. Face manage work for MSC. This social network is a graph structure ment does not require a user to attend to her level of avail where the nodes represent individuals at the workplace and ability. Face management includes a decay system that al le edges represent some relation that links two individuals. lows a user to specify how quickly her current level of ac The social network technique calculates the distance be- tivity will move toward being available. Face management person. If the son making the request and the recommended maintains a thread that periodically checks decay rates and he distance is less than a threshold then the rec modifies availability settings ommended person is added to the final recommendation. In summary, the specific modules that comprise profiling, Note that while these selection mechanisms use organiza- identification, selection, and interaction management differ tionally specific data for MSC, the techniques themselves in their level of specificity to the organizational setting. The would generalize to other organizations heuristics and algorithms used for identification and profil Interaction Management ing are very organizationally specific. While these identifi The interaction management modules in the MSC imple be used in another organization, mentation exemplify how the specific details necessary to implementation is for MSC and the specifics of their identi fication mechanisms are particular to that setting. The general, reusable components. While these components modules that make up selection appear organizationally were required by MSC in order for ER to be successful in their setting, all of the components are reusable within mental character of an organization. Yet, these modules other organizations have some flexibility and can be adapted to other organiza tions given the appropriate data. Lastly, the modules in For example, through our fieldwork at msC we saw par interaction management are perhaps the least organization- ticipants make specific choices with regard to how they ally specific. The modules in interaction management have controlled their accessibility by others and predilection to a face validity that stems from a basic understanding of the pecific communicative media. One expert had a unique technique for indicating how interruptible he was depend- tation shows, ER-Arch handles this combination of organ upon the importance of the problem that required his nationally specific and generic mechanisms for expertise attention. He posted a sign on his already closed door For him, the system needed a similar mechanism CONCLUSIONS Interaction management currently consists of three compo- The Expertise Recommender, through ER-Arch and the nents. the escalation tracker. a communication tracker. and implementation, demonstrates three key features. First, expert face management. This section will briefly cover each of these three components ER-Arch is a general architecture that can be tailored to many different recommendation situations. ER-Arch is

The MSC implementation includes three selection tech￾niques. The first technique is a simple “No Filter” capabil￾ity. This technique removes individuals who no longer work for MSC. As each recommendation is visited, there is a lookup in the user preference and organizational database. If the person’s record indicates that she is still working for MSC, she is added to the final recommendation list. The escalation level does not modify this filter. The second technique, “Departmental,” selects based on the organizational distance between the department of the per￾son making the request and the department of each person recommended. The filter module maintains a small graph structure that represents the “distance” between each of the departments at MSC. The departmental technique relies on the current escalation level and the departmental graph to determine who will be added to the final recommendation list. The distance is calculated, and if the distance is less than a threshold, then the person is added to the final rec￾ommendation list. The third selection technique, “Social Network,” filters recommendations based on an aggregate social network of the MSC workplace. Data for the social network were ob￾tained during the fieldwork at MSC. Extended interviews were conducted where the participants performed succes￾sive pile sorts, which resulted in an aggregate social net￾work for MSC. This social network is a graph structure where the nodes represent individuals at the workplace and the edges represent some relation that links two individuals. The social network technique calculates the distance be￾tween the person making the request and the recommended person. If the distance is less than a threshold, then the rec￾ommended person is added to the final recommendation. Note that while these selection mechanisms use organiza￾tionally specific data for MSC, the techniques themselves would generalize to other organizations. Interaction Management The interaction management modules in the MSC imple￾mentation exemplify how the specific details necessary to create a working system may lead to the creation of more general, reusable components. While these components were required by MSC in order for ER to be successful in their setting, all of the components are reusable within other organizations. For example, through our fieldwork at MSC we saw par￾ticipants make specific choices with regard to how they controlled their accessibility by others and predilection to specific communicative media. One expert had a unique technique for indicating how interruptible he was depend￾ing upon the importance of the problem that required his attention. He posted a sign on his already closed door. For him, the system needed a similar mechanism. Interaction management currently consists of three compo￾nents, the escalation tracker, a communication tracker, and expert face management. This section will briefly cover each of these three components. The escalation tracker manages the expiration of recom￾mendation requests on the server side. This includes main￾taining a database of all active requests for each ER user. When a client connects, the escalation tracking component looks up the user in the request database and sends the ac￾tive requests. As requests expire, escalation tracking marks the item, removes it from the database and notifies the ap￾propriate client. Lastly, the escalation tracker updates the user profile and preference data. The communications tracker is responsible for routing the chat messages between users. The tracker receives all in￾coming messages, validates that the recipient is still con￾nected and then sends the message out. The tracker modi￾fies the user preference data on each communications at￾tempt. This implicit preference data contains a list of all of the other users with whom contact was attempted and the total number of communication attempts. The communica￾tions tracker also provides a simple mechanism for a user to state explicit preference for any of the other users known by the system. Expert face management allows a user to have some con￾trol over her availability to other users through the system. Face management allows a user to state her availability on a four step scale from “available” to “do not disturb.” The default, unless otherwise set, is available. Face manage￾ment does not require a user to attend to her level of avail￾ability. Face management includes a decay system that al￾lows a user to specify how quickly her current level of ac￾tivity will move toward being available. Face management maintains a thread that periodically checks decay rates and modifies availability settings. In summary, the specific modules that comprise profiling, identification, selection, and interaction management differ in their level of specificity to the organizational setting. The heuristics and algorithms used for identification and profil￾ing are very organizationally specific. While these identifi￾cation heuristics might be used in another organization, the implementation is for MSC and the specifics of their identi￾fication mechanisms are particular to that setting. The modules that make up selection appear organizationally specific because they are tailored to the social and depart￾mental character of an organization. Yet, these modules have some flexibility and can be adapted to other organiza￾tions given the appropriate data. Lastly, the modules in interaction management are perhaps the least organization￾ally specific. The modules in interaction management have a face validity that stems from a basic understanding of the requirements of expertise location. As the MSC implemen￾tation shows, ER-Arch handles this combination of organi￾zationally specific and generic mechanisms for expertise recommendation. CONCLUSIONS The Expertise Recommender, through ER-Arch and the implementation, demonstrates three key features. First, ER-Arch is a general architecture that can be tailored to many different recommendation situations. ER-Arch is 239

capable of supporting more traditional recommendation 3. Cicourel, A.V. The Integration of Distributed Knowledge in situations by incorporating appro ng and identi Collaborative Medical Diagnosis. in Galegher, J, Kraut, R E fication modules. It is also highly suited for expertise rec- and egido, C eds Intellectual Teamwork, Lawrence Erlbaum Associates. Hillsdale. NJ. 1990. 221-24 4. Ehrlich, K. and Cash, D, Turning Information into Knowl Second, the implementation of Er for MSC relies on dge: Information Finding as a Collaborative Activity. in profiling techniques that are not common to other Digital Libraries 94, 119-125 recommendation systems. ER generates profiles of people 5. Foner, L N, Yenta: A Multi-Agent, Referral-Based Match- based on work products and work byproducts. This making System. in First International Conference on approach trades the problems of critical mass and sparse Autonomous Agents (Agent97) ratings for problems with dirty and inconsistent 6. Furnas. G W.Deerwester S. Dumais, S. T. Landauer. TK organizationally specific data sources. Harshman, R.A., Streeter, L.A. and Lochbaum, K.E., Informa- tion retrieval using a singular value decomposition model of to tease apart technical concerns involved with making latent semantic structure. S/GIR88. 465-480 recommendations from the social and collaborative con- rg, D, Nichols, D, Oki, B M. and Terry, D. Using cerns. This separation allows ER to support multiple col ative Filtering to Weave an Information Tapest 35(12.61-70. laborative models and provide more adaptive recommenda tions. The ER client exposes these features allowing 8. Hill, W, Stead, L, Rosenstein, M. and Furnas, G, Recom users to pick recommendation strategies appropriate for mending and Evaluating Choices in a Virtual Community of Use.CH5,194-201 their task 9. Hill, w. and Terveen, L, Using Frequency-of-mention in Recommendation systems, such as ER, provide another ublic Conversations for Social Filtering CSCH 96, 106-11 approach to supporting natural expertise locating behavior. 10. Kautz, H.A., Selman, B and Shah, M. Referral Web: Combin In situations where a senior employee, guru, information ng Social Networks and Collaborative Filtering. CACM, 40 mediator, or expertise concierge is not available, a system (3).63-65 ike ER can suggest alternatives where previously no help 11.Konstan, J.A., Miller, B N, Maltz, D, Herlocker, J. L would have been available. By relying on organizationally Gordon, L.R. and Riedel, J GroupLens: Applying Collabora- relevant sources of information and heuristics that are natu- e Filtering to Usenet News. CACM, 40(3). 77-87 rally used, systems like ER can assist in finding people who 12 Maltz, D and Ehrlich, K, Pointing The Way: Active Collabo- have expertise and who may not otherwise be identified. rative Filtering. CHI 95, 202-209 13. McDonald, D w. and Ackerman, M.s., Just Talk to Me: A Acknowledgements Field Study of Expertise Location. CSCW98, 315-324 This project has been funded, in part, by grants from Na- 14. O, J.E. Talking About Machines: An Ethnography of a Mod- tional Science Foundation (IRl-9702904), the UCI/NSF ern Job. Cornell University Press,Ithaca,1996 Industry/University Cooperative Research Center at the 15. Paepcke, A. Information Needs in Technical Work Settings Center for Research on Information Technology and Or- d Their Implications for the Design of Computer Tools ganizations(CRITO), and the University of California omputer Supported Cooperative Work: The Journal of Col- MICRO program. Additionally, the first author was sup- orative Computing, 5. 63-92 ported by the University of California Regents'Disserta- 16. Resnick, P, lacovou, N, Suchak, M, Bergstrom, P. and tion Fellowship Riedl, J, GroupLens: An Open Architecture for Collaborative This work has benefited from conversations with Kate Filtering of Netnews. CSCW 94, 1 17. Sarwar, B M, Konstan, J.A., Borchers, A, Herlocker, J Ehrlich, Saul Greenberg, Joe Konstan, John Riedl, Lynn Streeter. Loren Terveen and Clark Turner. Members of our Miller, B and Riedl, J, Using Filtering Agents to Improve Prediction Quality in the groupLens Research Collaborative research group, Wayne Lutters and Jack Muramatsu, Filtering System. CSCW 98, 345-354 contributed to our understanding of expertise. We thank the 18 Shardanand, U. and Maes. P, Social Information filtering anonymous reviewers for aback. We Algorithms for Automating"Word of Mouth".CHI 95, 210- like to thank the participants at MSC for their patience and insights over the last few years 19. Shaw, M. and Garlan, D Software Architecture: Perspectives REFERENCES on an emerging discipline. Prentice Hall, 1996. 1. Allen, T.J. Managing the Flow of Technology. MIT Press, 20. Streeter, L.A. and Lochbaum, K.E., Who Knows: A System Cambridge, 1977 Based on Automatic Representation of Semantic Structure. 2. Balabanovic, M. and Shoham, Y. Fab: Content-Based, Col RA088.380-388 aborative Recommendation CACM, 40 (3). 66-7 21. Terveen, L, Hill, w, Amento, B, McDonald, D. and Creter J. PHOAKS: A System for Sharing Recommendations. CACM,4003).59-62 240

capable of supporting more traditional recommendation situations by incorporating appropriate profiling and identi￾fication modules. It is also highly suited for expertise rec￾ommendation. Second, the implementation of ER for MSC relies on profiling techniques that are not common to other recommendation systems. ER generates profiles of people based on work products and work byproducts. This approach trades the problems of critical mass and sparse ratings for problems with dirty and inconsistent organizationally specific data sources. Lastly, ER-Arch and the MSC implementation of ER begin to tease apart technical concerns involved with making recommendations from the social and collaborative con￾cerns. This separation allows ER to support multiple col￾laborative models and provide more adaptive recommenda￾tions. The ER client exposes these features allowing ER users to pick recommendation strategies appropriate for their task. Recommendation systems, such as ER, provide another approach to supporting natural expertise locating behavior. In situations where a senior employee, guru, information mediator, or expertise concierge is not available, a system like ER can suggest alternatives where previously no help would have been available. By relying on organizationally relevant sources of information and heuristics that are natu￾rally used, systems like ER can assist in finding people who have expertise and who may not otherwise be identified. Acknowledgements This project has been funded, in part, by grants from Na￾tional Science Foundation (IRI-9702904), the UCI/NSF Industry/University Cooperative Research Center at the Center for Research on Information Technology and Or￾ganizations (CRITO), and the University of California MICRO program. Additionally, the first author was sup￾ported by the University of California Regents’ Disserta￾tion Fellowship. This work has benefited from conversations with Kate Ehrlich, Saul Greenberg, Joe Konstan, John Riedl, Lynn Streeter, Loren Terveen, and Clark Turner. Members of our research group, Wayne Lutters and Jack Muramatsu, contributed to our understanding of expertise. We thank the anonymous reviewers for their feedback. We would also like to thank the participants at MSC for their patience and insights over the last few years. REFERENCES 1. Allen, T.J. Managing the Flow of Technology. MIT Press, Cambridge, 1977. 2. Balabanovic, M. and Shoham, Y. Fab: Content-Based, Col￾laborative Recommendation. CACM, 40 (3). 66 - 72. 3. Cicourel, A.V. The Integration of Distributed Knowledge in Collaborative Medical Diagnosis. in Galegher, J., Kraut, R.E. and Egido, C. eds. Intellectual Teamwork, Lawrence Erlbaum Associates, Hillsdale, NJ, 1990, 221 - 242. 4. Ehrlich, K. and Cash, D., Turning Information into Knowl￾edge: Information Finding as a Collaborative Activity. in Digital Libraries '94, 119 - 125. 5. Foner, L.N., Yenta: A Multi-Agent, Referral-Based Match￾making System. in First International Conference on Autonomous Agents (Agent'97). 6. Furnas, G.W., Deerwester, S., Dumais, S.T., Landauer, T.K., Harshman, R.A., Streeter, L.A. and Lochbaum, K.E., Informa￾tion retrieval using a singular value decomposition model of latent semantic structure. SIGIR'88, 465 - 480. 7. Goldberg, D., Nichols, D., Oki, B.M. and Terry, D. Using Collaborative Filtering to Weave an Information Tapestry. CACM, 35 (12). 61 - 70. 8. Hill, W., Stead, L., Rosenstein, M. and Furnas, G., Recom￾mending and Evaluating Choices in a Virtual Community of Use. CHI '95, 194 - 201. 9. Hill, W. and Terveen, L., Using Frequency-of-mention in Public Conversations for Social Filtering. CSCW '96, 106-112. 10. Kautz, H.A., Selman, B. and Shah, M. Referral Web: Combin￾ing Social Networks and Collaborative Filtering. CACM, 40 (3). 63 - 65. 11. Konstan, J.A., Miller, B.N., Maltz, D., Herlocker, J.L., Gordon, L.R. and Riedel, J. GroupLens: Applying Collabora￾tive Filtering to Usenet News. CACM, 40 (3). 77 - 87. 12. Maltz, D. and Ehrlich, K., Pointing The Way: Active Collabo￾rative Filtering. CHI '95, 202 - 209. 13. McDonald, D.W. and Ackerman, M.S., Just Talk to Me: A Field Study of Expertise Location. CSCW'98, 315 - 324. 14. Orr, J.E. Talking About Machines: An Ethnography of a Mod￾ern Job. Cornell University Press, Ithaca, 1996. 15. Paepcke, A. Information Needs in Technical Work Settings and Their Implications for the Design of Computer Tools. Computer Supported Cooperative Work: The Journal of Col￾laborative Computing, 5. 63 - 92. 16. Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P. and Riedl, J., GroupLens: An Open Architecture for Collaborative Filtering of Netnews. CSCW '94, 175 - 186. 17. Sarwar, B.M., Konstan, J.A., Borchers, A., Herlocker, J., Miller, B. and Riedl, J., Using Filtering Agents to Improve Prediction Quality in the GroupLens Research Collaborative Filtering System. CSCW '98, 345-354. 18. Shardanand, U. and Maes, P., Social Information Filtering: Algorithms for Automating "Word of Mouth". CHI '95,210- 217. 19. Shaw, M. and Garlan, D. Software Architecture: Perspectives on an emerging discipline. Prentice Hall, 1996. 20. Streeter, L.A. and Lochbaum, K.E., Who Knows: A System Based on Automatic Representation of Semantic Structure. RIAO '88 , 380 - 388. 21. Terveen, L., Hill, W., Amento, B., McDonald, D. and Creter, J. PHOAKS: A System for Sharing Recommendations. CACM, 40 (3). 59 - 62. 240