Our"retention"category is intended to group those be- message to some form of group venue such as a mailing haviors that suggest some degree of intention to make fu list establishes the same sort of link. Goldberg et al re use of an object. Bookmarking a web page is a simple 1987) described a simple example of this in which users could construct an electronic mail filter to display mes- idea assave a reference "to accommodate a wider range sag of actions such as construction of symbolic links within a ypertext links from one web page to another and biblio- file system. Rucker Polanco(1997), for example, con- graphic citations in academic papers create links from a structed a recommender system using bookmark lists portion of an object (characterized, perhaps, by some Saving the object itself is the obvious alternative, some- neighborhood around the link itself) to another object, al thing Stevens(1993)used as implicit feedback for content- though the refinement to a portion of a document has not based filtering. In either case, the object may be saved been exploited often. Brin Page(1998)provide an ex with or without some form of annotation. For example, ample of how hypertext links might be used, although their web browsers typically default to using the page title in the focus is on a population statistics rather than individual bookmark list, but users may optionally provide a more preferences. Garfield(1979)describes the design of re meaningful entry if they desire. Although numerous con- trieval systems that are based on bibliographic citations founding factors would likely be present, it may be possi Alternatively, selective inclusion of another document ble to infer something about the value a user places on an using either cut-and-paste or a quotation, creates a link individual page by whether or not they go to the trouble of from an information object to a portion of another constructing an informative bookmark entry. Similarly, users may choose to save a reference or an object in an Using Implicit Feedback explicitly organized fashion or in the default manner. For example, storing electronic mail about this workshop in a The goal of a recommender system is to help users find new folder might provide greater support for an inference desirable information objects, That task combines infer that the user ascribes particular value to the message than ence and prediction, and Figures 1 and 2 show alternative originator. The salient issue in this case is not the act of duce ratings similar to those that a user would have ex organizing, but rather the way in which the organization plicitly assigned, and then the prediction stage uses those given to an individual object distinguishes it from the way estimated ratings to predict future ratings. Konstan et al in which similar forms of organization are assigned to adopted this perspective when evaluating how well ob other objects. This difference may not be easy to charac- terize, but it may be worth thinking about how to do it. We served reading time predicted explicit ratings for individ- ual articles. Figure 2 shows an alternative strategy in have chosen to group printing with retention because of the hich past observations are used to predict user behavior permanence of the printed page, but users may also print to facilitate examination because in response to new information, and then the inference per still has some decided advantages over electronic dis- stage seeks to estimate the value of the information based on the predicted behavior. We are not aware of any im plays in many applications. Printing overlaps with the plementations of this second approach, but Stevens(1993) next category(reference) as well, since users may print a implemented a simplified version of the strategy. He pre- document or image with the intention of forwarding them dicted the examination duration for a new usenet news to another individual or including portions in anothe document. Nevertheless, printing is often associated with a article based on the examination durations for similar arti- cles in the past and then constructed content-based queries desire for retention, so we find this grouping useful. A that would select articles with long predicted examination with examination, it may be possible to infer something durations. This essentially amounts to a degenerate infer- about the portions of a document that the user finds most valuable from the portions which he or she chooses to ence stage in which desirability is assumed to increase print. Finally, the retention category is distinguished by monotonically with examination duration The distinction between the two strategies is quite subtle the possibility of directly observing evidence of negative evaluations as well. when retention is a default condition in the case of content-based filtering. In as in some electronic mail systems, a decision by the user system, by contrast, the strategy shown in Figure 1 would characterize each article using the examination durations to delete an object might support to an inference that the reported by other users, while the strategy shown in Figure 2 would characterize each article using the predicted rat retained ngs for other users. Recommender systems based on the The"refer to" category may appear at first glance to of observable second strategy might be more flexible, since participating each has the effect of establishing some form of link be users might draw different inferences from the same ob- servations if they did not share a common set of objectives tween two objects. Forwarding a message, for example, establishes a link between the new message and the origi- On the other hand, recommender systems using the first strategy would likely have more context available locally nal. Similarly, replying individually or posting a follow up for interpreting observations than would be available atOur "retention" category is intended to group those be￾haviors that suggest some degree of intention to make fu￾ture use of an object. Bookmarking aweb page is a simple example of such a behavior, and we have generalized that idea as "save a reference" to accommodate a wider range of actions such as construction of symbolic links within a file system. Rucker & Polanco (1997), for example, con￾structed a recommender system using bookmark lists. Saving the object itself is the obvious alternative, some￾thing Stevens (1993) used as implicit feedback for content￾based filtering. In either case, the object may be saved with or without some form of annotation. For example, web browsers typically default to using the page title in the bookmark list, but users may optionally provide a more meaningful entry if they desire. Although numerous con￾founding factors would likely be present, it may be possi￾ble to infer something about the value a user places on an individual page by whether or not they go to the trouble of constructing an informative bookmark entry. Similarly, users may choose to save a reference or an object in an explicitly organized fashion or in the default manner. For example, storing electronic mail about this workshop in a new folder might provide greater support for an inference that the user ascribes particular value to the message than would the use of some default scheme such as placing it in the folder routinely used for mail from the message’s originator. The salient issue in this case is not the act of organizing, but rather the way in which the organization given to an individual object distinguishes it from the way in which similar forms of organization are assigned to other objects. This difference may not be easy to charac￾terize, but it may be worth thinking about how to do it. We have chosen to group printing with retention because of the permanence of the printed page, but users may also print document or images to facilitate examination because pa￾per still has some decided advantages over electronic dis￾plays in many applications. Printing overlaps with the next category (reference) as well, since users may print document or image with the intention of forwarding them to another individual or including portions in another document. Nevertheless, printing is often associated with a desire for retention, so we find this grouping useful. As with examination, it may be possible to infer something about the portions of a document that the user finds most valuable from the portions which he or she chooses to print. Finally, the retention category is distinguished by the possibility of directly observing evidence of negative evaluations as well. When retention is a default condition, as in some electronic mail systems, a decision by the user to delete an object might support to an inference that the deleted object is less valued than other objects that are retained. The "refer to" category may appear at first glance to contain a fairly eclectic group of observable activities, but each has the effect of establishing some form of link be￾tween two objects. Forwarding a message, for example, establishes a link between the new message and the origi￾nal. Similarly, replying individually or posting a follow up message to some form of group venue such as a mailing list establishes the same sort of link. Goldberg et al. (1987) described a simple example of this in which users could construct an electronic mail filter to display mes￾sages that their colleagues had taken the time to reply to. Hypertext links from one web page to another and biblio￾graphic citations in academic papers create links from a portion of an object (characterized, perhaps, by some neighborhood around the link itself) to another object, al￾though the refinement to a portion of a document has not been exploited often. Brin & Page (1998) provide an ex￾ample of how hypertext links might be used, although their focus is on a population statistics rather than individual preferences. Garfield (1979) describes the design of re￾trieval systems that are based on bibliographic citations. Alternatively, selective inclusion of another document, using either cut-and-paste or a quotation, creates a link from an information object to a portion of another. Using Implicit Feedback The goal of a recommender system is to help users fred desirable information objects. That task combines infer￾ence and prediction, and Figures 1 and 2 show alternative strategies for accomplishing this. Figure 1 depicts a modular strategy in which the inference stage seeks to pro￾duce ratings similar to those that a user would have ex￾plicitly assigned, and then the prediction stage uses those estimated ratings to predict future ratings. Konstan et al. adopted this perspective when evaluating how well ob￾served reading time predicted explicit ratings for individ￾ual articles. Figure 2 shows an alternative strategy in which past observations are used to predict user behavior in response to new information, and then the inference stage seeks to estimate the value of the information based on the predicted behavior. We are not aware of any im￾plementations of this second approach, but Stevens (1993) implemented a simplified version of the strategy. He pre￾dicted the examination duration for a new USENET news article based on the examination durations for similar arti￾cles in the past and then constructed content-based queries that would select articles with long predicted examination durations. This essentially amounts to a degenerate infer￾ence stage in which desirability is assumed to increase monotonically with examination duration. The distinction between the two strategies is quite subtle in the case of content-based filtering. In a recommender system, by contrast, the strategy shown in Figure 1 would characterize each article using the examination durations reported by other users, while the strategy shown in Figure 2 would characterize each article using the predicted rat￾ings for other users. Recommender systems based on the second strategy might be more flexible, since participating users might draw different inferences from the same ob￾servations if they did not share a common set of objectives. On the other hand, recommender systems using the first strategy would likely have more context available locally for interpreting observations than would be available at 82