《电子商务 E-business》阅读文献：Encouraging user participation in a course recommender system-An impact on user behavior

Computers in Human Behavior 27(2011)276-284 Contents lists available at Science Direct Computers in Human behavior ELSEVIER journalhomepagewww.elsevier.com/locate/comphumbeh Encouraging user participation in a course recommender system: An impact on user behavior Rosta Farzan, Peter Brusilovsky Human-Computer Interaction Institute(HCl). School of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213-3891, United States ARTICLE INFO A BSTRACT User participation emerged as a critical issue for collaborative and social recommender systems as well as Available online 16 September 2010 for a range of other systems based on the power of user community. A range of mechanisms to encourage user participation in social systems has been proposed over the last few years: however, the impact of nese mechanisms on users behavior in recommender systems has not been studied sufficiently. This paper investigates the impact of encouraging user pa n the context of Incentive mechanism User study nity-based course recommender system. The recommendation power of CourseAgent is based on course tings provided by a community of students. To increase the number of course ratings, CourseAgent applies an incentive mechanism which turns user feedback into a self-beneficial activity. In this paper. ye describe the design and implementation of our course recommendation system and its incentive mechanism. We also report a dual impact of this mechanism on user behavior discovered in two user G 2010 Elsevier Ltd. All rights reserved. 1 Introduction ems including forums, wikis, social bookmarking, and social link- ng systems. In all these areas the value of the system and the very In our information age, everyone is faced with abundance of survival of online communities is highly dependent on user contri- nformation while making decisions about almost everything: bution. Naturally, the rise of community-based systems in the age movies to watch, restaurant to go, or research papers to read. Peo- of Web 2.0 lead to an increased interest to the problem of user par ple traditionally turn into others who they trust when over- ticipation and various incentive mechanisms to encourage partici which is recommended by a friend, or go to a busy and popular ovie pation(Beenen et al., 2004; Cheng Vassileva, 2005, 2006: Harper. whelmed with decision making process. They will watch a Li, Chen, Konstan, 2005: Rashid et al., 2006) taurant. Social web applications such as collaborative This paper explores the effect of incentive mechanism on users mender systems, social networking sites, and social bookmarking contribution in a specific community-based recommender sys- sites capitalize on natural tendency of people to follow each other. tems. We attempt to extend existing research in three ways. First, They try to bring together the collective wisdom of the community we explore a relatively non-traditional social recommender system and exploit this wisdom to guide their users. driven by the community of users. Second we evaluate the effec- Collaborative recommender systems could be considered as the tiveness of a new incentive mechanism to encourage user ratings most explored kind of community-powered systems. A range of for the recommender system. Finally, we want to assess both posi- recommendation techniques have been proposed and extensively tive and negative impacts of the incentive mechanism on studies over the last 15 years in order to improve the quality of rec- behavior. ommendation(Schafer, Frankowski, Herlocker, Sen, 2007). How- The incentive mechanism explored in our studies appeals to ver, the last 5 years of research in the area of collaborativ ser personal needs. The core idea of this approach is turning user recommender systems demonstrated that the amount and the contributions into an activity which can benefit to the users them- quality of user ratings can be as vital for the success of a recom- selves. This approach is specifically useful when the kind of ex- mender system as the quality of its recommendations(herlocker, ected contribution has no inherent benefit for the users. while Konstan, Terveen, Riedl, 2004). being beneficial to the community as a whole. a good example of rec he problem of user contribution is not unique for collaborative this context is provided by social course recommendation system commender systems, but shared by all community-based sys- CourseAgent presented in this paper. CourseAgent employs social navigation approach to provide course recommendations based Corresponding author. Tel: +1 412 268 1615: fax: +1 412 268 1266. on students' assessment of course relevance to their career goals. E-mailaddressesrfarzan@cs.cmu.edu,rostaf@gmail.com(rFarzan). In brief, course ratings left by students after completing their 0747-5632/s It matter 2010 Elsevier Ltd. All rights reserved. do:10.1016chb201008005

Encouraging user participation in a course recommender system: An impact on user behavior Rosta Farzan ⇑ , Peter Brusilovsky Human–Computer Interaction Institute (HCII), School of Computer Science, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213-3891, United States article info Article history: Available online 16 September 2010 Keywords: User participation Incentive mechanism User study abstract User participation emerged as a critical issue for collaborative and social recommender systems as well as for a range of other systems based on the power of user community. A range of mechanisms to encourage user participation in social systems has been proposed over the last few years; however, the impact of these mechanisms on users behavior in recommender systems has not been studied sufficiently. This paper investigates the impact of encouraging user participation in the context of CourseAgent, a commu￾nity-based course recommender system. The recommendation power of CourseAgent is based on course ratings provided by a community of students. To increase the number of course ratings, CourseAgent applies an incentive mechanism which turns user feedback into a self-beneficial activity. In this paper, we describe the design and implementation of our course recommendation system and its incentive mechanism. We also report a dual impact of this mechanism on user behavior discovered in two user studies. 2010 Elsevier Ltd. All rights reserved. 1. Introduction In our information age, everyone is faced with abundance of information while making decisions about almost everything: movies to watch, restaurant to go, or research papers to read. Peo￾ple traditionally turn into others who they trust when over￾whelmed with decision making process. They will watch a movie which is recommended by a friend, or go to a busy and popular res￾taurant. Social web applications such as collaborative recom￾mender systems, social networking sites, and social bookmarking sites capitalize on natural tendency of people to follow each other. They try to bring together the collective wisdom of the community and exploit this wisdom to guide their users. Collaborative recommender systems could be considered as the most explored kind of community-powered systems. A range of recommendation techniques have been proposed and extensively studies over the last 15 years in order to improve the quality of rec￾ommendation (Schafer, Frankowski, Herlocker, & Sen, 2007). How￾ever, the last 5 years of research in the area of collaborative recommender systems demonstrated that the amount and the quality of user ratings can be as vital for the success of a recom￾mender system as the quality of its recommendations (Herlocker, Konstan, Terveen, & Riedl, 2004). The problem of user contribution is not unique for collaborative recommender systems, but shared by all community-based sys￾tems including forums, wikis, social bookmarking, and social link￾ing systems. In all these areas the value of the system and the very survival of online communities is highly dependent on user contri￾bution. Naturally, the rise of community-based systems in the age of Web 2.0 lead to an increased interest to the problem of user par￾ticipation and various incentive mechanisms to encourage partici￾pation (Beenen et al., 2004; Cheng & Vassileva, 2005, 2006; Harper, Li, Chen, & Konstan, 2005; Rashid et al., 2006). This paper explores the effect of incentive mechanism on users’ contribution in a specific community-based recommender sys￾tems. We attempt to extend existing research in three ways. First, we explore a relatively non-traditional social recommender system driven by the community of users. Second, we evaluate the effec￾tiveness of a new incentive mechanism to encourage user ratings for the recommender system. Finally, we want to assess both posi￾tive and negative impacts of the incentive mechanism on user behavior. The incentive mechanism explored in our studies appeals to user personal needs. The core idea of this approach is turning user contributions into an activity, which can benefit to the users them￾selves. This approach is specifically useful when the kind of ex￾pected contribution has no inherent benefit for the users, while being beneficial to the community as a whole. A good example of this context is provided by social course recommendation system CourseAgent presented in this paper. CourseAgent employs social navigation approach to provide course recommendations based on students’ assessment of course relevance to their career goals. In brief, course ratings left by students after completing their 0747-5632/$ - see front matter 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2010.08.005 ⇑ Corresponding author. Tel.: +1 412 268 1615; fax: +1 412 268 1266. E-mail addresses: rfarzan@cs.cmu.edu, rostaf@gmail.com (R. Farzan). Computers in Human Behavior 27 (2011) 276–284 Contents lists available at ScienceDirect Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh

R Farzan, P. Brusilovsky Computers in Human Behavior 27(2011)276-284 courses are used to guide future students to most relevant courses. Lui, Lang, and Kwok(2002), community contribution can be moti- While this context looks similar to collaborative recommender sys- vated by individual and interpersonal factors. Individual factors in tems, CourseAgent is different because the volume and the quality clude extrinsic motivations, such as rewards and personal need of student ratings does not influence the quality of recommenda- and intrinsic motivations, such as reputation and altruism. Inter- tion given to this student, it only affects the community, i. e, future personal factors are motivations such as liking and affiliation. udents of the rated courses. Thus users of CourseAgent have rel tively weak motivation to rate courses such as liking the commu- 2.1. Intrinsic motivation nity and affiliation with community. Our experience shows that this is not sufficient to fuel the social recommendation system. Intrinsic motivation happens when people engage in activities Following the"personal needs"approach, we extended Course- for the activity itself and without any obvious external incentives Agent with progress feature, which turns user feedback such as rewards. Intrinsic motivation can be affected by different to an activity that is both attractive and meaningful for each stu- factors. According to"social loafing"theory, people are more likely dent. The career progress feature allowed the students to view the to make less effort when performing a task as part of a group and it progress towards their career goals. The progress report considers tends to be robust and generalized across tasks and populations. all courses evaluated by a student to calculate their progress with Social loafing mainly occurs because uniqueness and effect of indi- this feature course rating becomes meaningful for the students vidual effort is not clear in a group task. "Collective Effort Model themselves: the more taken courses are rated, the more accurate suggests that social loafing can be decreased by clarifying the is the displayed progress report To investigate the effect of our importance and uniqueness of individual contributions(Karau proposed incentive mechanism on students' rating behavior we Williams, 1993). Additionally, goal setting theory"suggests that run two user studies. This paper presents the results of our studies a challenging, short-term goal, rather than a vague, long-term goal focusing on both advantages and shortcomings of the incentive stimulates high performance in users (Locke Latham, 2002). mechanism These ideas have been explored by several projects focusing on This paper is organized as follows: Section 2 reviews the current increasing user participation. pproaches in encouraging user participation in online communi- Beenen et al. studied the application of collective effort and goal ties. Section 3 describes our system, CourseAgent and the design setting principles in motivating contribution in online communi of the incentive mechanism in CourseAgent. Section 4 presents ties. They conducted several experiment in MovieLens, the online two studies, which evaluated the effectiveness of our approach to movie recommender system(Beenen et al, 2004). They studie motivate users' ratings. Sections 5 and 6 talk about possible draw- the effect of revealing to the user the uniqueness and benefit of backs of different incentive systems and present the evaluation of their contribution to determine which motivated users to rate the potential negative effect of our motivational forces for course more movies. In their analysis, they examined the differences be- atings. We conclude the paper with the summary of results and tween revealing the benefit to oneself versus the benefit to others some ideas for the future work. Their result shows that users are more likely to participate when they are reminded about benefit to oneself and the others. Coher 2. Encouraging user participation ent with goal setting theory, their result shows that specific goals resulted in higher number of ratings. Furthermore, they find out It has been recognized that success of all kinds of social soft- that group goals stimulate higher contribution than individual ware and online communities is highly dependent on participation goals. of their users. This recognition influenced large interest to the pplication of goal setting theory in online communities can problem of user participation and increasing volume of research also be observed in social networking sites such as LinkedIn, which on this problem. One of the most frequently cited issues in the area provide information about how complete a users profile is of user participation is the under-contribution and inequality of contribution in online communities. In most online communities 2. 2. Extrinsic motivation 1% of users account for 90% of content(Nielsen, 2006). Wikipedia, ia is one of Extrinsic motivation comes from outside. It happens when peo- cessful example of online communities. As of 2008, the site has 684 ple engage in activities as a result of an external incentive mecha- million unique visitors; however, only 75,000 (0.01%)of them are nism such as contingent rewards. In those cases the task is not active contributors and very small percentage of users account satisfying enough and the external incentives add into the pleasure for large amount of data on the site(wikipedia). While survival and satisfaction of the task. The following subsections discuss of small online communities with small number of users is highly some examples of external incentive mechanisms. Each incentive dependent on contribution of majority of the users, bigger commu- mechanism examined below extends the interface and functional- nities like Wikipedia with large number of users can survive even ity of a social system in order to increase the volume of user with small percentage of users contributing. However, even large contributions. community can be affected by the under-contribution problem through participation inequality bias when small percentage of 2. 2. 1. Rewards population represent the views of larger population. While it is Researchers in the filed of human-computer interaction have not possible to encourage all users to contribute, it is important tried to study the effect of rewards on user contribution in online A substantial amount of research in the field of social sciences source-sharing system Comtella(Bretzke Vassileva, 2003) have focused on understanding user motivation for participation Comtella is a resource-sharing system that allows the members in communities and group works and different incentive mecha- of an online community share web resources amongst each other. nisms. Social exchange theory is a social psychological theory The system rewards more cooperative users with incentives such (Emerson, 1976). The purpose of the exchange is maximizing the munity. a more recent version of Comtella uses an adaptive reward benefit and minimizing the cost. According to social exchange the- mechanism to influence the quality of participation. This new ory, users need to be motivated to contribute. As summarized by incentive mechanism only rewards high-quality participation

courses are used to guide future students to most relevant courses. While this context looks similar to collaborative recommender sys￾tems, CourseAgent is different because the volume and the quality of student ratings does not influence the quality of recommenda￾tion given to this student, it only affects the community, i.e., future students of the rated courses. Thus users of CourseAgent have rel￾atively weak motivation to rate courses such as liking the commu￾nity and affiliation with community. Our experience shows that this is not sufficient to fuel the social recommendation system. Following the ‘‘personal needs” approach, we extended Course￾Agent with a career progress feature, which turns user feedback into an activity that is both attractive and meaningful for each stu￾dent. The career progress feature allowed the students to view the progress towards their career goals. The progress report considers all courses evaluated by a student to calculate their progress. With this feature, course rating becomes meaningful for the students themselves: the more taken courses are rated, the more accurate is the displayed progress report. To investigate the effect of our proposed incentive mechanism on students’ rating behavior we run two user studies. This paper presents the results of our studies focusing on both advantages and shortcomings of the incentive mechanism. This paper is organized as follows: Section 2 reviews the current approaches in encouraging user participation in online communi￾ties. Section 3 describes our system, CourseAgent and the design of the incentive mechanism in CourseAgent. Section 4 presents two studies, which evaluated the effectiveness of our approach to motivate users’ ratings. Sections 5 and 6 talk about possible draw￾backs of different incentive systems and present the evaluation of the potential negative effect of our motivational forces for course ratings. We conclude the paper with the summary of results and some ideas for the future work. 2. Encouraging user participation It has been recognized that success of all kinds of social soft￾ware and online communities is highly dependent on participation of their users. This recognition influenced large interest to the problem of user participation and increasing volume of research on this problem. One of the most frequently cited issues in the area of user participation is the under-contribution and inequality of contribution in online communities. In most online communities 1% of users account for 90% of content (Nielsen, 2006). Wikipedia, the web-based collaborative encyclopedia, is one of the most suc￾cessful example of online communities. As of 2008, the site has 684 million unique visitors; however, only 75,000 (0.01%) of them are active contributors and very small percentage of users account for large amount of data on the site (Wikipedia). While survival of small online communities with small number of users is highly dependent on contribution of majority of the users, bigger commu￾nities like Wikipedia with large number of users can survive even with small percentage of users contributing. However, even large community can be affected by the under-contribution problem through participation inequality bias when small percentage of population represent the views of larger population. While it is not possible to encourage all users to contribute, it is important for both small and large online communities to motivate larger percentage of users to contribute. A substantial amount of research in the field of social sciences have focused on understanding user motivation for participation in communities and group works and different incentive mecha￾nisms. Social exchange theory is a social psychological theory which explains social behavior as a result of an exchange process (Emerson, 1976). The purpose of the exchange is maximizing the benefit and minimizing the cost. According to social exchange the￾ory, users need to be motivated to contribute. As summarized by Lui, Lang, and Kwok (2002), community contribution can be moti￾vated by individual and interpersonal factors. Individual factors in￾clude extrinsic motivations, such as rewards and personal need, and intrinsic motivations, such as reputation and altruism. Inter￾personal factors are motivations such as liking and affiliation. 2.1. Intrinsic motivation Intrinsic motivation happens when people engage in activities for the activity itself and without any obvious external incentives such as rewards. Intrinsic motivation can be affected by different factors. According to ‘‘social loafing” theory, people are more likely to make less effort when performing a task as part of a group and it tends to be robust and generalized across tasks and populations. Social loafing mainly occurs because uniqueness and effect of indi￾vidual effort is not clear in a group task. ‘‘Collective Effort Model” suggests that social loafing can be decreased by clarifying the importance and uniqueness of individual contributions (Karau & Williams, 1993). Additionally, ‘‘goal setting theory” suggests that a challenging, short-term goal, rather than a vague, long-term goal stimulates high performance in users (Locke & Latham, 2002). These ideas have been explored by several projects focusing on increasing user participation. Beenen et al. studied the application of collective effort and goal setting principles in motivating contribution in online communi￾ties. They conducted several experiment in MovieLens, the online movie recommender system (Beenen et al., 2004). They studied the effect of revealing to the user the uniqueness and benefit of their contribution to determine which motivated users to rate more movies. In their analysis, they examined the differences be￾tween revealing the benefit to oneself versus the benefit to others. Their result shows that users are more likely to participate when they are reminded about benefit to oneself and the others. Coher￾ent with goal setting theory, their result shows that specific goals resulted in higher number of ratings. Furthermore, they find out that group goals stimulate higher contribution than individual goals. Application of goal setting theory in online communities can also be observed in social networking sites such as LinkedIn, which provide information about how complete a users profile is. 2.2. Extrinsic motivation Extrinsic motivation comes from outside. It happens when peo￾ple engage in activities as a result of an external incentive mecha￾nism such as contingent rewards. In those cases the task is not satisfying enough and the external incentives add into the pleasure and satisfaction of the task. The following subsections discuss some examples of external incentive mechanisms. Each incentive mechanism examined below extends the interface and functional￾ity of a social system in order to increase the volume of user contributions. 2.2.1. Rewards Researchers in the filed of human–computer interaction have tried to study the effect of rewards on user contribution in online communities. For example, Bretzke and Vassileva have tried sev￾eral reward mechanisms for encouraging contributions to their re￾source-sharing system Comtella (Bretzke & Vassileva, 2003). Comtella is a resource-sharing system that allows the members of an online community share web resources amongst each other. The system rewards more cooperative users with incentives such as greater bandwidth for download or higher visibility in the com￾munity. A more recent version of Comtella uses an adaptive reward mechanism to influence the quality of participation. This new incentive mechanism only rewards high-quality participation R. Farzan, P. Brusilovsky / Computers in Human Behavior 27 (2011) 276–284 277

rather than all kinds. This is done by allowing the users to rate the mechanism, which we developed to increase the volume of student contributions of others. Ratings are averaged and negative ratings course ratings. serve to decrease the rewards given to low-quality contributions (Cheng vassileva, 2005, 2006). 3. 1. Community-based recommendation in CourseAge 2.2.2. Reputation The goal of CourseAgent is to attract user attention to courses. As suggested by Kollock(1999). reputation is an important fac- which are most relevant to their career goals motivating their study The social recommendation engine of CourseAgent attempts tor affecting motivations for community contribution. Wasko and to predict how relevant each offered course is to the career goals of Faraj(2005)surveyed the users of an electronic network of a pro- each individual student. However in contrast to traditional recom- fessional legal association to study the effect of reputation on users mender system, the recommendation power of CourseAgent is ex- articipation. They showed that people are more likely to share their knowledge when it enhances their reputation. At a basic level pressed not as a ranked list of courses, but in the form of in-context adaptive annotations. Course information is annotated with adap- nany social networking site employ reputation-based incentives tive visual cues that help students to select their most appropriate by displaying the number of connections and friends a user has courses. Fig. 1 demonstrates the use of in-CoI ntext adaptive com Other community-based systems such as Flickr address user repu tation by highlighting specific user content, such as Flickrs the munity-based annotations on the schedule page of CourseAgent. most interesting photos. Farzan et al.(2008a)experimented course, such as course number. course title, date, time, location, cial networking site to motivate employees to add content into the and information about the instructor. If the student finds a specific course relevant and interesting, they can use a system provided site. Their experiment indicates that employees are motivated by link to register for this course or to plan to pursue this in the future both reward and reputation within their test platform. Further- ("Action"column). To help the student with registering and plan the site inspired other users to visit more and comment more ning decisions, the system augments each link with two kinds of community-based annotation displayed as icons to the left of the links. One icon expresses the expected course workload (one sho- 2. 2.3. Personal needs vel for low, two for average, and three for a high workload ). The A substantial number of works in the field of economic and other icon expresses the expected relevance of the course to the ca- cial psychology have shown that extrinsic motivations such as re- reer goals of the given student( from one thumb up for a relevant Koestner,&Ryan, 1999). In this light, an incentive approach based load and relevance of a specific course is calculated using commu on addressing user personal needs could be considered as an alter- nity feedback about past offerings of this course, as taught by the native to the mechanisms discussed above. The idea and the chal- same instructor as indicated in the schedule. In addition, the stu- lenge of this approach is to turn user contribution into an activity dent's advisor can directly recommend a course as relevant. This that is both attractive and meaningful for the user. In some sense, information is represented by an"instructor"icon in the relevance this approach bridges the gap between extrinsic and intrinsic moti- vations instead of causing a conflict between them. This approach Similar community-based recommendations are provided in is less explored in the literature. An example of the personal needs the Course Catalog section of the system. In Course Catalog.courses are grouped by areas of study defined by the program as shown in students feedback for course related educational materials in the Fig. 2. For example, an Information Science degree includes areas form of annotations(Farzan Brusilovsky, 2005). Several studies such as Cognitive Science, Cognitive Systems, and Mathematica in the field of education has shown that annotations turn passive and Formal Foundation. Each course in the catalog is annotated arning into active learning and can be self-beneficial (bonifazi, with social recommendation information representing the rele- Levialdi, Rizzo, Trinchese, 2002). On the other hand, students vance and workload of the course. Since different instructors might annotations can serve as an implicit indicator of interest in a re- teach the same course, the average relevance and workload of each source and importance of this resource In our system, AnnotatEd, course is based upon the average score over all instructors who we employ annotation-based social navigation support to guide taught the course in the past. students into important resources To increase the volume of stu- dent annotations, we engineered an annotation interface, which 3.2. From community feedback to social recommendations encourages students to annotate educational content by furth increasing the value of annotations for the students themselves CourseAgent provides social recommendations by collecting The work presented in this paper expands our earlier work and three kinds of information from the community of students:(a) explores an incentive mechanism based on personal need in a dif- the students self-selected career goals, (b) the students explicit ferent context In Section 3.3 we describe our approach in details. evaluation of course workload, and c) the students explicit evalu- ation of relevance of courses taken to their career goals. We have defined an extendable list of 22 career goals that cover different 3. CourseAgent ranges of careers related to the information science field. Students are able to select as many career goals as they want from the list to CourseAgent is a community-based recommender system that add to their hese goals are communicated to student provides personalized access to information about courses. Course- advisors and also serve as a basis for social recommendatio Agent was developed for students and instructors in the School of Information Sciences at the University of Pittsburgh. While the cur- ken. Students are asked to evaluate the relevance of each taken rent version is based on information about courses offered at the course to each of their career goals on a scale of 1-5 and to evalu- School, the system can easily be adopted for different programs ate the workload of the course on a scale of 1-3. Leaving the feed- by merely integrating the program-specific course data into the back is not mandatory in the system. Students are free to leave system. The following subsections present both the interface of feedback for a course taken or to evaluate the course in relation to the system, its social recommendation engine, and the incentive just one or two of their career goals. Fig 3 presents the evaluation

rather than all kinds. This is done by allowing the users to rate the contributions of others. Ratings are averaged and negative ratings serve to decrease the rewards given to low-quality contributions (Cheng & Vassileva, 2005, 2006). 2.2.2. Reputation As suggested by Kollock (1999), reputation is an important fac￾tor affecting motivations for community contribution. Wasko and Faraj (2005) surveyed the users of an electronic network of a pro￾fessional legal association to study the effect of reputation on users participation. They showed that people are more likely to share their knowledge when it enhances their reputation. At a basic level many social networking site employ reputation-based incentives by displaying the number of connections and friends a user has. Other community-based systems such as Flickr address user repu￾tation by highlighting specific user content, such as Flickrs the most interesting photos. Farzan et al. (2008a) experimented point-based reward and reputation incentives in an enterprize so￾cial networking site to motivate employees to add content into the site. Their experiment indicates that employees are motivated by both reward and reputation within their test platform. Further￾more, they found evidence that the increase in contributions to the site inspired other users to visit more and comment more. 2.2.3. Personal needs A substantial number of works in the field of economic and so￾cial psychology have shown that extrinsic motivations such as re￾ward and reputation undermines intrinsic motivations (Deci, Koestner, & Ryan, 1999). In this light, an incentive approach based on addressing user personal needs could be considered as an alter￾native to the mechanisms discussed above. The idea and the chal￾lenge of this approach is to turn user contribution into an activity that is both attractive and meaningful for the user. In some sense, this approach bridges the gap between extrinsic and intrinsic moti￾vations instead of causing a conflict between them. This approach is less explored in the literature. An example of the personal needs approach can be provided by our our earlier work on encouraging students feedback for course related educational materials in the form of annotations (Farzan & Brusilovsky, 2005). Several studies in the field of education has shown that annotations turn passive learning into active learning and can be self-beneficial (Bonifazi, Levialdi, Rizzo, & Trinchese, 2002). On the other hand, students’ annotations can serve as an implicit indicator of interest in a re￾source and importance of this resource. In our system, AnnotatEd, we employ annotation-based social navigation support to guide students into important resources. To increase the volume of stu￾dent annotations, we engineered an annotation interface, which encourages students to annotate educational content by further increasing the value of annotations for the students themselves. The work presented in this paper expands our earlier work and explores an incentive mechanism based on personal need in a dif￾ferent context. In Section 3.3 we describe our approach in details. 3. CourseAgent CourseAgent is a community-based recommender system that provides personalized access to information about courses. Course￾Agent was developed for students and instructors in the School of Information Sciences at the University of Pittsburgh. While the cur￾rent version is based on information about courses offered at the School, the system can easily be adopted for different programs by merely integrating the program-specific course data into the system. The following subsections present both the interface of the system, its social recommendation engine, and the incentive mechanism, which we developed to increase the volume of student course ratings. 3.1. Community-based recommendation in CourseAgent The goal of CourseAgent is to attract user attention to courses, which are most relevant to their career goals motivating their study. The social recommendation engine of CourseAgent attempts to predict how relevant each offered course is to the career goals of each individual student. However, in contrast to traditional recom￾mender system, the recommendation power of CourseAgent is ex￾pressed not as a ranked list of courses, but in the form of in-context adaptive annotations. Course information is annotated with adap￾tive visual cues that help students to select their most appropriate courses. Fig. 1 demonstrates the use of in-context adaptive com￾munity-based annotations on the schedule page of CourseAgent. The schedule provides various information about each offered course, such as course number, course title, date, time, location, and information about the instructor. If the student finds a specific course relevant and interesting, they can use a system provided link to register for this course or to plan to pursue this in the future (‘‘Action” column). To help the student with registering and plan￾ning decisions, the system augments each link with two kinds of community-based annotation displayed as icons to the left of the links. One icon expresses the expected course workload (one sho￾vel for low, two for average, and three for a high workload). The other icon expresses the expected relevance of the course to the ca￾reer goals of the given student (from one thumb up for a relevant course to three for a highly relevant course). The estimated work￾load and relevance of a specific course is calculated using commu￾nity feedback about past offerings of this course, as taught by the same instructor as indicated in the schedule. In addition, the stu￾dent’s advisor can directly recommend a course as relevant. This information is represented by an ‘‘instructor” icon in the relevance column. Similar community-based recommendations are provided in the Course Catalog section of the system. In Course Catalog, courses are grouped by areas of study defined by the program as shown in Fig. 2. For example, an Information Science degree includes areas such as Cognitive Science, Cognitive Systems, and Mathematical and Formal Foundation. Each course in the catalog is annotated with social recommendation information representing the rele￾vance and workload of the course. Since different instructors might teach the same course, the average relevance and workload of each course is based upon the average score over all instructors who taught the course in the past. 3.2. From community feedback to social recommendations CourseAgent provides social recommendations by collecting three kinds of information from the community of students: (a) the students self-selected career goals, (b) the students explicit evaluation of course workload, and (c) the students explicit evalu￾ation of relevance of courses taken to their career goals. We have defined an extendable list of 22 career goals that cover different ranges of careers related to the information science field. Students are able to select as many career goals as they want from the list to add to their profile. These goals are communicated to student advisors and also serve as a basis for social recommendations. The system provides an interface to evaluate courses already ta￾ken. Students are asked to evaluate the relevance of each taken course to each of their career goals on a scale of 1–5 and to evalu￾ate the workload of the course on a scale of 1–3. Leaving the feed￾back is not mandatory in the system. Students are free to leave no feedback for a course taken or to evaluate the course in relation to just one or two of their career goals. Fig. 3 presents the evaluation 278 R. Farzan, P. Brusilovsky / Computers in Human Behavior 27 (2011) 276–284

R Farzan, P. Brusilovsky Computers in Human Behavior 27(2011)276-284 Schedule of spring 2009 raken Courses, Planned courses, currently Taken Courses, Bs: Recommend by Advisor, d: Degree of Relevance to Career Goals cRN Course No Title Duration Day Time Location Instructor Workload Relevance Action INESCL2000 itro to Informaton Science 00:50 Paul Munro Evaluate It uehu|1:002:15p 005;50 Marek oruzdzel Evaluate t 6460NE2140 Vagina He Planit 6462| INESCL2300 procas information mester Monday 6: 00-8: 50( 411 stephen Hitle c 6E Evaluate It +INESCL2350 Human Factors in Systems Michael Lewis abal Reasoning for GIs mester wednesday 3: 00-5: 50P s405 stephen Hile Eae planT 自自自 LEsCL2SD0 ata structure 6:00-:50P Roger Enn Plan It lhescisu lessa e 自t nologies and 50P 自dd 31224 INFSCI 2591 Algorithm Design semester Monday 3: 00-5: 50P Is 501 Roger Flynn 20 Developing Secure Systems sem 1:002:15 iames Joshi 26484INESCL2621 Securty Management uesday 6: 00-8:50P anced Topics in nesday 6: 00-8: 50P 411 INESC12731E-Commerce Secunty semester Monday 6: 00P-8: 50 Is 502 Michael ning Fig 1. Community-based recommendations in Course Agent schedule of courses. please select one of the programs to view the course list: Course in IST Program List the Courses visualize (Click on each"AREA to see the list of related courses Taken Courses, Planned courses, curently Taken Courses, ES: Recommend by Advisor, b: Degree of Relevance to Career Goals Cognitive science Area Workload RelevanceAction INFSCI 2300 Human Information Processing Leave Feedback ations of Congnitive Science an Factors in Systems b图 Cognitive Systems Area Course Title workload Relevance Action INFSCI 2410 Introduction to Neutral Networks I 自自 E Leave Feedback INFSCI 2420 Natural Language Processing t自 INESCI 2440 Artificial Intelligence ood Mew.feedba INFSCI 2460 Spatial Reasoning for GIS 自 d It +Doctoral Courses +Mathematical and Formal Foundations Area Systems and Technology Area- Applications of Information Technology +Systems and Technology Area-Computing Systems Concepts Fig. 2. Community-based recommendations in CourseAgent course catalog

Fig. 1. Community-based recommendations in CourseAgent schedule of courses. Fig. 2. Community-based recommendations in CourseAgent course catalog. R. Farzan, P. Brusilovsky / Computers in Human Behavior 27 (2011) 276–284 279

R Farzan, P. Brusilovsky/Computers in Human Behavior 27 (2011)276-284 INFSCI 2470-Interactive Systems Design 1. workload of the course: is this course to each of your career goals College Professor Research in Industry Web Application Developer C C Irrelevant Relevant Relevant Very Relevant Essental omments Fig. 3. Evaluation interface in CourseAgent course catalog interface. The collected information is used to deliver adaptive ers, such as MovieLens(Miller, Albert, Lam, Konstan, ried annotations presented in the previous section. The overall work- 2003)in one important aspect: recommendations that are pro- load level of the course is computed by simply averaging all ratings vided to a specific student do not take into account her own rat- provided by the students. The relevance of a course to a student is ings, but only the ratings of students who took potentially computed based on the relevance of the course to each of the stu- interesting courses earlier. As a result, ratings provided by the stu- dents career interests. To compute total relevance, we cannot eas- dents in CourseAgent are beneficial solely to the community but ly average the relevance to all career goals of the student: a not to the author of the ratings. this typical contradictory situation worthy course might be irrelevant to most of the students career motivated us to try an incentive approach based on personal needs goals while being critical to only one goal. In this case, a simple and to find some way to turn rating courses into a self-beneficial verage will give it a poor relevance rating, while the student activity might actually be especially interested in taking the course since Therefore, our challenge has been to design an activity that is it is essentially relevant to one of their career goals. To overcome both attractive and meaningful for the students and can use course this, we designed a simple case-based algorithm to compute course ratings provided by the student for the benefit of the author of the relevance The relevance of a course to each career interest of the ratings. Since the main goal of students from taking courses is their student ranges from 1 to 5, where 1 is not relevant and 5 is very future career, connecting course rating to their career planning can relevant Courses with a relevance level of 3 and above to at least be an attractive candidate to integrate career planning with stu- ne of the students career goals contribute to the overall relevance dent course evaluation, we developed the Career Progress section of the course to the student. The relevance of the course to the stu- of the system. In Career Progress, students can view the progress dent is visualized with a thumbs-up icon (one icon means reason- they have made towards each of their career goals. Courses they able relevance and three means the highest relevance). Table 1 have taken and evaluated are used to compute their progress to- presents part of the cases in our algorithm for computing course wards the career goals. The more relevant the course to the career relevance. For example, if a course is essentially relevant (rele- goal, the more progress they will make towards the goal. Also, the vance level of 5 )in two of the students career goals, the course will difficulty level of the course will affect the progress. A low-load be considered highly relevant to the student. The complete set of course would not necessarily cause the same progress as a high rules consists of 16 cases load course. To calculate the progress, we have assumed that a spe- cific career goal can be"covered"by taking four relevant courses 3.3. Adding incentive mechanism in CourseAgent with medium level difficulty. More difficult courses with higher elevance will contribute higher progress. To take into account Similar to any other community-based adaptive system, the non-additive knowledge accumulation in courses contributing to he same goal, we used a logarithmic-style contribution function vided by the community. However, during the first semesters of (Fig. 4)instead of a lines one. With this approach, each next course CourseAgent's use, we observed that few courses were rated by less. the first course adds 40% of progress while the rest of courses few students. We hypothesized that low level of user participation add only 25%, 15%, and 10%, respectively, and taking more than stems from the nature of social recommendation engine. Course- Agent's engine differs from traditional collaborative recommend. four courses does not contribute to career progress. Fig. 5 shows a screen-shot of the Career Progress section. For Table 1 each of student 's career goals, there is a progress bar that displays Case-based algorithm for computation of course relevance. the contribution of relevant courses taken and planned towards Final relevance achieving this goal. As mentioned before, the contribution of taken career goals courses into the progress depends on student's evaluation of rele- with relevance 5 with relevance 4 with relevance 3 vance and difficulty of the course. A taken, but not evaluated course does not contribute to student progress towards a goal Since planned courses are not yet taken(and rated) by this student, the total contribution of the students' planned courses is computed from the average relevance and average difficulty level provided by Means the parameter can take any value or zero or more. the community of students. To distinguish actual progress from

interface. The collected information is used to deliver adaptive annotations presented in the previous section. The overall work￾load level of the course is computed by simply averaging all ratings provided by the students. The relevance of a course to a student is computed based on the relevance of the course to each of the stu￾dents career interests. To compute total relevance, we cannot eas￾ily average the relevance to all career goals of the student: a worthy course might be irrelevant to most of the students career goals while being critical to only one goal. In this case, a simple average will give it a poor relevance rating, while the student might actually be especially interested in taking the course since it is essentially relevant to one of their career goals. To overcome this, we designed a simple case-based algorithm to compute course relevance. The relevance of a course to each career interest of the student ranges from 1 to 5, where 1 is not relevant and 5 is very relevant. Courses with a relevance level of 3 and above to at least one of the students career goals contribute to the overall relevance of the course to the student. The relevance of the course to the stu￾dent is visualized with a thumbs-up icon (one icon means reason￾able relevance and three means the highest relevance). Table 1 presents part of the cases in our algorithm for computing course relevance. For example, if a course is essentially relevant (rele￾vance level of 5) in two of the students career goals, the course will be considered highly relevant to the student. The complete set of rules consists of 16 cases. 3.3. Adding incentive mechanism in CourseAgent Similar to any other community-based adaptive system, the success of CourseAgent is highly dependent upon the feedback pro￾vided by the community. However, during the first semesters of CourseAgent’s use, we observed that few courses were rated by few students. We hypothesized that low level of user participation stems from the nature of social recommendation engine. Course￾Agent’s engine differs from traditional collaborative recommend￾ers, such as MovieLens (Miller, Albert, Lam, Konstan, & Riedl, 2003) in one important aspect: recommendations that are pro￾vided to a specific student do not take into account her own rat￾ings, but only the ratings of students who took potentially interesting courses earlier. As a result, ratings provided by the stu￾dents in CourseAgent are beneficial solely to the community but not to the author of the ratings. This typical contradictory situation motivated us to try an incentive approach based on personal needs and to find some way to turn rating courses into a self-beneficial activity. Therefore, our challenge has been to design an activity that is both attractive and meaningful for the students and can use course ratings provided by the student for the benefit of the author of the ratings. Since the main goal of students from taking courses is their future career, connecting course rating to their career planning can be an attractive candidate. To integrate career planning with stu￾dent course evaluation, we developed the Career Progress section of the system. In Career Progress, students can view the progress they have made towards each of their career goals. Courses they have taken and evaluated are used to compute their progress to￾wards the career goals. The more relevant the course to the career goal, the more progress they will make towards the goal. Also, the difficulty level of the course will affect the progress. A low-load course would not necessarily cause the same progress as a high￾load course. To calculate the progress, we have assumed that a spe￾cific career goal can be ‘‘covered” by taking four relevant courses with medium level difficulty. More difficult courses with higher relevance will contribute higher progress. To take into account non-additive knowledge accumulation in courses contributing to the same goal, we used a logarithmic-style contribution function (Fig. 4) instead of a lines one. With this approach, each next course taken towards the same career progress contribute to that career less. The first course adds 40% of progress while the rest of courses add only 25%, 15%, and 10%, respectively, and taking more than four courses does not contribute to career progress. Fig. 5 shows a screen-shot of the Career Progress section. For each of student’s career goals, there is a progress bar that displays the contribution of relevant courses taken and planned towards achieving this goal. As mentioned before, the contribution of taken courses into the progress depends on student’s evaluation of rele￾vance and difficulty of the course. A taken, but not evaluated course does not contribute to student progress towards a goal. Since planned courses are not yet taken (and rated) by this student, the total contribution of the students’ planned courses is computed from the average relevance and average difficulty level provided by the community of students. To distinguish actual progress from Fig. 3. Evaluation interface in CourseAgent course catalog. Table 1 Case-based algorithm for computation of course relevance. Number of career goals with relevance 5 Number of career goals with relevance 4 Number of career goals with relevance 3 Final relevance P2 3 1 >1 3 010 1 002 1 * Means the parameter can take any value or zero or more. 280 R. Farzan, P. Brusilovsky / Computers in Human Behavior 27 (2011) 276–284

R Farzan, P. Brusilovsky Computers in Human Behavior 27(2011)276-284 experimental group. To compare the contribution of control and experimental group we calculated number of courses taken, courses planned, career goals, and ratings entered into the system by each group. Table 2 shows the result. The result is compatible with our expectation and shows that experimental group contrib- uted more into the system. They added more information about courses taken, and planned, and their career goal and they rate larger number of courses. While analyzing the data we noticed that some users in the experimental group never clicked on career progress page. The ca- eer progress feature was not advertised in any specific way. As result even though the feature was available to all users, some users apparently have not noticed it since they have not accessed the career progress page even once. To evaluate the real effect of Course taken the feature we decided to compare the contribution of users who Fig 4. Contribution of each course taken in calculation of career progress. at least visited career progress once with those who did not use it at all whether in control or experimental group Table 3 show planned progress, the contribution of planned courses is shown in the result of this comparison. The data shows that the difference the progress bar with a different color(blue for planned courses of the group is much more visible and it suggests that career pre and brown for taken courses). gress is successful in encouraging users' contribution. These result are considered preliminary and due to the small ned in details by expanding a career goal as shown in Fig. 5. An ex- number of subjects. we did not conduct any statistical analys panded career goal lists three possible groups of courses: taken. As a result, it is difficult to draw any reliable conclusions; never planned, and recommended. The students are able to see their theless, the results are very encouraging. This has motivated our wn evaluation of taken courses taken but not evaluated courses longer term evaluation. Since we believe career progress is an mportant feature of the system, we decided to make it available are presented in the Taken Courses table with a lighter background. to all users and in our long term evaluation study the correlation This prompts the students to evaluate the course (using the link to of usage of it and contribution to the site. Section 4.2 presents goal. They are also able to view the community s evaluation of their our long-term evaluation in details. planned courses, as rated by relevancy to each specific career goal. 4.2. Long-term evaluation The list of recommended courses(based on the community's eval ation) is provided for each specific career goal and students are able to plan any of the recommended courses. Once our preliminary stage evaluation was over, we opened the career progress feature to all students and advertised the usage of he system among all graduate students at the School of Informa- tion Sciences at University of Pittsburgh. We were interested in assessing the effect of our incentive mechanism on long-term real To study the impact of our incentive mechanism on user contri usage of the system. The system was advertised before each bution in CourseAgent, we conducted two series of evaluations at semester registration. The system has been used for three years the School of Information Sciences at the University of Pittsburgh. Table 4 shows the general statistics about the usage of the system At the first stage, we conducted a preliminary study to assess the over three years. We separated data for Masters and Ph.D. students motivating effect of career progress report feature by opening the since they have different goals from taking courses For Masters feature to only half of the students. The preliminary study includes students courses contribute much stronger into their career goals one semester data from 20 graduate students. It compared the and they plan their courses according to their career goals while behavior of students who had access to this feature with the con- for Ph D students research plays the most important role and trol group, which had no access to career progress. We hypothe- courses have small contribution towards their career goals.Given sized that the ability to track career progress encourages the fact that the incentive mechanism of the system relies on rel- students to rate more courses. At the second stage, we were inter- evancy of courses to students' career goal, we focus the rest of eval- ested on evaluating the effect of career scope on long-term usage of uation on masters students the system. On this stage the career progress feature was available As it can be seen in Table 4. overall about 23% of students have at to all students. The longitudinal study includes data from 3 years least rated one course. While this seems a low number it is in fact usage of 171 graduate students. CourseAgent is an opt-in system higher than average contribution rate in online communities. So and the usage of the system was optional throughout both studies. overall the system has achieved less inequality of contribution and higher percentage of users are contributing to the rating of courses. To analyze the effect of career progress page on students' navi- gation behavior we looked at the correlation between number of For the preliminary evaluation, the system was advertised to times students have looked at career graduate students of the School of Information Sciences 2 weeks of ratings they have provided. We divided the students into two prior to registration deadline, when we expected high demand groups depending on whether they have visited career progress for its services. When a student requested to use the system, they page or not. In the rest of the paper, we call the group who have vere randomly assigned to either control or experimental group visited career progress at least once as CP and the group who have Throughout the period of the study, 20 students used the sys- not visited career progress page No-CP. Since rating is only possi- tem. Eleven students were assigned to control group and 9 to the ble after student have entered their courses taken and career goals. in the analysis we have included only those students who have at 1 For interpretation of the references to color in this figure, the reader is referred to least one course taken and one career goal selected. That includes the web version of this article 76 students in total, 32 in CP group and 44 in No-CP group

planned progress, the contribution of planned courses is shown in the progress bar with a different color (blue1 for planned courses, and brown for taken courses). The career progress visualized by the progress bar can be exam￾ined in details by expanding a career goal as shown in Fig. 5. An ex￾panded career goal lists three possible groups of courses: taken, planned, and recommended. The students are able to see their own evaluation of taken courses. Taken but not evaluated courses are presented in the Taken Courses table with a lighter background. This prompts the students to evaluate the course (using the link to the right) in order to be count as a part of progress toward the career goal. They are also able to view the community’s evaluation of their planned courses, as rated by relevancy to each specific career goal. The list of recommended courses (based on the community’s evalu￾ation) is provided for each specific career goal and students are able to plan any of the recommended courses. 4. Evaluation To study the impact of our incentive mechanism on user contri￾bution in CourseAgent, we conducted two series of evaluations at the School of Information Sciences at the University of Pittsburgh. At the first stage, we conducted a preliminary study to assess the motivating effect of career progress report feature by opening the feature to only half of the students. The preliminary study includes one semester data from 20 graduate students. It compared the behavior of students who had access to this feature with the con￾trol group, which had no access to career progress. We hypothe￾sized that the ability to track career progress encourages students to rate more courses. At the second stage, we were inter￾ested on evaluating the effect of career scope on long-term usage of the system. On this stage, the career progress feature was available to all students. The longitudinal study includes data from 3 years usage of 171 graduate students. CourseAgent is an opt-in system and the usage of the system was optional throughout both studies. 4.1. Preliminary evaluation For the preliminary evaluation, the system was advertised to graduate students of the School of Information Sciences 2 weeks prior to registration deadline, when we expected high demand for its services. When a student requested to use the system, they were randomly assigned to either control or experimental group. Throughout the period of the study, 20 students used the sys￾tem. Eleven students were assigned to control group and 9 to the experimental group. To compare the contribution of control and experimental group we calculated number of courses taken, courses planned, career goals, and ratings entered into the system by each group. Table 2 shows the result. The result is compatible with our expectation and shows that experimental group contrib￾uted more into the system. They added more information about courses taken, and planned, and their career goal and they rated larger number of courses. While analyzing the data we noticed that some users in the experimental group never clicked on career progress page. The ca￾reer progress feature was not advertised in any specific way. As a result even though the feature was available to all users, some users apparently have not noticed it since they have not accessed the career progress page even once. To evaluate the real effect of the feature we decided to compare the contribution of users who at least visited career progress once with those who did not use it at all whether in control or experimental group. Table 3 show the result of this comparison. The data shows that the difference of the group is much more visible and it suggests that career pro￾gress is successful in encouraging users’ contribution. These result are considered preliminary and due to the small number of subjects, we did not conduct any statistical analysis. As a result, it is difficult to draw any reliable conclusions; never￾theless, the results are very encouraging. This has motivated our longer term evaluation. Since we believe career progress is an important feature of the system, we decided to make it available to all users and in our long term evaluation study the correlation of usage of it and contribution to the site. Section 4.2 presents our long-term evaluation in details. 4.2. Long-term evaluation Once our preliminary stage evaluation was over, we opened the career progress feature to all students and advertised the usage of the system among all graduate students at the School of Informa￾tion Sciences at University of Pittsburgh. We were interested in assessing the effect of our incentive mechanism on long-term real usage of the system. The system was advertised before each semester registration. The system has been used for three years. Table 4 shows the general statistics about the usage of the system over three years. We separated data for Masters and Ph.D. students since they have different goals from taking courses. For Masters students courses contribute much stronger into their career goals and they plan their courses according to their career goals while for Ph.D. students research plays the most important role and courses have small contribution towards their career goals. Given the fact that the incentive mechanism of the system relies on rel￾evancy of courses to students’ career goal, we focus the rest of eval￾uation on Masters students. As it can be seen in Table 4, overall about 23% of students have at least rated one course. While this seems a low number, it is in fact higher than average contribution rate in online communities. So overall the system has achieved less inequality of contribution and higher percentage of users are contributing to the rating of courses. To analyze the effect of career progress page on students’ navi￾gation behavior we looked at the correlation between number of times students have looked at career progress page and number of ratings they have provided. We divided the students into two groups depending on whether they have visited career progress page or not. In the rest of the paper, we call the group who have visited career progress at least once as CP and the group who have not visited career progress page NO-CP. Since rating is only possi￾ble after student have entered their courses taken and career goals, in the analysis we have included only those students who have at least one course taken and one career goal selected. That includes 76 students in total, 32 in CP group and 44 in NO-CP group. Fig. 4. Contribution of each course taken in calculation of career progress. 1 For interpretation of the references to color in this figure, the reader is referred to the web version of this article. R. Farzan, P. Brusilovsky / Computers in Human Behavior 27 (2011) 276–284 281

282 R Farzan, P Brusilovsky/ Computers in Human Behavior 27(2011 )276-284 progress( Taken, P anned, Recommended -Research in Industry Course Number□ Course Title My Ratin「 Action tro to Informaton Science JINESCL 2130 Decision Analysis and Decision Support Systems Evalua 2300 Human Information Processing ateIt! SCI 2410 Introduction to Neutral Networks Evaluate! INESCL 2931 Special Topics: Mathematics Planned Courses Course Number Course Title Community RatingAction INESCL 2140 Information Storage and Retrieval Recommended Courses Course Title INESCL 2020 Mathematical Fo ndations for Information Science rtrtrtr INESCI2040 Fig. 5. Career progress interface in CourseAgent. Table 2 page and number of ratings they provided. We expected that the omparison of contribution of users in control an nental group. more visit to career progress page correlates with higher number Mean course Mean course Mean career Mean of of ratings since career progress encourages rating and is dependent on ratings. Fig. 7 shows the average number of ratings provided by Control each group. Nonparametric test of correlation of career progress visits and number of ratings shows a significant positive correla tion(Spearmans p= 27, 2-tailed p-value=0.04). However, it can be argued that students who visited career pro- gress page are generally active contributors on the site and are Comparison of contribution of users using Career progress. generally more active on the site. To verify that we compared their Mean course ac Mean career Mean of activities in terms of number of course taken number of course planned, and number of career goals. The result is shown in Ing Career 187 4.33 Fig 8 Mann-Whitney test shows no significant difference between CP and No-CP groups(Zaken=-1. 23 and 2-tailed p-value=. 22, areer progress 8.8 34 and 2-tailed p-value =.73, Z, ergas =-.49 and p-value=. 62). The result suggests that the general activity of CP and No-CP groups in Course Agent is comparable, while the vol- Table 4 ume of their course rating activity is significantly different. General statistics of usage of the system. The result of our analysis suggests that our proposed incentive mechanism is successful in encouraging users to rate more courses. We have observed that extrinsic motivations are activated when Used the Adde personal needs are targeted. Thus we confirmed the ability of yet system taken another incentive mechanism to effectively change users'behavior However. can we assume that our incentive mechanism affects MS143 65%)105(73%) 33(23%) Phd 7(61‰ users' behavior only in positive direction? In the next section we discuss some possible problems of changing users'behavior by incentives and examine possible negative effects of our incentive 4. 1. Percentage of students rating mechanism on CourseAgent users First we compare the percentage of students in each group who rated any courses. Fig. 6 shows that higher percentage of students 5. Problems and drawbacks of incentive systems CP group rated courses(70% versus 47%). Chi-square test of equality of proportions suggest marginal significant difference of percentage of CP versus NO-CP students who rated courses(Pea While there has been wide rage of research on encouraging user son 72=2.71, df=1, 2-sided p-value =0.09). participation in online communities, little has been done on study- ing the drawbacks of incentive systems. As mentioned in Section 2. 2.3, a possible drawback of extrinsic motivation is reducing inter 4.2.2. Correlation of usage of career progress and number of ratings nal incentives. Another easily anticipated drawback of incentive Next we were interested to evaluate whether there is a correla- systems is gaming. Users are known to game various incentive sys- tion between the number of times a student visited career progress tems to achieve higher reputation or rewards. Gaming is a serious

4.2.1. Percentage of students rating First we compare the percentage of students in each group who rated any courses. Fig. 6 shows that higher percentage of students in CP group rated courses (70% versus 47%). Chi-square test of equality of proportions suggest marginal significant difference of percentage of CP versus NO-CP students who rated courses (Pear￾son v2 ¼ 2:71, df = 1, 2-sided p-value = 0.09). 4.2.2. Correlation of usage of career progress and number of ratings Next we were interested to evaluate whether there is a correla￾tion between the number of times a student visited career progress page and number of ratings they provided. We expected that the more visit to career progress page correlates with higher number of ratings since career progress encourages rating and is dependent on ratings. Fig. 7 shows the average number of ratings provided by each group. Nonparametric test of correlation of career progress visits and number of ratings shows a significant positive correla￾tion (Spearman’s q ¼ :27, 2-tailed p-value = 0.04). However, it can be argued that students who visited career pro￾gress page are generally active contributors on the site and are generally more active on the site. To verify that we compared their activities in terms of number of course taken, number of course planned, and number of career goals. The result is shown in Fig. 8. Mann–Whitney test shows no significant difference between CP and NO-CP groups (Ztaken ¼ 1:23 and 2-tailed p-value = .22, Zmhboxplanned ¼ :34 and 2-tailed p-value = .73, Zmhboxcareergoals ¼ :49 and p-value = .62). The result suggests that the general activity of CP and NO-CP groups in CourseAgent is comparable, while the vol￾ume of their course rating activity is significantly different. The result of our analysis suggests that our proposed incentive mechanism is successful in encouraging users to rate more courses. We have observed that extrinsic motivations are activated when personal needs are targeted. Thus we confirmed the ability of yet another incentive mechanism to effectively change users’ behavior. However, can we assume that our incentive mechanism affects users’ behavior only in positive direction? In the next section we discuss some possible problems of changing users’ behavior by incentives and examine possible negative effects of our incentive mechanism on CourseAgent users. 5. Problems and drawbacks of incentive systems While there has been wide rage of research on encouraging user participation in online communities, little has been done on study￾ing the drawbacks of incentive systems. As mentioned in Section 2.2.3, a possible drawback of extrinsic motivation is reducing inter￾nal incentives. Another easily anticipated drawback of incentive systems is gaming. Users are known to game various incentive sys￾tems to achieve higher reputation or rewards. Gaming is a serious Fig. 5. Career progress interface in CourseAgent. Table 2 Comparison of contribution of users in control and experimental group. Mean course taken Mean course planned Mean career goal Mean of ratings Control 5 2 0.91 4.55 Experimental 5.89 5 2.2 6.22 Table 3 Comparison of contribution of users using or not using career progress. Mean course taken Mean course planned Mean career goal Mean of ratings Not using career progress 4.27 1.87 1 4.33 Using career progress 8.8 7 3 8.2 Table 4 General statistics of usage of the system. Number of students Used the system Added taken courses Added planned courses Added career goals Added course ratings MS 143 93 (65%) 105 (73%) 76 (53%) 33 (23%) Ph.D. 28 25 (89%) 17 (61%) 22 (79%) 15 (54%) 282 R. Farzan, P. Brusilovsky / Computers in Human Behavior 27 (2011) 276–284

R Farzan, P. Brusilovsky Computers in Human Behavior 27(2011)276-284 E NO-CP C a no rating Fig. 8. Average number of courses taken, courses planned, and career gaols provided by students in CP and NO-CP group. tion of the effect of the incentive mechanism in CourseAgent Fig. 6. Percentage of students rating courses. students'self-deception. concern for all kinds of social systems since gaming leads to lot 6. Positive rating bias in CourseAgent quality contributions, which in turn can discourage other users from using the system. Cheng and vassileva observed that enhanc. An example of self-deception, which can be provoked by our ing users' status as a response to their contribution could motivate incentive approach is what we call "positive rating bias". The more users in adding low-quality resources and inaccurate ratings relevant a course is rated to student's career goal, the more pro- nism to prevent"gammers'" Farzan et al. (2008b)showed that a dents may have an implicit motivation to rate courses higher in point-based reward and reputation incentive for encouraging con- order to attain higher visible progress. The positive rating bias tribution to a social networking site motivated several different may not be as destructive as gaming. but it is still not harmless. gaming behaviors. Some users started gaming the incentive syster Artificially increase course ratings may affect systems recomm to criticize the validity of the reward based contributions. Others dations encouraging students to take courses, which are not as rel- tried to earn points by adding fake content or automatically gener- evant to their goals as shown by the system ated content. Preventing gaming stays a challenge for incentive To explore the presence of the positive rating bias in Course- systems specially those based on reputation, rewards, and inter Agent, we analyzed the correlation of average rating value and personal motivations(Ellis, Halverson, Erickson, 2005). the number of visits to career progress page. Fig 9 shows the aver- ge value of ratings given by students in CP and No-CP group The mmune to direct gaming, they may be subject to another problem: result suggests that CP students are likely to give higher ratings to self-deception. Balcetis defines self-deception as"process of ignor- courses. Nonparametric test of correlation of career progress visits ng, rationalizing, or manipulating some thought or behavior to and average ratings shows a significant positive correlation (Spear- reate consistency between that thought or behavior and one's nan s p=44, 2-tailed p-value=0.03). sense of self(Balcetis, 2008). Balcetis discusses that motivation While these results may not be considered as a definite proof of nfluence cognition in four different ways by biasing perception the positive rating bias, the evidence is strong enough to consider and memory retrieval. She suggests that these motivational biases incentive mechanisms with care, taking into account both positive make self-deception successful. Unlike gaming, the effect of motivations in online communities on self-deception has not been explored The incentive mechanism 7. Discussion and future work in CourseAgent capitalizes on individual motivation and personal eeds. We anticipated a potential adverse effect of the incentive Social recommender systems and other kinds of social software n students' rating of courses Next section describes our explora- are highly dependent on the users'feedback and participation. The 12 0 Fig. 7. Average number of ratings provided by students in CP and No-CP group. Fig 9. Average value of ratings given by students in CP and NO-CP group

concern for all kinds of social systems since gaming leads to low quality contributions, which in turn can discourage other users from using the system. Cheng and Vassileva observed that enhanc￾ing users’ status as a response to their contribution could motivate users in adding low-quality resources and inaccurate ratings (Cheng & Vassileva, 2006). They had to adapt their reward mecha￾nism to prevent ‘‘gammers”. Farzan et al. (2008b) showed that a point-based reward and reputation incentive for encouraging con￾tribution to a social networking site motivated several different gaming behaviors. Some users started gaming the incentive system to criticize the validity of the reward based contributions. Others tried to earn points by adding fake content or automatically gener￾ated content. Preventing gaming stays a challenge for incentive systems specially those based on reputation, rewards, and inter￾personal motivations (Ellis, Halverson, & Erickson, 2005). While individual motivations based on personal needs can be immune to direct gaming, they may be subject to another problem: self-deception. Balcetis defines self-deception as ‘‘process of ignor￾ing, rationalizing, or manipulating some thought or behavior to create consistency between that thought or behavior and one’s sense of self” (Balcetis, 2008). Balcetis discusses that motivations influence cognition in four different ways by biasing perception of information, attention to information, processing of information, and memory retrieval. She suggests that these motivational biases make self-deception successful. Unlike gaming, the effect of motivations in online communities on self-deception has not been explored. The incentive mechanism in CourseAgent capitalizes on individual motivation and personal needs. We anticipated a potential adverse effect of the incentive on students’ rating of courses. Next section describes our explora￾tion of the effect of the incentive mechanism in CourseAgent on students’ self-deception. 6. Positive rating bias in CourseAgent An example of self-deception, which can be provoked by our incentive approach is what we call ‘‘positive rating bias”. The more relevant a course is rated to student’s career goal, the more pro￾gress will be contributed towards the goal. With this design, stu￾dents may have an implicit motivation to rate courses higher in order to attain higher visible progress. The positive rating bias may not be as destructive as gaming, but it is still not harmless. Artificially increase course ratings may affect system’s recommen￾dations encouraging students to take courses, which are not as rel￾evant to their goals as shown by the system. To explore the presence of the positive rating bias in Course￾Agent, we analyzed the correlation of average rating value and the number of visits to career progress page. Fig. 9 shows the aver￾age value of ratings given by students in CP and No-CP group. The result suggests that CP students are likely to give higher ratings to courses. Nonparametric test of correlation of career progress visits and average ratings shows a significant positive correlation (Spear￾man’s q ¼ :44, 2-tailed p-value = 0.03). While these results may not be considered as a definite proof of the positive rating bias, the evidence is strong enough to consider the incentive mechanism based on personal needs, as well as other incentive mechanisms with care, taking into account both positive and negative effects, which it may cause. 7. Discussion and future work Social recommender systems and other kinds of social software are highly dependent on the users’ feedback and participation. The Fig. 6. Percentage of students rating courses. Fig. 7. Average number of ratings provided by students in CP and NO-CP group. Fig. 8. Average number of courses taken, courses planned, and career gaols provided by students in CP and NO-CP group. Fig. 9. Average value of ratings given by students in CP and NO-CP group. R. Farzan, P. Brusilovsky / Computers in Human Behavior 27 (2011) 276–284 283

need to attract a high quality and volume of participation has been Bonifazi, F. Levialdi, S, Rizzo, P-& Trinchese, R(2002). Uca: A web-based annotation tool supporting e-learning In Working conference on advanced visual increasing interest in both developing various incentive mecha Bretzke, H, Vassileva, J(2003). Motivating cooperation in peer to peer networks nisms and studying the impact of these mechanisms on user In Proc of international conference on user modeling(VoL. 2702, pp. 218-227). ehavior. Over the last few years, several incentive mechanisms vere explored in the context of different social systems. In our Cheng, R,& Vassileva, J(2005). User motivation and persua work we explored a new type of incentive mechanism based on system sciences (mini-track "online communities in the digital economy")(pp. users'personal needs to increase the volume of user feedback about taken courses. In CourseAgent, this mechanism was imple- Cheng, R, vassileva, 3(2006). Design and evaluation of an adaptive incentive es. Journal of User track their progress towards career goals. Our results suggest that Deci upporting CoMia user-Adapted Interac mented through career goal interface, which turned course rating into a valuable activity for the users by allowing them to better ds on intrinsic motivation. American this specific incentive mechanism can significantly affect the ehavior of Course Agent users. Users who did not use career pro- Ellis J Halverson, C& Erickson, T(2005)-Sustainingo gress feature of CourseAgent rated significantly fewer courses even though the average activity of both groups of users was compara- Emerson, R. M.(1976) Social exchange theory Annual Review of Sociology, 2 ble. However, we also observed that an incentive based on personal 335-362. needs motivates self-deception which causes positive rating bias il Farzan, R,& Brusilovsky, P (2005). Social navigation support thr this case. Users who used career progress feature more were more based group modeling. In Proc of 10th international user modeling conference ikely to rate courses higher. This result along with previous re- a (2008a) Results from deploying a participation incentive mechanism within tives on intrinsic ones hints that incentive systems should be employed with careful consideration of possible drawbacks Differ- Farzan, R. Dimiccotingmsmmsepp 563-572)1 CM ent incentive mechanisms trigger different side effects and are 2008b). when the experiment is over: Deploying ntive system to subject to different problems Understanding positive and negative sers. In Proc. of persuasive technology symposium, in conjunction with the Al nfluencesofthesemechanismsonuserbehaviorisimportantflIckr.Availableatchttp://www.flickr.com>. while designing and deploying any incentive mechanism. Harper, M.L. Y,& Konstan, (2005). An economic model of user rating in We are interested in deeper analysis of the positive rating bias an online recommender system. In User modeling (pp. 307-316). Berlin, understand users'behavior better. From our current evaluation, Herlocker, 1. L Konstan, I.A, Terveen, L. G,& Riedl, J. T(2004). Evaluating vated the users to think more carefully about their ratings. User Kar ab it is not clear whether the impact of our incentive mechanism made ratings artificially higher or whether, instead, it just moti- williams, K (1993). Social loafing: A meta-analytic review an theoretical integration. Joumal of personality and Social Psychology terviews and questionnaires can help to conduct a more pro- found analysis. Deeper evaluation will inform us to modify the Kollock, P (1999). The economies of online cooperation: gifts and public incentive mechanism in CourseAgent to minimize the positive rat cyberspace In M. mith P Kollock(Eds ) Communities in cyberspace. ngbiasWearealsointerestedonexploringdifferentincentiveLinkedinAvailableatchttp://www.linkedin.com>. mechanisms known in the literature and studying their effects Locke, E.A.& Latham, G P(2002). Building a practically useful theory of goal on users'behavior. Following collective effort model, we would like th to increase user awareness about the importance of their contribu- Lui, S, Lang, K& Kwok, S (2002). Parti tion for themselves and for the community and assess its effect on ouraging users contribution and self-deception. Miller, B, Albert, L, Lam, SK, Konstan, -& Riedl J(2003). Movielens knowledgement sXh entenesi nat a ference en inr lisem se interfaces e e. 26e-26 This research is partially supported by the National Science alertbox.avAilableat<http://www.useit.com Foundation through Grant IIS-0447083 and Graduate Research Fel- Rashid, A. M, Ling. K Tassone, R. D. Resnick, P- Kraut, R ,& riedl, lowship to R Farzan Motivating participation by displaying the value of contribution. USA: ACM Pr References Schafer, J. B, Frankowski, D, Herlocker, ],& Sen, S(2007) Collaborat :ivated cognition aceomplishesnsels-idec pido ,e social tod personal tion(Vol. 4321). Spr Wasko, M.& Faraj, S(2005) Why should ology Compass, 2(1), 361-381 knowledge contribution in electronic networks of practice. MiS Quarter WikipediaAvailableat<http://en.wikipedia.org/wiki/wikipedia:About. re work (pp 212-221) New York, NY, USA: ACM Press

need to attract a high quality and volume of participation has been recognized by both researchers and practitioners and caused an increasing interest in both developing various incentive mecha￾nisms and studying the impact of these mechanisms on user behavior. Over the last few years, several incentive mechanisms were explored in the context of different social systems. In our work we explored a new type of incentive mechanism based on users’ personal needs to increase the volume of user feedback about taken courses. In CourseAgent, this mechanism was imple￾mented through career goal interface, which turned course rating into a valuable activity for the users by allowing them to better track their progress towards career goals. Our results suggest that this specific incentive mechanism can significantly affect the behavior of CourseAgent users. Users who did not use career pro￾gress feature of CourseAgent rated significantly fewer courses even though the average activity of both groups of users was compara￾ble. However, we also observed that an incentive based on personal needs motivates self-deception which causes positive rating bias in this case. Users who used career progress feature more were more likely to rate courses higher. This result along with previous re￾search on gaming incentive systems and effect of extrinsic incen￾tives on intrinsic ones hints that incentive systems should be employed with careful consideration of possible drawbacks. Differ￾ent incentive mechanisms trigger different side effects and are subject to different problems. Understanding positive and negative influences of these mechanisms on user behavior is important while designing and deploying any incentive mechanism. We are interested in deeper analysis of the positive rating bias to understand users’ behavior better. From our current evaluation, it is not clear whether the impact of our incentive mechanism made ratings artificially higher or whether, instead, it just moti￾vated the users to think more carefully about their ratings. User interviews and questionnaires can help to conduct a more pro￾found analysis. Deeper evaluation will inform us to modify the incentive mechanism in CourseAgent to minimize the positive rat￾ing bias. We are also interested on exploring different incentive mechanisms known in the literature and studying their effects on users’ behavior. Following collective effort model, we would like to increase user awareness about the importance of their contribu￾tion for themselves and for the community and assess its effect on encouraging users contribution and self-deception. Acknowledgement This research is partially supported by the National Science Foundation through Grant IIS-0447083 and Graduate Research Fel￾lowship to R. Farzan. References Balcetis, E. (2008). Where the motivation resides and self-deception hides: How motivated cognition accomplishes self-deception. Social and Personality Psychology Compass, 2(1), 361–381. Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., et al. (2004). Using social psychology to motivate contributions to online communities. In Proc. on international conference on computer supported collaborative work (pp. 212–221). New York, NY, USA: ACM Press. Bonifazi, F., Levialdi, S., Rizzo, P., & Trinchese, R. (2002). Uca: A web-based annotation tool supporting e-learning. In Working conference on advanced visual interfaces: Avi 2002 (pp. 123–128). New York, NY, USA: ACM Press. Bretzke, H., & Vassileva, J. (2003). Motivating cooperation in peer to peer networks. In Proc. of international conference on user modeling (Vol. 2702, pp. 218–227). Springer Verlag LNCS. Cheng, R., & Vassileva, J. (2005). User motivation and persuasion strategy for peer￾to-peer communities. In Proc. of the 38th Hawaii international conference on system sciences (mini-track ‘‘online communities in the digital economy”) (pp. 193–202). IEEE Press. Cheng, R., & Vassileva, J. (2006). Design and evaluation of an adaptive incentive mechanism for sustained educational online communities. Journal of User Modelling and User-Adapted Interaction (special issue on User Modelling Supporting Collaboration and Online Communities), 16(3–4), 321–348. Deci, E. L., Koestner, R., & Ryan, R. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. American Psychological Association, 125(6), 627–668. Ellis, J., Halverson, C., & Erickson, T. (2005). Sustaining community-incentive mechanisms in online systems: Final report of the group 2005 workshop. Unpublished. Available at: . Emerson, R. M. (1976). Social exchange theory. Annual Review of Sociology, 2, 335–362. Farzan, R., & Brusilovsky, P. (2005). Social navigation support through annotation￾based group modeling. In Proc. of 10th international user modeling conference (pp. 463–472). Berlin: Springer Verlag. Farzan, R., DiMicco, J. M., Millen, D. R., Brownholtz, B., Geyer, W., & Dugan, C. (2008a). Results from deploying a participation incentive mechanism within the enterprise. In Proc. of the twenty-sixth annual SIGCHI conference on human factors in computing systems (pp. 563–572). ACM. Farzan, R., DiMicco, J. M., Millen, D. R., Brownholtz, B., Geyer, W., & Dugan, C. (2008b). When the experiment is over: Deploying an incentive system to all the users. In Proc. of persuasive technology symposium, in conjunction with the AISB 2008 convention. Flickr. Available at: . Harper, M., Li, X., Chen, Y., & Konstan, J. (2005). An economic model of user rating in an online recommender system. In User modeling (pp. 307–316). Berlin, Heidelberg: Springer. Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1), 5–53. Karau, S., & Williams, K. (1993). Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology, 65(4), 681–706. Kollock, P. (1999). The economies of online cooperation: gifts and public goods in cyberspace. In M. mith & P. Kollock (Eds.), Communities in cyberspace. London: Routledge. LinkedIn. Available at: . Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation. American Psychologist, 57(9), 705–717 (goal setting theory September). Lui, S., Lang, K., & Kwok, S. (2002). Participation incentive mechanisms in peer-to￾peer subscription systems. In IEEE (Ed.), Hicss’02 (Vol. 9, pp. 302–309). (user participation, contribution). Miller, B., Albert, I., Lam, S. K., Konstan, J., & Riedl, J. (2003). Movielens unplugged: Experiences with a recommender system on four mobile devices. In Proc. of the 8th international conference on intelligent user interfaces (pp. 263–266). ACM. Nielsen, J. (2006). Participation inequality: Encouraging more users to contribute. In it Jakob Nielsens alertbox. Available at: . Rashid, A. M., Ling, K., Tassone, R. D., Resnick, P., Kraut, R., & Riedl, J. (2006). Motivating participation by displaying the value of contribution. In Sigchi conference on human factors in computing systems (pp. 955–958). New York, NY, USA: ACM Press. Schafer, J. B., Frankowski, D., Herlocker, J., & Sen, S. (2007). Collaborative filtering recommender systems. In W. N. Peter Brusilovksy & A. Kobsa (Eds.). The adaptive web: Methods and strategies of web personalization (Vol. 4321). Springer. Wasko, M., & Faraj, S. (2005). Why should i share? Examining social capital and knowledge contribution in electronic networks of practice. MIS Quarterly, 29(1), 35–47. Wikipedia. Available at: . 284 R. Farzan, P. Brusilovsky / Computers in Human Behavior 27 (2011) 276–284