C Porcel et aL /Expert Systems with Applications 36(2009)5173-518 51 to analyze if the users are researchers or company employees area. For a whole evaluation we must include the collaboration rec and take into account the users preferences about it. For exam- ommendations, but in this initial version there arent very much ple, a researcher could want to collaborate only with others users, so we evaluate only the recommendations about research researchers of different research group, resources to calculate the linguistic similarity measure between the users, Gr(VUx, VUy 4.1. Evaluation metrics to obtain the compatibility degree between x and y, expressing o(VUx, VU,)as a linguistic label in S3(using the transformation For the evaluation of recommender syste n, recall function defined in Definition 6) to send it to the user. and fl are measures widely used to evaluate the quality of the rec- ommendations(Cao &Li, 2007: Cleverdon et al, 1966: Sarwar Finally, the system sends to the users of c the resource infor- Karypis, Konstan, 2000). To calculate these me mation, its calculated linguistic relevance degree and the collabo- contingency table to categorize the items with respect to the infor- ration possibilities along with a linguistic compatibility degree. mation needs. The items are classified both as relevant or irrele- The Fig. 7 shows all the process. vant and selected (recommended to the user) or not selected The contingency table (Table 4.1) is created using these four 3.3. 4 Feedback categories. This phase is related to the activity developed by the system Precision is defined as the ratio of the selected relevant items to once the user has taken some of the resources delivered by the sys- the selected items, that is, it measures the probability of a selected tem. As we said, user profiles represent the user's information item be relevant: they should be adaptable since user's needs could change continu- p_Nrs ously. Because of this, the system allows users to update their pro- iles to improve the filtering process. In our system this feedback Recall is calculated as the ratio of the selected relevant items to t process is developed in the following steps: relevant items, that is, it represents the probability of a relevant The user accesses the system entering his/her.g and password items be selected user can do the foll to edit his/her collaboration preferences. to edit his/her preferences about minimum and maximum amount FI is a combination metric that gives equal weight to both precision to edit his/her topics of interest: and recall(Cao Li, 2007: Sarwar et al, 2000): to add new UNESCO codes with its importance degrees, i.e 2×RxP F1= to delete an existing UNESCO code, s to modify the importance degree(2-tuple)assigned to an existing UNESCO code. 4.2. Experiment result Example 3. Going back to the Example 1. let us suppose the user The purpose of the experiments is to test the performance of g wants to update his/her profile because g thinks he/she the proposed recommender system, so we take into account the should belong to the category 3309-Food Technology In this case recommendations made about the research resources. We the user wants to add a new UNESco code and assigns it a consider a data set with 25 research resources of different areas importance degree of High(b3, 0), withb3 E S1): this code is in the collected by the tto experts from different information sources position 118 of the UNESCO codes list and therefore is in about research resources. These resources are included into the position 118 of the vector. system following the indications described above and the system After this the user g has a new profile represented by a new recommends these resources to the suitable users of the ICT vector with the following values area. The system considers that nine resources in all 25 resources are interesting for researchers of the ICT area. There- sy=(b4,0),ify=100 fore the system recommends nine resources to the users. In par- VUW=(b3,0),ify=118 ticular, 10 researchers use our experimental recommender otherwis system and evaluate the relevance of the recommended resources. The contingency table for each one is shown in Table 4.2 4. Experiment and evaluation The corresponding precision, recall and Fl are shown in Table 4.3. The average of precision, recall and Fl metrics are 51. 11% his section presents the evaluation of SIRE2IN, which has been 67.67% and 57.62%, respectively. The Fig. 8 shows a graph with implemented in the Tto of University of Granada. The main focus the precision, recall and FI values for each user. These values re in evaluating the system is to determinate if it fulfills the proposed veals a good performance of the proposed system. bjectives, that is, the recommended information is useful for the users. Now we have implemented a trial version, in which the sys- m works only with few researchers. In a later version we will in- clude the possibility of a free register in the system for all research Contingency table community and companies. To evaluate this primary version of SirE2IN we have designed Not selected experiments in which the proposed system is used to recommend Relevant research resources that best satisfy the preferences of ten users Irrelevant that work in Information and Communication Technologies(ICT) otdto analyze if the users are researchers or company employees and take into account the users preferences about it. For exam￾ple, a researcher could want to collaborate only with others researchers of different research group, to calculate the linguistic similarity measure between the users, rlðVUx; VUyÞ, to obtain the compatibility degree between x and y, expressing rlðVUx; VUyÞ as a linguistic label in S3 (using the transformation function defined in Definition 6) to send it to the user. Finally, the system sends to the users of UC the resource infor￾mation, its calculated linguistic relevance degree and the collabo￾ration possibilities along with a linguistic compatibility degree. The Fig. 7 shows all the process. 3.3.4. Feedback phase This phase is related to the activity developed by the system once the user has taken some of the resources delivered by the sys￾tem. As we said, user profiles represent the user’s information needs or interests and a desirable property for user profiles is that they should be adaptable since user’s needs could change continu￾ously. Because of this, the system allows users to update their pro- files to improve the filtering process. In our system this feedback process is developed in the following steps: The user accesses the system entering his/her ID and password. The user can do the following operations: – to edit his/her collaboration preferences, – to edit his/her preferences about minimum and maximum amount, – to edit his/her topics of interest:  to add new UNESCO codes with its importance degrees, i.e. 2-tuple linguistic ðbi; aiÞ with bi 2 S1 and ai 2 ½:5; :5Þ,  to delete an existing UNESCO code,  to modify the importance degree (2-tuple) assigned to an existing UNESCO code. Example 3. Going back to the Example 1, let us suppose the user ID wants to update his/her profile because ID thinks he/she should belong to the category 3309 – Food Technology. In this case the user wants to add a new UNESCO code and assigns it a importance degree of High (ðb3; 0Þ; withb3 2 S1); this code is in the position 118 of the UNESCO codes list and therefore is in the position 118 of the vector. After this the user ID has a new profile represented by a new vector with the following values: VUID½y¼ðb4; 0Þ; if y ¼ 100 VUID½y¼ðb3; 0Þ; if y ¼ 118 VUID½y¼ðb0; 0Þ; otherwise: 4. Experiment and evaluation This section presents the evaluation of SIRE2IN, which has been implemented in the TTO of University of Granada. The main focus in evaluating the system is to determinate if it fulfills the proposed objectives, that is, the recommended information is useful for the users. Now we have implemented a trial version, in which the sys￾tem works only with few researchers. In a later version we will in￾clude the possibility of a free register in the system for all research community and companies. To evaluate this primary version of SIRE2IN we have designed experiments in which the proposed system is used to recommend research resources that best satisfy the preferences of ten users that work in Information and Communication Technologies (ICT) area. For a whole evaluation we must include the collaboration rec￾ommendations, but in this initial version there aren’t very much users, so we evaluate only the recommendations about research resources. 4.1. Evaluation metrics For the evaluation of recommender systems precision, recall and F1 are measures widely used to evaluate the quality of the rec￾ommendations (Cao & Li, 2007; Cleverdon et al., 1966; Sarwar, Karypis, & Konstan, 2000). To calculate these metrics we need a contingency table to categorize the items with respect to the infor￾mation needs. The items are classified both as relevant or irrele￾vant and selected (recommended to the user) or not selected. The contingency table (Table 4.1) is created using these four categories. Precision is defined as the ratio of the selected relevant items to the selected items, that is, it measures the probability of a selected item be relevant: P ¼ Nrs Ns Recall is calculated as the ratio of the selected relevant items to the relevant items, that is, it represents the probability of a relevant items be selected: R ¼ Nrs Nr : F1 is a combination metric that gives equal weight to both precision and recall (Cao & Li, 2007; Sarwar et al., 2000): F1 ¼ 2  R  P R þ P : 4.2. Experiment result The purpose of the experiments is to test the performance of the proposed recommender system, so we take into account the recommendations made about the research resources. We consider a data set with 25 research resources of different areas collected by the TTO experts from different information sources about research resources. These resources are included into the system following the indications described above and the system recommends these resources to the suitable users of the ICT area. The system considers that nine resources in all 25 resources are interesting for researchers of the ICT area. There￾fore the system recommends nine resources to the users. In par￾ticular, 10 researchers use our experimental recommender system and evaluate the relevance of the recommended resources. The contingency table for each one is shown in Table 4.2. The corresponding precision, recall and F1 are shown in Table 4.3. The average of precision, recall and F1 metrics are 51.11%, 67.67% and 57.62%, respectively. The Fig. 8 shows a graph with the precision, recall and F1 values for each user. These values re￾veals a good performance of the proposed system. Table 4.1 Contingency table Selected Not selected Total Relevant Nrs Nrn Nr Irrelevant Nis Nin Ni Total Ns Nn N C. Porcel et al. / Expert Systems with Applications 36 (2009) 5173–5183 5181