Industrial and Government Applications Track Paper where a controls the amount of overlap in expertise desired in the An important aspect algorithm is that it can easily be 1, then it is sufficient to have one reviewer started from a partial solution. This turns out to be a very useful whose expertise about a term equals the importance of that term property when considering the context in which the system is to the proposal. If E is 0.5, then two reviewers should have used By allowing program directors to provide a partial solution expertise on every term in the proposal that will then guide the system towards its final solution, we allow the experts to use Revaide as a tool to assist them to complete To compare alternative sets of reviewers and alternative their jobs rather than using it to completely replace their approaches for finding reviewers we define a measure called Sum of Residual Term Weight (SRTW)to be Another benefit of sRTM is that it may be used to determine whether a proposal has reviewers with adequate expertise. Whe SRTM Pr there is no reviewer with expertise on an aspect of the proposal the value of SRTM for that proposal would be higher than others This might occur if the pool of reviewers is too small or if the We define the goal of assigning I to be finding a set proposal is on a topic that had not received submissions in the of reviewers that reduces the sum of res ms to be o and the ast. One way to find a reviewer in this case is to use the terms one set of reviewers is better suited to review a proposal than a with the highest residual weights as query to a specialized search number if that set of reviewers has a lower Srtm engine such as Google Scholar. Figure I illustrates the results of We have implemented a hill-climbing search algorithm to Google Scholar using the three terms with the highest residual weights from table 1. Although Google Scholar is not integrated find a set of reviewers for each proposal. We start by finding the with the entire workflow of Revaide(e.g, it doesn't identify the best" reviewer and then iteratively select another reviewer until N are found. At each step, the reviewer that minimizes SRTM is e-mail address and affiliation of the authors), it still provides a selected. This iterative process will reduce the residual term useful way of recommending reviewers eight. The residual term weight with no reviewers is 1.0(since As we have described assigning reviewers and srtm so far e work with normalized vectors). As each reviewer is selected, the goal is to find a set of reviewers for a single propos the term weights are adjusted according to the expertise of the However, at NSF panels, reviewers typically review several reviewer. By subtracting the expertise vector from the document proposals in a panel. Revaide can easily be used to recommend panelists for a set of proposals. Recall that in cluster checking, will decrease Revaide creates a term vector for each panel that is the centroids Table I shows a trace of how the residual term weights are of the proposals in the panel. This clu reduced by selecting reviewers. The row shows the most the terms that are most important to the proposals in the pan To invite panelists, Revaide simply finds the panelists whose remaining table shows the residual term vector after subtracting expertise best reduces the SRTM of the centroid of the panel.In feedback for image retrieval is to be reviewed. The first reviewer reviewers might be selected for a panel of 24 proposals. A lower each expertise vector(with 8=0.5). A proposal on relevance selected is an expert on image retrieval. Once that contribution value of s is used when selecting reviewers for a panel. For has been accounted for, we see terms such as"image" have a example, a value of 0. 2 will bias Revaide toward finding 5 lower term weight, reducing their impact on finding the next viewers with expertise in the major areas. In reality not everyone who is invited to review actually agrees to. Therefore reviewer. The second reviewer has greater experience M The we typically ask 20 with the expectation of getting a 50% yield rocess repeats until the desired number of reviewers are found vIewer using the confirmed reviewers as a starting point and finding reviewers to complement their expertise After Reviewer 1 judgments 0.280 feedback 0.023 relevance 0022im 0.020 multimodal.020 After Reviewer 2 feedback.023image 0.020 multimodal 0.020 preference 0.016 judgments 0.015 After Reviewer 3 feedback 0.020 multimodal 0.019 preference 0.016 judgments 0.015 solici Table 1. a trace of the residual term vectors after assigning reviewers 67where ε controls the amount of overlap in expertise desired in the reviewers. If ε is 1, then it is sufficient to have one reviewer whose expertise about a term equals the importance of that term to the proposal. If ε is 0.5, then two reviewers should have expertise on every term in the proposal. To compare alternative sets of reviewers and alternative approaches for finding reviewers we define a measure called Sum of Residual Term Weight (SRTW) to be: = ∑ − ∑ i k j i j e i SRTM p ) , max(0, ε We define the goal of assigning reviewers to be finding a set of reviewers that reduces the sum of residual terms to be 0 and the one set of reviewers is better suited to review a proposal than a number if that set of reviewers has a lower SRTM. We have implemented a hill-climbing search algorithm to find a set of reviewers for each proposal. We start by finding the “best” reviewer and then iteratively select another reviewer until N are found. At each step, the reviewer that minimizes SRTM is selected. This iterative process will reduce the residual term weight. The residual term weight with no reviewers is 1.0 (since we work with normalized vectors). As each reviewer is selected, the term weights are adjusted according to the expertise of the reviewer. By subtracting the expertise vector from the document vector, the sum of residual term weights in the document vector will decrease. Table 1 shows a trace of how the residual term weights are reduced by selecting reviewers. The row shows the most important terms in the term vector of a proposal and the remaining table shows the residual term vector after subtracting each expertise vector (with ε =0.5). A proposal on relevance feedback for image retrieval is to be reviewed. The first reviewer selected is an expert on image retrieval. Once that contribution has been accounted for, we see terms such as “image” have a lower term weight, reducing their impact on finding the next reviewer. The second reviewer has greater experience in image relevance judgments and these terms are reduced in weight. The process repeats until the desired number of reviewers are found. An important aspect of this algorithm is that it can easily be started from a partial solution. This turns out to be a very useful property when considering the context in which the system is used. By allowing program directors to provide a partial solution that will then guide the system towards its final solution, we allow the experts to use Revaide as a tool to assist them to complete their jobs rather than using it to completely replace their judgments. Another benefit of SRTM is that it may be used to determine whether a proposal has reviewers with adequate expertise. When there is no reviewer with expertise on an aspect of the proposal, the value of SRTM for that proposal would be higher than others. This might occur if the pool of reviewers is too small or if the proposal is on a topic that had not received submissions in the past. One way to find a reviewer in this case is to use the terms with the highest residual weights as query to a specialized search engine such as Google Scholar. Figure 1 illustrates the results of Google Scholar using the three terms with the highest residual weights from table 1. Although Google Scholar is not integrated with the entire workflow of Revaide (e.g., it doesn’t identify the e-mail address and affiliation of the authors), it still provides a useful way of recommending reviewers. As we have described assigning reviewers and SRTM so far, the goal is to find a set of reviewers for a single proposal. However, at NSF panels, reviewers typically review several proposals in a panel. Revaide can easily be used to recommend panelists for a set of proposals. Recall that in cluster checking, Revaide creates a term vector for each panel that is the centroids of the proposals in the panel. This cluster term vector represents the terms that are most important to the proposals in the panel. To invite panelists, Revaide simply finds the panelists whose expertise best reduces the SRTM of the centroid of the panel. In this case, rather than assigning four reviewers to a proposal, 12 reviewers might be selected for a panel of 24 proposals. A lower value of ε is used when selecting reviewers for a panel. For example, a value of 0.2 will bias Revaide toward finding 5 reviewers with expertise in the major areas. In reality not everyone who is invited to review actually agrees to. Therefore, we typically ask 20 with the expectation of getting a 50% yield. Once many reviewers have accepted, Revaide can be run again using the confirmed reviewers as a starting point and finding reviewers to complement their expertise. Proposal image 0.031 judgments 0.028 feedback 0.027 relevance 0.026 multimodal 0.020 After Reviewer 1 judgments 0.280 feedback 0.023 relevance 0.022 image 0.020 multimodal 0.020 After Reviewer 2 feedback 0.023 image 0.020 multimodal 0.020 preference 0.016 judgments 0.015 After Reviewer 3 feedback 0.020 multimodal 0.019 preference 0.016 judgments 0.015 solicit 0.011 Table 1. A trace of the residual term vectors after assigning reviewers. 867 Industrial and Government Applications Track Paper