Evaluation Measures Happiness: elusive to measure Most common proxy: relevance of search results But how do you measure relevance? We will detail a methodology here then examine Its IsSues Relevance measurement requires 3 elements: 1. a benchmark document collection 2. a benchmark suite of queries 3. a usually binary assessment of either relevant or Nonrelevant for each query and each document Some work on more-than- binary, but not the standard 8Evaluation 8 Happiness: elusive to measure ▪ Most common proxy: relevance of search results ▪ But how do you measure relevance? ▪ We will detail a methodology here, then examine its issues ▪ Relevance measurement requires 3 elements: 1. A benchmark document collection 2. A benchmark suite of queries 3. A usually binary assessment of either Relevant or Nonrelevant for each query and each document ▪ Some work on more-than-binary, but not the standard Measures