Evaluation Benchmarks Standard relevance benchmarks TREC -National Institute of standards and Technology nist) has run a large ir test bed for many years Reuters and other benchmark doc collections used Retrieval tasks" specified sometimes as queries Human experts mark, for each query and for each doc, Relevant or nonrelevant or at least for subset of docs that some system returned for that queryEvaluation 10 Standard relevance benchmarks ▪ TREC - National Institute of Standards and Technology (NIST) has run a large IR test bed for many years ▪ Reuters and other benchmark doc collections used ▪ “Retrieval tasks” specified ▪ sometimes as queries ▪ Human experts mark, for each query and for each doc, Relevant or Nonrelevant ▪ or at least for subset of docs that some system returned for that query Benchmarks