Relevance Judgments Obtaining relevance judgments is an expensive, time-consuming process who does it? what are the instructions? what is the level of agreement? TREC judgments depend on task being evaluated generally binary agreement good because of"narrative 10/NRelevance Judgments • Obtaining relevance judgments is an expensive, time-consuming process – who does it? – what are the instructions? – what is the level of agreement? • TREC judgments – depend on task being evaluated – generally binary – agreement good because of “narrative” 10/N