Pooling Exhaustive judgments for all documents in a collection is not practical Pooling technique is used in TREC top k results for TREC, k varied between 50 and 200) from the rankings obtained by different search engines for retrieval algorithms are merged into a pool duplicates are removed documents are presented in some random order to the relevance judges Produces a large number of relevance judgments for each query, although still incomplete 1/NPooling • Exhaustive judgments for all documents in a collection is not practical • Pooling technique is used in TREC – top k results (for TREC, k varied between 50 and 200) from the rankings obtained by different search engines (or retrieval algorithms) are merged into a pool – duplicates are removed – documents are presented in some random order to the relevance judges • Produces a large number of relevance judgments for each query, although still incomplete 11/N