Term Vocabulary and Postings Lists Vocabulary of Terms Stop words With a stop list, you exclude from the dictionary entirely the commonest words. Intuition They have little semantic content: the, a, and, to, be There are a lot of them: 30%of postings for top 30 words But the trend is away from doing this Good compression techniques (lecture 6)means the space for including stop words in a system is very small Good query optimization techniques(lecture 8 mean you pay little at query time for including stop words You need them for Phrase queries: King of Denmark Various song titles, etc. Let it be,"to be or not to be “ Relational" queries:“ flights to LondonTerm Vocabulary and Postings Lists 14 Stop words ▪ With a stop list, you exclude from the dictionary entirely the commonest words. Intuition: ▪ They have little semantic content: the, a, and, to, be ▪ There are a lot of them: ~30% of postings for top 30 words ▪ But the trend is away from doing this: ▪ Good compression techniques (lecture 6) means the space for including stop words in a system is very small ▪ Good query optimization techniques (lecture 8) mean you pay little at query time for including stop words. ▪ You need them for: ▪ Phrase queries: “King of Denmark” ▪ Various song titles, etc.: “Let it be”, “To be or not to be” ▪ “Relational” queries: “flights to London” Vocabulary of Terms