Term Vocabulary and Postings Lists Vocabulary of Terms Tokenization language issues Arabic (or hebrew) is basically written right to left but with certain items like numbers written left to right Words are separated but letter forms within a word form complex ligatures 132-19624 usui sl Satul →<→> Ale gera achieved its independence in 1962 after 132 years of French occupation With Unicode the surface presentation is complex but the stored form is straightforwardTerm Vocabulary and Postings Lists 13 Tokenization: language issues ▪ Arabic (or Hebrew) is basically written right to left, but with certain items like numbers written left to right ▪ Words are separated, but letter forms within a word form complex ligatures ▪ ← → ← → ← ▪ ‘Algeria achieved its independence in 1962 after 132 years of French occupation.’ ▪ With Unicode, the surface presentationis complex, but the stored form is straightforward Vocabulary of Terms