Term Vocabulary and Postings Lists Vocabulary of Terms Tokenization language issues Chinese and japanese have no spaces between woras ■莎拉波娃现在居住在美国东南部的佛罗里达。 Not always guaranteed a unique tokenization Further complicated in Japanese, with multiple alphabets intermingled Dates/ amounts in multiple formats -元500社(情辍不足在的時刷方5005000万 Katakana Hiragana Kanji romaji End-user can express query entirely in hiraganaTerm Vocabulary and Postings Lists 12 Tokenization: language issues ▪ Chinese and Japanese have no spaces between words: ▪ 莎拉波娃现在居住在美国东南部的佛罗里达。 ▪ Not always guaranteed a unique tokenization ▪ Further complicated in Japanese, with multiple alphabets intermingled ▪ Dates/amounts in multiple formats フォーチュン500社は情報不足のため時間あた$500K(約6,000万円) Katakana Hiragana Kanji Romaji End-user can express query entirely in hiragana! Vocabulary of Terms