西安交通大学Natural languageprocessingwith deeplearningXIANHAOTONGUNIVERSITYLanguage Model&Distributed Representation (1)交通大学ChenLicli@xjtu.edu.cn2023
Chen Li cli@xjtu.edu.cn 2023 Language Model & Distributed Representation (1) Natural language processing with deep learning
Outlines1. Traditional Language Model (LM)2.Distributed Representation交道大学
Outlines 1. Traditional Language Model (LM) 2. Distributed Representation
Outlines1.Traditional Language Model (LM)2.Distributed Representation交道大学
Outlines 1. Traditional Language Model (LM) 2. Distributed Representation
OverviewAnewNLPtaskLanguage modelingRelated conceptsMarkov chainn-gram modelMaximumLikelihoodEstimation(MLELaplace smoothing, additive smoothingGood Turing
• Language modeling A new NLP task Overview • Markov chain • n-gram model • Maximum Likelihood Estimation (MLE) • Laplace smoothing, additive smoothing • Good Turing Related concepts
LanguageModelDefinition: compute probability distribution over the words in a sentence交道大学
• Definition: compute probability distribution over the words in a sentence Language Model
LanguageModelDefinition: compute probability distribution over the words in a sentenceApplication: predicting the next word in a sequence given the sequence ofwordsalreadypresentbooklaptopthe girl opened herdoormind交通大学
• Definition: compute probability distribution over the words in a sentence • Application: predicting the next word in a sequence given the sequence of words already present Language Model the girl opened her book laptop door mind
LanguageModelDefinition: compute probability distribution overthe words in a sentenceApplication: predicting the next word in a sequence given the sequence ofwords already presentbooklaptopthe girl opened herdoormindFormal Definition:Given a sequence of words x(1), x(2),., x(t), estimate the probabilitydistribution of the next word x(t+1)P(x(t+1)|x(t), .. (1)x(t+1) could be any word in the vocabulary V = (W1, .., Wivi} Language Model is a system like this
Language Model the girl opened her book laptop door • Formal Definition: Given a sequence of words � (1) , � (2) ,., � (�) , estimate the probability distribution of the next word � (t+1) 。 � � (�+1)|� (�) , ., � (1) � (t+1) could be any word in the vocabulary V = {�1, ., �|�|} 。 • Language Model is a system like this mind • Definition: compute probability distribution over the words in a sentence • Application: predicting the next word in a sequence given the sequence of words already present
LanguageModelLanguage Model could also be regarded as a system assigning a probabilitytoatext.交道大学
• Language Model could also be regarded as a system assigning a probability to a text. Language Model
LanguageModelLanguage Model could also be regarded as a system assigning a probabilityto a text.For example, the probability of a text x(1), x(2) ,., x(T) (Language Model)couldbedenotedasP(x(1), . (T) = P(x(1) × P(x(2)|x(1) × . × P(x(T)|x(T-1),.. (1)= I/= P(x() |x(t-1), ., (1)ThisiswhatLanguageModelcomputes
• Language Model could also be regarded as a system assigning a probability to a text. • For example, the probability of a text � (1) , � (2) ,., � (�) (Language Model) could be denoted as P � (1) , ., � (�) = � � (1) × � � (2) � (1) × ∙∙∙ × � � (�) � (�−1) , ., � (1) = �=1 � � � (�) � (�−1) , ., � (1) Language Model This is what Language Model computes
LanguageModelHave youeverused Language Model?交通大学
• Have you ever used Language Model？ Language Model