西安交通大学Natural languageprocessingwith deeplearningXIANHAOTONGUNIVERSITYLanguage Model&Distributed Representation (2)交通大学ChenLicli@xjtu.edu.cn2023
Chen Li cli@xjtu.edu.cn 2023 Language Model & Distributed Representation (2) Natural language processing with deep learning
Outlines1. Word2Vector (W2V)2. Training LM3. Evaluating LM
Outlines 1. Word2Vector (W2V) 2. Training LM 3. Evaluating LM
Outlines1.Word2Vector(W2V2. Training LM3. Evaluating LM
Outlines 1. Word2Vector (W2V) 2. Training LM 3. Evaluating LM
WordEmbeddingConstruct a dense vector to represent each word, making this vector similartothevectorofwordsinsimilarcontexts0.2860.792-0.177banking:-0.1070.109-0.5420.3490.271
Word Embedding • Construct a dense vector to represent each word, making this vector similar to the vector of words in similar contexts. 0.286 0.792 −0.177 −0.107 0.109 −0.542 0.349 0.271 banking =
WordEmbeddingConstruct a dense vectorto representeach word,makingthis vector similartothevectorofwordsinsimilarcontexts0.2860.792-0.177banking:-0.1070.109-0.5420.3490.271Word vectorsaresometimes calledword embeddings orword representationsThese are distributed representations
Word Embedding • Construct a dense vector to represent each word, making this vector similar to the vector of words in similar contexts. 0.286 0.792 −0.177 −0.107 0.109 −0.542 0.349 0.271 banking = Word vectors are sometimes called word embeddings or word representations. These are distributed representations
Word vectors representword meaningvisualizationneedhelpcomeqo0.286take0.792keepqive-0.177getmakemeetcontinue-0.107seeexpect=0.109wantbecomeexpect-0.542thinkremainsay0.349areis0.271beweraas0.487beingbeenhadhashave
Word vectors represent word meaning- visualization 0.286 0.792 −0.177 −0.107 0.109 −0.542 0.349 0.271 0.487 expect =
Word vectorsrepresentword meaning-visualizationfoodeatlaptopDistributional vectorsrepresentedby a D-dimensionalvector where D<<V,where V is size of VocabularyThegreatest contribution of distributed representation is tomakerelatedorsimilarwordscloserindistanceand solve the problem of curseof dimensiontoa certain extent
Word vectors represent word meaning- visualization The greatest contribution of distributed representation is to make related or similar words closer in distance and solve the problem of curse of dimension to a certain extent
Word vectorsrepresentword meaning-visualization著名的类比King-Man+Woman=QueenMale-FemaleThegreatest contributionof distributed representationisto makerelatedorsimilarwordscloserindistanceandsolvetheproblemofcurseofdimensiontoacertainextent
Word vectors represent word meaning- visualization The greatest contribution of distributed representation is to make related or similar words closer in distance and solve the problem of curse of dimension to a certain extent
Distributed and Distributional RepresentationNotes:Distributed representation refers to the form of textrepresentation,whichislowdimensionaland densecontinuousvectorDistributional Representationis akind of methodto obtaintextrepresentation,whichuses co-occurrence matrixto obtainthesemanticrepresentationofwords.Eachlineof co-occurrencematrixcanbe regarded as the vector representation ofcorrespondingwords逸大
Distributed and Distributional Representation Distributed representation refers to the form of text representation, which is low dimensional and dense continuous vector Distributional Representation is a kind of method to obtain text representation, which uses co-occurrence matrix to obtain the semantic representation of words. Each line of co-occurrence matrix can be regarded as the vector representation of corresponding words. Notes:
Word2vecWord2vecTomasMikolov.etal.2013交通大学1.MikolovT,ChenK,Corrado G,etal.Efficient estimation ofword representations in vectorspace[J].arXivpreprintarXiv:1301.3781,2013
Word2vec • Tomáš Mikolov, et al. 2013 Word2vec 1. Mikolov T, Chen K, Corrado G, et al. Efficient estimation of word representations in vector space[J]. arXiv preprint arXiv:1301.3781, 2013