Limitations of current techniques Language models in pre-training are unidirectional,they restrict the power of the pre-trained representations .OpenAl GPT used left-to-right architecture ELMo concatenates forward and backward language models 。Solution BERT:Bidirectional Encoder Representations from Transformers 国产之小丝Limitations of current techniques 2024/5/13 14 •Language models in pre-training are unidirectional, they restrict the power of the pre-trained representations •OpenAI GPT used left-to-right architecture •ELMo concatenates forward and backward language models • Solution BERT: Bidirectional Encoder Representations from Transformers