Self-attention Extract information based on attention scores =∑叫w b1 01,1 01,2 01,4 k3 a Q3 ￥ vi=Wval v2 Wva2 v3 WUa3 v4 Wva4 18 Self-attention 𝛼1,1 ′ 𝛼1,2 ′ 𝛼1,3 ′ 𝛼1,4 ′ 𝒌 𝟏 𝒒 𝟏 𝒗 𝟏 𝒌 𝟐 𝒗 𝟐 𝒌 𝟑 𝒗 𝟑 𝒌 𝟒 𝒗 𝟒 𝒂 𝟒 𝒂 𝟑 𝒂 𝟐 𝒂 𝟏 𝒃 𝟏 𝒃 𝟏 = ෍ 𝑖 𝛼1,𝑖 ′ 𝒗 𝒊 Extract information based on attention scores 𝒗 𝟏 = 𝑊𝑣𝒂 𝟏 𝒗 𝟐 = 𝑊𝑣𝒂 𝟐 𝒗 𝟑 = 𝑊𝑣𝒂 𝟑 𝒗 𝟒 = 𝑊𝑣𝒂 𝟒 18