Root mean square 9f1 smaller of larger step gl 01 g51 larger g smaller step i=0 Used in Adagrad 02 7 Root Mean Square 𝜎𝑖 𝑡 = 1 𝑡 + 1 ෍ 𝑖=0 𝑡 𝒈𝑖 𝒕 2 𝜽𝑖 𝒕+𝟏 ← 𝜽𝑖 𝒕 − 𝜂 𝜎𝑖 𝑡 𝒈𝑖 𝒕 𝜽1 𝜽2 𝒈1 𝒕−𝟏 𝒈1 𝒕 𝒈2 𝒕−𝟏 𝒈2 𝒕 smaller 𝜎1 𝑡 larger𝜎2 𝑡 larger step smaller step Used in Adagrad 7