Original paper:https://arxiv.org/pdf/1412.6980.pdf Adam:RMSProp Momentum Algorithm 1:Adam,our proposed algorithm for stochastic optimization.See section 2 for details, and for a slightly more efficient (but less clear)order of computation.g?indicates the elementwise square gtgt.Good default settings for the tested machine learning problems are a =0.001, B1 =0.9,B2 =0.999 and e=10-8.All operations on vectors are element-wise.With B and B we denote Bi and B2 to the power t. Require:a:Stepsize Require:B1,B2 E 0,1):Exponential decay rates for the moment estimates Require:f():Stochastic objective function with parameters 0 Require:0o:Initial parameter vector mo0(Initialize 1st moment vector)-for momentum vo0(Initialize 2nd moment vector) t0(Initialize timestep) *for RMSprop while 0:not converged do t←-t+1 gtVoft(-1)(Get gradients w.r.t.stochastic objective at timestep t) mt←B1·mt-1+(1-B1)·9t(Update biased first moment estimate) vt←-B2·vt-1+(1-B2)·gf(Update biased second raw moment estimate) mt mt/(1-B)(Compute bias-corrected first moment estimate) vt/(1-B2)(Compute bias-corrected second raw moment estimate) a:←0t-1-a·mt/(√ii+e)(Update parameters) end while return 0:(Resulting parameters) Adam: RMSProp + Momentum for momentum for RMSprop Original paper: https://arxiv.org/pdf/1412.6980.pdf 11