ReLU A Thinner linear network a=0 Do not have smaller gradients ReLU 1 x 2 x 1 y 2 y A Thinner linear network Do not have smaller gradients 𝑧 𝑎 𝑎 = 𝑧 𝑎 = 0