Maxout More than ReLU Z Input W ReLU a Input W Max →a 12 maxtz ,z2 Learnable Activation Function a z=wx+b Z1=Wx +b X Z2 =w'x +b' Maxout Max x1 Input + 𝑧1 + 𝑧2 𝑎 𝑚𝑎𝑥 𝑧1 , 𝑧2 𝑤 𝑏 𝑤′ 𝑏 ′ 𝑥 𝑧 = 𝑤𝑥 + 𝑏 𝑎 x1 Input ReLU 𝑧 𝑤 𝑏 𝑎 𝑥 𝑧1 = 𝑤𝑥 + 𝑏 𝑎 𝑧2 = 𝑤′𝑥 + 𝑏 ′ Learnable Activation Function More than ReLU