Sample x1,x2,...,xm}from Paata(x) m m o=arg mgx ΠP6=arg mxlog(xy m arg max logP(xargmx ExalogPo()] 台 (not related to arg max Paata(x)logPg(x)dx-Paata(x)logPaata(x)dx Pe(x) Difference between Pdata and Pe arg max Pata(x)P dx =arg min KL(PaatallPe) Maximum Likelihood Minimize KL DivergenceSample 𝑥 1 , 𝑥 2 , … , 𝑥 𝑚 from 𝑃𝑑𝑎𝑡𝑎 𝑥 𝜃 ∗ = 𝑎𝑟𝑔 max 𝜃 ෑ 𝑖=1 𝑚 𝑃𝜃 𝑥 𝑖 = 𝑎𝑟𝑔 max 𝜃 𝑙𝑜𝑔ෑ 𝑖=1 𝑚 𝑃𝜃 𝑥 𝑖 = 𝑎𝑟𝑔 max 𝜃 ෍ 𝑖=1 𝑚 𝑙𝑜𝑔𝑃𝜃 𝑥 𝑖 ≈ 𝑎𝑟𝑔 max 𝜃 𝐸𝑥~𝑃𝑑𝑎𝑡𝑎[𝑙𝑜𝑔𝑃𝜃 𝑥 ] = 𝑎𝑟𝑔 max 𝜃 න 𝑥 𝑃𝑑𝑎𝑡𝑎 𝑥 𝑙𝑜𝑔𝑃𝜃 𝑥 𝑑𝑥 − න 𝑥 𝑃𝑑𝑎𝑡𝑎 𝑥 𝑙𝑜𝑔𝑃𝑑𝑎𝑡𝑎 𝑥 𝑑𝑥 = 𝑎𝑟𝑔 min 𝜃 𝐾𝐿 𝑃𝑑𝑎𝑡𝑎||𝑃𝜃 = 𝑎𝑟𝑔 max 𝜃 න 𝑥 𝑃𝑑𝑎𝑡𝑎 𝑥 𝑙𝑜𝑔 𝑃𝜃 𝑥 𝑃𝑑𝑎𝑡𝑎 𝑥 𝑑𝑥 Maximum Likelihood = Minimize KL Divergence (not related to 𝜃) Difference between 𝑃𝑑𝑎𝑡𝑎 and 𝑃𝜃