where r( is the solution of the initial value problem i(s)=∫(x(s),u(s),t≤s≤t1, (t) Here, tE[to, ti] is a "variable"initial time, u( )is a control defined on t, ti] taking values space of admissible controls, containing at least the piecewise continuous control The value function is defined by infJ(t,x;(·) for (t, r)E[to, ti x R". The dymamic programming principle states that for every r E t,til v(tr inf L((s),u(s)ds+V(r,a(r) ()∈,r (we will prove this later on). From this, one can derive formally the equation at v(t, )+H(r, Vav(t, a ))=0 in(to, t1)X R with terminal data V v(ar) in R Here, the Hamiltonian is given by H(x,入)=inf{λ·f(x,U)+L(x,U)} The nonlinear first order PDE (7) is the dynamic programming PDE or Hamilton-Jacobi- Bellman(hjb) equation. The pair(7),( 8) specify what is called a Cauchy problem, and can be viewed as a special case of (1)together with suitable boundary conditions, usin Q=(to, ti)x Rn. Notice that the Hamiltonian( 9)is concave in the variable A(since it is the infimum of linear functions) Let us see how(7)is obtained. Set r=t+h, h>0, and rearrange(6)to yield inf (V(t+h, c(t+h))-v(t, a))+ If V and u() are sufficiently smooth, then (V(+h,a(t+h)-v(t, 2)-atv(a, t)+v2V(r, t). f(, u(t)as h-0 L(x(s),u(s))ds→L(x,u(t)ash→0.where x(·) is the solution of the initial value problem    x˙(s) = f(x(s), u(s)), t ≤ s ≤ t1, x(t) = x. (4) Here, t ∈ [t0, t1] is a “variable” initial time, u(·) is a control defined on [t, t1] taking values in, say, U ⊂ Rm (U closed), and x(·) is the state trajectory in Rn . We denote by Ut,t1 a space of admissible controls, containing at least the piecewise continuous controls. The value function is defined by V (t, x) = inf u(·)∈Ut,t1 J(t, x; u(·)) (5) for (t, x) ∈ [t0, t1] × Rn . The dynamic programming principle states that for every r ∈ [t, t1], V (t, x) = inf u(·)∈Ut,r Z r t L(x(s), u(s)) ds + V (r, x(r)) (6) (we will prove this later on). From this, one can derive formally the equation ∂ ∂tV (t, x) + H(x, ∇xV (t, x)) = 0 in (t0, t1) × Rn , (7) with terminal data V (t1, x) = ψ(x) in Rn . (8) Here, the Hamiltonian is given by H(x, λ) = inf v∈U {λ · f(x, v) + L(x, v)} (9) The nonlinear first order PDE (7) is the dynamic programming PDE or Hamilton-Jacobi￾Bellman (HJB) equation. The pair (7), (8) specify what is called a Cauchy problem, and can be viewed as a special case of (1) together with suitable boundary conditions, using Ω = (t0, t1) × Rn . Notice that the Hamiltonian (9) is concave in the variable λ (since it is the infimum of linear functions). Let us see how (7) is obtained. Set r = t + h, h > 0, and rearrange (6) to yield inf u(·)  1 h (V (t + h, x(t + h)) − V (t, x)) + 1 h Z t+h t L(x(s), u(s)) ds = 0. If V and u(·) are sufficiently smooth, then 1 h (V (t + h, x(t + h)) − V (t, x)) → ∂ ∂tV (x, t) + ∇xV (x, t) · f(x, u(t)) as h → 0 and 1 h Z t+h t L(x(s), u(s)) ds → L(x, u(t)) as h → 0. 6