Mem. Differential Equations Math. Phys. 13(1997), 121–129

M. Mania and R. Tevzadze

SOLUTION OF BELLMAN’S EQUATION BY MEANS OF A SYSTEM OF NONLINEAR SINGULAR INTEGRAL EQUATIONS

(Reported on September 22, 29, 1997)

1. Introduction and the Main Results

The purpose of this paper is to show the existence of a unique generalized solution of Bellman’s equation

S_{t}(t, x) + max_{a∈A} [\frac{1}{2} σ^{2}(t, x, a)S_{xx}(t, x) + b(t, x, a)S_{x}(t, x))] = 0  (1)

with the boundary condition

S(T, x) = g(x)  (2)

under the following conditions on the coefficients b, σ and on the terminal reward function g:

A1) the functions b and σ are measurable and bounded, i.e., for some C > 0

|b(t, x, a)| + |σ(t, x, a)| ≤ C;

A2) there exists a constant λ > 0 such that for all t ∈ [0, T], x ∈ R, a ∈ A

σ^{2}(t, x, a) > λ;

A3) the functions b and σ are continuous in a for each t ∈ [0, T], x ∈ R;

A4) the function g belongs to the Sobolev space W^{1}.

It is well known that the problem (1), (2) is closely connected to a stochastic control problem for a system whose dynamics is discribed by the stochastic differential equation (SDE)

dX_{t} = b(t, X_{t}, u_{t})dt + σ(t, X_{t}, u_{t})dW_{t}, X_{0} = x_{0} ∈ R.  (3)

Here (W_{t}, t ≥ 0) is a standard Wiener process defined on a complete probability space (Ω, F, P) and the control u = (u_{t}, t ∈ [0, T]) is a feedback of the current state, i.e., u_{t} = u(t, X_{t}) for some given function u(t, x) taking values in a decision set A which is assumed to be a separable metric space. To each control u we associate one (fixed) solution of SDE (1) (the conditions A1)-A3) imply the existence of a weak solution of SDE (1) ([3])) and the notation P_{t,x}^{u} is used for the distributon of this solution starting at X_{t} = x. The problem is to maximize the expected cost E^{u}g(X_{T}) by a suitable choice of feedback controls.

The formal application of Bellman’s ”dynamic programming” idea leads to the Bellman equation (1), (2) whose solution, if it exists, is easily shown to be the value function

S(t, x) = sup_{u} E_{t,x}^{u}g(X_{T})  (4)

1991 Mathematics Subject Classification. 35K55.

Key words and phrases. Controlled diffusions, Bellman’s equation, singular integral equation, Fourier transform. of the control problem (E_{t,x}^{u} is the expectation relative to the measure P_{t,x}^{u}). Moreover, if S solves (1), (2), then the optimal control u^{∗} may be constructed by the pointwise maximization of the Hamiltonian

H(t, x, a) = \frac{1}{2} σ^{2}(t, x, a)S_{xx}(t, x) + b(t, x, a)S_{x}(t, x).  (5)

Therefore the main problem consists in finding conditions under which the solution of Bellman’s equation exists.

The novelty of this paper is that the question of existence of optimal controls is solved without any regularity assumptions on the coefficients and the use is made of singular integral equations.

Our method is as follows: for the problem (1), (2) we compose a system of nonlinear singular integral equations, namely

ψ(t, x) = ∫_{R} \frac{y − x}{r(T − t)} ρ^{r}(T − t, y − x)g(y)dy +

+ ∫_{t}^{T} ∫_{R} \frac{y − x}{r(u − t)} ρ^{r}(u − t, y − x)G(u, y, ψ(u, y), ψ̂(u, y))dydu,  (6)

ψ̃(t, x) = ∫_{R} \frac{(y − x)^{2} − r(T − t)}{r^{2}(T − t)^{2}} ρ^{r}(T − t, y − x)(g(y) − g(x))dy +

+ ∫_{t}^{T} ∫_{R} \frac{(y − x)^{2} − r(u − t)}{r^{2}(u − t)^{2}} ρ^{r}(u − t, y − x)[G(u, y, ψ(u, y), ψ̃(u, y)) −

−G(u, x, ψ(u, x), ψ̃(t, x))]dydu,  (7)

where ρ^{r}(t, x) = (2πtr)^{− \frac{1}{2}} exp\{− \frac{x^{2}}{2rt} \} and

G(t, x, p, q) = max_{a} [\frac{1}{2} (σ^{2}(t, x, a) − r)p + b(t, x, a)q].  (8)

This system can be obtained from the equation

S_{t}(t, x) + \frac{r}{2} S_{xx}(t, x) + G(t, x, S_{x}(t, x), S_{xx}(t, x)) = 0,

which is equivalent to (1), using the Feynmann-Kac representation

S(t, x) = ∫_{t}^{T} ∫_{R} ρ^{r}(s − t, y − x)G(s, y, S_{x}(s, y), S_{xx}(s, y))dsdy +

+ ∫_{R} ρ^{r}(T − s, y − x)g(y)dy  (9)

and taking the first and second derivatives in x. We show that there exists a constant r > 0 for which the operator on the right-hand side of (6), (7), as a mapping of (L^{2}([0, T]× R) × L^{2}([0, T] × R))^{2} onto itself, is a contraction for sufficiently small T and thus the system (6), (7) has a unique solution in the class (L^{2}([0, T] × R) × L^{2}([0, T] × R))^{2} for every small time interval. Further, the solution of the system (6), (7) is used to construct the solution of Bellman’s equation.

Define the Sobolev space W^{1,2} as the completion of the space of infinitely differentiable finite functions C_{0}^{∞} in the norm

∥u∥_{W^{1,2}} = sup_{(t,x)∈[0,T]×R} |f(t, x)|+∥f_{t}∥_{L^{2}([0,T]×R)} + ∥f_{x}∥_{L^{2}([0,T]×R)}+∥f_{xx}∥_{L^{2}([0,T]×R)}.

The class W^{1} is defined as the completion of the space of differentiable functions in the norm

∥f∥_{W^{1}} = ∥f∥_{L^{2}(R)} + ∥f_{x}∥_{L^{2}(R)}.

The following statements are proved in Sections 3 and 4 of this paper. In particular, it is shown that the problems (1), (2) and (6), (7) are equivalent.

Theorem 1. Let the conditions A1) and A4) be satisfied. Then

a) If V is a solution of Bellman’s equation (1), (2) from the class W^{1,2}, then the pair (V_{x}, V_{xx}) of generalized derivatives will be a solution of the system (6) (7) for each r > 0.

b) If for some r > 0 there exists a pair (ψ, ψ̃) from the class L^{2}([0, T] × R) × L^{2}([0, T] × R) which solves the system (6), (7), then the function

V (t, x) = ∫_{R} g(y)ρ^{r}(T − t, y − x)dy +

+ ∫_{t}^{T} ∫_{R} G(v, y, ψ(v, y), ψ̃(v, y))ρ^{r}(v − t, y − x)dydv  (10)

will be a solution of Bellman’s equation (1), (2).

Theorem 2. Let r^{∗} = max((C + 1)^{2}, 1/λ), where C and λ are constants from A1)-A2). Then for any r > r^{∗} there exists a unique solution of the system (6), (7) which belongs to the class (L^{2}([0, T] × R) × L^{2}([0, T] × R)).

As a corollary of Theorems 1 and 2, we obtain the existence of a generalized solution of Bellman’s equation. Moreover, it is shown in Section 4 that this solution coincides with the value function of the optimal control problem under consideration.

Theorem 3. The value function (4) uniquely solves the Bellman equation (1), (2) in the class W^{1,2}. If the decision set A is a compact subset of a metric space, then there exists an optimal control in the class of markovian strategies. The optimal control u^{∗} is constructed from the maximizing of the Hamiltonian (5)

H(t, x, u^{∗}(t, x)) = max_{a∈A} H(t, x, a)

for each (t, x) ∈ [0, T] × R.

The dynamic programming method of proving the existence of an optimal control in the case of diffusion processes for the first time was applied in [6] (Rishel) and [2] (Davis, Varaiya). Theorem 3 was proved in [3] (Krylov) under the Lipschitz condition on the coefficients b and σ.

2. Estimates of the Norms of Singular Integral Operators

Let us consider in L^{2}([0, T] × R) the operators

ψ(t, x) = ∫_{t}^{T} ∫_{R} \frac{y − x}{r(s − t)} ρ^{r}(s − t, y − x)φ(s, y)dsdy,  (11)

ψ̃(t, x) = ∫_{t}^{T} ∫_{R} \frac{y − x^{2} − r(s − t)}{r^{2}(s − t)^{2}} ρ^{r}(s − t, y − x)[φ̃(s, y) − φ̃(t, x)]dsdy  (12)

and the operators from L^{2}(R) to L^{2}([0, T] × R)

ψ(t, x) = ∫_{R} \frac{y − x}{r(T − t)} ρ^{r}(T − t, y − x)φ(y)dy,  (13)

ψ̃(t, x) = ∫_{R} \frac{(y − x)^{2} − r(T − t)}{r^{2}(T − t)^{2}} ρ^{r}(T − t, y − x)[φ̃(y) − φ̃(x)]dy.  (14)

We make the convention ρ^{r}(t, x) = 0 for all t < 0. Using the function K(t, x) = ρ^{r}(−t, x) defined on (−T, T) × R, one can rewrite the integral ∫_{t}^{T} ∫_{R} ρ(s − t, x − y)φ(s, y)dsdy as a convolution

(K ∗ φ)(t, x) = ∫_{−T}^{T} ∫_{R} K(t − s, x − y)φ(s, y)dsdy = ∫_{−T}^{T} ∫_{R} K(s, y)φ(t − s, x − y)dsdy.

Similarly, introducing the L^{2}[0, T]-valued function M(x) = ρ(T − ·, x) the integral ∫_{R} ρ(T − t, y − x)φ(y)dy can be written as (M ∗ φ)(t, x).

Proposition 1. If φ ∈ C_{0}^{∞}((−T, T) × R), then the integrals (11) and (12) coincide with

\frac{∂}{∂x} (K ∗ φ)(t, x), \frac{∂^{2}}{∂x^{2}} (K ∗ φ)(t, x)

and for φ ∈ C_{0}^{∞}(R) the integrals from (13) and (14) are equal to

\frac{∂}{∂x} (M ∗ φ)(t, x), \frac{∂^{2}}{∂x^{2}} (M ∗ φ)(t, x),

respectively.

Proof. The case of the first derivatives immediately follows from the integrability of ρ_{x}^{r}(s, y) = \frac{y}{rs} ρ^{r}(s, y). The second derivative ρ_{xx}^{r}(s, y) = \frac{y^{2}−rs}{r^{2}s^{2}} ρ^{r}(s, y) is not integrable

and the equality \frac{∂^{2}}{∂x^{2}} (K ∗ φ)(t, x) = (K_{xx} ∗ φ)(t, x) is true if K_{xx} is understood as a generalized function. The standard calculation gives (see, e.g., [4])

(K_{xx}, ψ) = ∫_{0}^{T} ∫_{R} ρ_{xx}(s, y)(ψ(s, y) − ψ(0, 0))dsdy

and consequently,

(K_{xx} ∗ ψ)(t, x) = ∫_{t}^{T} ∫_{R} ρ_{xx}(s − t, y − x)(ψ(s, y) − ψ(t, x))dsdy.

Similarly one can show that

\frac{∂^{2}}{∂x^{2}} (M ∗ φ)(t, x) = (M_{xx} ∗ φ)(t, x) = ∫_{R} ρ_{xx}(T − t, y − x)(ψ(y) − ψ(x))dy. ■

Corollary 1. The operators (11)-(14) are convolutions

(K_{x} ∗ ψ)(t, x), (K_{xx} ∗ ψ)(t, x), (M_{x} ∗ ψ)(t, x), (M_{xx} ∗ ψ)(t, x),

respectively, where K_{x}, K_{xx}, M_{x}, M_{xx} are understood as generalized functions. The following statement is well-known (see, e.g.,[5]). Proposition 2. The mapping

f → f̂ = \frac{1}{2\sqrt{πT}} ∫_{−T}^{T} ∫_{R} f(t, x)e^{− \frac{itnπ}{T} −ipx}dtdx

from L^{2}([−T, T] × R) to L^{2}(Z × R, dndx) is the unitary operator with the inverse

f → f^{∨} = \frac{1}{\sqrt{2T}} ∑_{n}^{∞} e^{\frac{itnπ}{T}} \frac{1}{\sqrt{2π}} ∫_{R} f(n, p)e^{ipx}dp.

Moreover, f̂ ∗ g = 2\sqrt{πT} f̂· ĝ.

Now, using this proposition, we will represent the operators from (11)-(14) as multiplicative operators.

Lemma 1. K̂(n, p) = \frac{T}{2\sqrt{πT}} \frac{(−1)^{n}e^{− \frac{T Rp2}{2}} −1}{inπ− \frac{T Rp2}{2}} .

Proof. By the definition of K, we have

K̂(n, p) = \frac{1}{2\sqrt{πT}} ∫_{−T}^{0} ∫_{R} e^{−in \frac{π}{T} t−ipx} \frac{e^{−x^{2}/2r|t|}}{\sqrt{2πr|t|}} dxdt,

and it remains to use the formula (e^{−x^{2}/2α})^{∧} = \sqrt{α}e^{−αp^{2}/2} [5, p.139]. ■

Lemma 2. The operators (11) and (12) are bounded. The norm of (2.1) is equal to \frac{2}{r} and the norm of (2.1) is estimated from above by \sqrt{\frac{2T}{r}} . Proof. By Proposition 2, the norms of the operators (11) and (12) are equal to 2\sqrt{πT} ∥K̂_{x}∥_{∞} and 2\sqrt{πT} ∥K̂_{xx}∥_{∞}, respectively. Let us calculate 2\sqrt{πT}∥K̂_{xx}∥ _{∞}. From Lemma 1 and by the properties of the Fourier transform we have

K̂_{xx}(n, p) = −p^{2} \frac{T}{2\sqrt{πT}} \frac{(−1)^{n}e^{−T rp^{2}/2} − 1}{inπ − T rp^{2}/2} .

Therefore

2\sqrt{πT} ∥K̂_{xx}∥_{∞} = sup_{p,n} |T p^{2} \frac{(−1)^{n}e^{−T rp^{2}/2} − 1}{inπ − T rp^{2}/2} | =

\frac{2}{r} sup_{q>0,n∈Z} |q \frac{(−1)^{n}e^{−q} − 1}{inπ − q} | = \frac{2}{r} sup_{q>0,n∈Z} |q \frac{(−1)^{n}e^{−q} − 1}{\sqrt{n^{2}π^{2} + q^{2}}} |.

Since

|q \frac{(−1)^{n}e^{−q} − 1}{\sqrt{n^{2}π^{2} + q^{2}}} | ≤ \begin{cases}1 − e^{−q}, & if n = 2k \\ \frac{q}{\sqrt{π^{2}+q^{2}}} (1 + e^{−q}), & if n = 2k + 1, \end{cases}

we have

2\sqrt{πT} ∥K̂_{xx}∥_{∞} = \frac{2}{r} sup_{q} \{(\frac{q}{\sqrt{π^{2} + q^{2}}} (1 + e^{−q})) ∨ (1 − e^{−q})\} = \frac{2}{r} .

The last equality is true because 1 − e^{−q} < 1, r(q) ≡ \frac{q(1+e^{−q})}{\sqrt{π^{2}+q^{2}}} < 1 and lim_{q→∞} r(q) = 1.

Similarly one can obtain the estimation 2\sqrt{πT} ∥K̂_{x}∥_{∞} ≤ \sqrt{\frac{2T}{r}} . ■

Corollary 2. The mapping ψ → (K_{x} ∗ ψ, K_{xx} ∗ ψ) is a bounded operator and his norm is estimated from above by \frac{1}{r} (2 + \sqrt{2rT}).

Lemma 3. The mappings

L^{2}(R) ∈ ψ → ∫_{R} ψ(y)ρ(T − ·, y − ·)dy ∈ L^{2}([0, T] × R)  (15)

and (13) are bounded operators. The operator (14) is bounded as a mapping from W^{1}(R) to L^{2}([0, 1] × R).

Proof. Denote by ρ̂^{x} the Fourier-image of ρ with respect to the variable x, i.e., ρ̂^{x}(t, p) = \frac{1}{\sqrt{2π}} × ∫_{R} ρ(t, x)e^{−ipx}dx. The operator (15) is equivalent to ψ̂ → ρ̂^{x}(T − ·, ·)ψ̂, where

ρ̂^{x}(t, p) = exp(− \frac{p^{2}}{2} |t|). By the equalities ρ̂_{x}^{x}(t, p) = −ipρ̂^{x}(t, p), ρ̂_{xx}^{x}(t, p) = −p^{2}ρ̂^{x}(t, p), the norms of the operators(15), (13), (14) are equal to

I_{1} = sup_{p} ∥ρ̂^{x}(·, p)∥_{L^{2}[0,T]}, I_{2} = sup_{p} ∥ρ̂_{x}^{x}(·, p)∥_{L^{2}[0,T]}, I_{3} = sup_{p} ∥ρ̂_{xx}^{x}(·, p)∥_{L^{2}[0,T]},

respectively. Calculating this expression, we obtain I_{1} = \sqrt{T}, I_{2} = 1, I_{3} = ∞. Since g ∈ W^{1}(R), we have ∫ p^{2}|g(p)|^{2}dp < ∞ and

∫_{R} ∥ρ̂_{xx}^{x}(·, p)∥_{L^{2}[0,T]}|ĝ(p)|^{2}dp = ∫_{R} |p^{2}(1 − e^{−p^{2}T})|ĝ(p)|^{2}dp ≤ ∫_{R} p^{2}|ĝ(p)|^{2}dp < ∞.

Consequently, the function x → ∫_{R} ρ_{xx}(T −·, y−x)g(y)dy belongs to L^{2}(R, L^{2}[0, T]). ■

3. The Equivalence of Solvability of the Bellman Equation and the System of Integral Equations

In this section we show that the solvability of the problems (1), (2) and (6), (7) are equivalent. Proof of Theorem 1. Suppose, that there exists a solution of the Bellman equation (1),

(2) which belongs to the class W^{1,2}. Let r be a strictly positive constant (the meaning of which will be seen later). Applying the generalized Itô formula ([3], [1]) for the process ξ = (\sqrt{r}W_{t}, t ∈ [0, T]) and using the equality (1), we obtain

S(t, ξ_{t}) = ∫_{0}^{t} S_{x}(s, ξ_{s})\sqrt{r}dW_{s} − ∫_{0}^{t} G(s, ξ_{s}, S_{x}(s, ξ_{s}), S_{xx}(s, ξ_{s}))ds,  (16)

where G(s, x, p, q) is defined by (9).

Since the coefficients b, σ are bounded and S_{x}, S_{xx} ∈ L^{2}([0, T] × R), we have

E| ∫_{0}^{t} G(s, ξ_{s}, S_{x}(s, ξ_{s}), S_{xx}(s, ξ_{s}))ds| ≤ const ∫_{0}^{t} ∫_{R} (|S_{x}(s, x)| + |S_{xx}(s, x)|)ρ^{r}(s, x)ds ≤

≤ const ∥ρ^{r}∥_{L^{2}([0,T]×R)}(∥S_{x}∥_{L^{2}([0,T]×R)} + ∥S_{xx}∥_{L^{2}([0,T]×R)}) < ∞.  (17)

Since S(t, ξ_{t}) is bounded, the relations (16), (17) imply that the stochastic integral ∫_{0}^{t} S_{x}(s, S_{s})\sqrt{r}dW_{s} is a uniformly integrable martingale and from (16) we obtain

E(S(t, ξ_{t}) − S(s, ξ_{s})/F_{s}) = −E( ∫_{s}^{t} G(s, ξ_{s}, S_{x}(s, ξ_{s}), S_{xx}(s, ξ_{s}))ds/F_{s})

for any 0 ≤ s ≤ t ≤ T.

Therefore, it follows from the boundary condition (2) and Markov property of ξ_{t} (taking the inequality (17) and the boundedness and continuity of S into account) that

S(t, ξ_{t}) = E(g(ξ_{T}) + ∫_{t}^{T} G(s, ξ_{s}, S_{x}(s, ξ_{s}), S_{xx}(s, ξ_{s}))ds/ξ_{t}),  (18)

and, hence, dt × dP -a. e. the function S satisfies the relation (9). The differentiation of the equation (9) in x implies that the pair (S_{x}, S_{xx}) of the first and second generalized derivatives of the function S will satisfy the system (6), (7).

If the pair (ψ, ψ̃) is a solution of the system (6), (7) such that ψ, ψ̃ ∈ L^{2}([0, T] × R), then it can be easily seen that the function V defined by (10) will be a solution of the equation (6), (7).

Since ψ, ψ̃ ∈ L^{2}([0, T] × R), the function G also belongs to the same class and, therefore, the function V (t, x) defined by (10) will be a solution of the Cauchy problem (see, e.g., [4])

\frac{∂}{∂t} V (t, x) + \frac{1}{2} r \frac{∂^{2}}{∂x^{2}} V (t, x) = G(t, x, ψ(t, x), ψ̃(t, x)),  (19)

with the boundary condition V (T, x) = g(x).

Since the pair (ψ, ψ̃) is a solution of the system (6), (7), taking the first and the second order generalized derivatives (at x) in the equality (10), we obtain that dt × dx-a. e.

V_{x}(t, x) = ψ(t, x), V_{xx}(t, x) = ψ̃(t, x).  (20)

Therefore, (19) and (20) imply that

\frac{∂}{∂t} V (t, x) + \frac{1}{2} r \frac{∂^{2}}{∂x^{2}} V (t, x) = G(t, x, V_{x}(t, x), V_{xx}(t, x))  (21)

which gives that the function V = (V (t, x), t ∈ [0, T], x ∈ R) satisfies the Bellman equation (1), (2). ■

4. The Solvability of the Bellman Equation and the System of Nonlinear Singular Equations

Now consider the nonlinear part of the singular operators (6), (7). The function G(t, x, p, q) defines the nonlinear operator

(ψ, ψ̃) → G̃(ψ, ψ̃) ≡ {G(t, x, ψ(t, x), ψ̃(t, x))}_{(t,x)∈[0,T]×R}

from L^{2}([0, T] × R)^{2} into L^{2}([0, T] × R).

Lemma 4. For each r > r^{∗} = max((C + 1)^{2}, \frac{1}{λ}), the function G and the operator G̃ satisfy the Lipschitz condition with the constant \frac{1}{2} (r − \frac{1}{r}).

Proof. It is sufficient to show that G is Lipschitzian. First note that for r > r^{∗}, r^{∗} = max((C +1)^{2}, \frac{1}{λ}), the inequalities |σ^{2} −r| < r − \frac{1}{r} and |b| < \frac{1}{2} (r − \frac{1}{r}) are true. Therefore, using the inequality | sup_{a∈A} f_{1}(a)−sup_{a∈A} f_{2}(a)| ≤ sup_{a∈A} |f_{1}(a)−f_{2}(a)| which is valid for any f_{1}, f_{2} : A → R, we have

|G(t, x, p_{1}, q_{1}) − G(t, x, p_{2}, q_{2})| ≤ sup_{a∈A} | \frac{1}{2} (σ(t, x, a)^{2} − r))(q_{1} − q_{2}) +

+b(t, x, a)(p_{1} − p_{2})| ≤ \frac{1}{2} (r − \frac{1}{r})(|q_{1} − q_{2}| + |p_{1} − p_{2}|). ■

Proposition 3. The mapping from L^{2}([0, T] × R)^{2} to L^{2}([0, T] × R)^{2} defined by

(φ, φ̃) → ( ∫_{t}^{T} ∫_{R} ρ_{x}^{r}(s − t, y − x)G(s, y, φ(s, y), φ̃(s, y))dsdy,  (22)

∫_{t}^{T} ∫_{R} ρ_{xx}^{r}(s − t, y − x)[G(s, y, φ(s, y), φ̃(s, y)) − G(t, x, φ(t, x), φ̃(t, x))]dsdy) +

+( ∫_{R} ρ_{x}^{r}(T − t, y − x)g(y)dy, ∫_{R} ρ_{xx}^{r}(T − t, y − x)(g(y) − g(x))dy)

is a contraction if T < \frac{2}{r^{5}} .

Proof. By Lemma 4 and Corollary of Lemma 2, the Lipschitz constant of the mapping (22) is equal to \frac{1}{2} (r − \frac{1}{r})( \frac{1}{r} (2 + \sqrt{2T r})) = 1 − \frac{1}{r^{2}} + \frac{1}{2} (1 − \frac{1}{r^{2}} )\sqrt{2T r}.

If T < \frac{2}{r^{5}} , then 1 − \frac{1}{r^{2}} + \frac{1}{2} (1 − \frac{1}{r^{2}} )\sqrt{2T r} < 1 − \frac{1}{r^{2}} + (1 − \frac{1}{r^{2}} ) \frac{1}{r^{2}} = 1 − \frac{1}{r^{4}} < 1. ■

Proof of Theorems 2 and 3. Let 0 ≤ t_{0} < ⋯ < t_{n} < T be a partition of the time interval [0, T] such that t_{i+1} − t_{i} < \frac{2}{r^{5}} . Consider the interval ]t_{n−1} − t_{n}]. According to Proposition 3, the operator (22) is contractive as a mapping from L^{2}([t_{n−1}, t_{n}] × R)^{2} to L^{2}([t_{n−1}, t_{n}]×R)^{2} and there exists a pair (ψ^{n}, ψ̃^{n}) ∈ L^{2}([t_{n−1}, t_{n}]×R)^{2} which uniquely solves the system (6), (7). Therefore Theorem 1 implies that the function

S^{n}(t, x)=∫_{R} g(y)ρ^{r}(T − t, y − x)dy+∫_{t}^{T}∫_{R} G(v, y, ψ^{n}(v, y), ψ̃^{n}(v, y))ρ^{r}(v − t, y − x)dydv

is a solution of the problem (1), (2) on the set ]t_{n−1}, t_{n}] × R with S^{n}(t_{n−1}, x) ∈ W^{1}. Let us consider now the system (6), (7) on the interval ]t_{n−2}, t_{n−1}] with the function g(x) replaced by S^{n}(t_{n−1}, x). Since S^{n}(t_{n−1}, x) ∈ W^{1}, there exsists a solution (ψ^{n−1}, ψ̃^{n−1}) ∈ L^{2}([t_{n−2}, t_{n−1}] × R)^{2} of the system (6), (7) and again by (10) one can construct the solution S^{n−1} of the Bellman equation (1), (2) on the time interval ]t_{n−2}, t_{n−1}], etc. Evidently, S^{i}(t_{i−1}, x) = S^{i−1}(t_{i−1}, x) for each 1 ≤ i ≤ n and it is easy to see that the function

S(t, x) = ∑_{i=1}^{n} S^{i}(t, x)I_{]t_{i−1},t_{i}]}

is a solution of (1), (2).

Now Theorem 1 implies that the pair (S_{x}, S_{xx}) satisfies the system (6), (7).

Let V be a solution of Bellman’s equation (1) (2) from the class W^{1,2}. Let us show that it coincides with the value function of the considered optimal control problem.

Applying the generalized Itô formula ([3], [1]) for the function V and the controlled process X^{u}, we have

V (t, X_{t}^{u}) = V (0, X_{0}) + ∫_{0}^{t} V_{x}(s, X_{s}^{u})σ(s, X_{s}^{u}, u_{s})dW_{s} + ∫_{0}^{t} (L^{u}V)(s, X_{s}^{u})ds, (23)

where

(L^{u}f)(t, X_{t}^{u}) = f_{t}(t, X_{t}^{u}) + b(t, X_{t}^{u}, u_{t})f_{x}(t, X_{t}^{u}) + \frac{1}{2} σ^{2}(t, X_{t}^{u}, u_{t})f_{xx}(t, X_{t}^{u}).

Since the process V (t, X_{t}^{u}) is bounded and E ∫_{0}^{T} |(L^{u}V)(s, X_{s}^{u})|ds < ∞, the stochastic integral in the right-hand side of (23) is a uniformly integrable martingale. On the other hand, we have from (1) that L^{u}V (s, X_{s}^{u}) ≤ 0, and taking expectations in (23) we obtain from the boundary condition (2) that

V (t, X_{t}^{u}) ≥ E^{u}(V (T, X_{T}^{u})/F_{t}) = E^{u}(g(X_{T}^{u})/F_{t}).

Therefore

V (t, x) ≥ sup_{u} E_{t,x}^{u}g(X_{T}^{u}) = S(t, x).  (24)

Let us prove the inverse inequality. Since the function H defined by (5) is continuous in a for each (t, x) and the decision set A is compact by Filippov’s lemma, a measureble function u^{∗} = (u(t, x), t ∈ [0, T], x ∈ R) exists such that

H(t, x, u^{∗}(t, x)) = max_{a∈A} H(t, x, a).

Therefore (L^{u}V)(s, X_{s}^{u^{∗}}) = 0, and using again the Itô formula, we obtain

V (t, x) = E_{t,x}V (T, X_{T}^{u^{∗}}) = E_{t,x}g(X_{T}^{u^{∗}}),

hence V (t, x) = S(t, x). ■

References

1. R. J. Chitashvili and M. G. Mania, Generalized Itô’s formula and derivation of Bellman’s equation. Stochastic Processes and Related Topics, Stochastic Monographs, Gordon & Breach Science Publishers, London, 10(1995), 7–27.

2. M. H. A. Davis, P. P. Varaiya, Dynamic programming conditions for partially observable stochastic systems. SIAM J. of Control Optim. 2(1973), 226–261.

3. N. V. Krylov, Controlled diffusion processes. Springer-Verlag New York, 1980.

4. O. A. Ladizhenskaya, V. A. Solonnikov and N. N. Uraltseva, (1967). Linear and quasi-linear equations of parabolic type (in Russian)

5. M. Reed and B. Simon, Methods of Modern Mathematical Physics, Part 2, Academic Press. (1975).

6. R. Rishel Necessary and sufficient dynamic programming conditions for continuous-time stochastic optimal control, SIAM J. of Control Optim. 8(1970), 559-571.

Authors’ address:

A. Razmadze Mathematical Institute

Georgian Academy of Sciences

1, Aleksidze St., Tbilisi 390093

Georgia