Topic 1: Introduction to Intertemporal Optimization

Yulei Luo

FBE, HKU

September 5, 2018

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 1 / 59

Warm-Up: Intertemporal/Dynamic Optimization

In static optimization, the task is to …nd a single value for each

control variable, such that the objective function will be maximized or
minimized. In contrast, in a dynamic setting, time enters explicitly
and we encounter a dynamic optimization problem. In such a
problem, we need to …nd the optimal time path of control and state
during an entire planning period.
Speci…cally, at any time t, we have to choose the value of some
control variable, c (t ), which will then a¤ect the value of the state
variable, x (t + 1), via a state transition equation given the current
state x (t ).
Two leading methods to solve DO problems are: optimal control (OC)
and dynamic programming (DP). They can be applied in deterministic
or stochastic and discrete-time or continuous-time settings.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 2 / 59

The Setting of Optimal Control (OC)

In the dynamic discrete-time model, the objective should take the

form of summation from t = 0 to t = T :
T
max ∑ r (c (t ) , x (t ) , t ) ,
fc (t ),x (t +1 )g t =0
(1)

subject to

x (t + 1) x (t ) = f (c (t ) , x (t ) , t ) , (2)
x0 = x ( 0 ) , x ( T + 1 ) 0 , (3)

for any t 0.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 3 / 59

(Conti.) As in the static optimization case, the Lagrangian for this
intertemporal (dynamic) problem is
T
L= ∑ fr (ct , xt , t ) + λt [f (ct , xt , t ) + xt xt +1 ]g + λT +1 xT +1 ,
t =0
(4)
where λt is the shadow price (i.e., the Lagrange multiplier).
The FOCs for ct are:
∂L
= rc0 (ct , xt , t ) + λt fc0 (ct , xt , t ) = 0 for t = 0, 1, ,T. (5)
∂ct
The FOCs for xt +1 are complicated because each xt appears in two
terms in the Lagrangian function.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 4 / 59

(Conti.) Consider some relevant terms in the above Lagrangian
function:
T T 1
∑ λt (xt xt +1 ) = (λ0 x0 λT xT +1 ) + ∑ xt +1 (λt +1 λt ) .
t =0 t =0

The FOCs for xt +1 :

∂L
= rx0 (ct +1 , xt +1 , t + 1) + λt +1 fx0 (ct +1 , xt +1 , t + 1)
∂xt +1
+λt +1 λt = 0,

for t = 0, ,T 1.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 5 / 59

Theorem (The Maximum Principle)
The necessary FOCs for maximizing (1) subject to (2) are:
(i) ∂L
∂ct = 0 for all t = 0,
t
,T.
(ii) xt +1 xt = f (ct , xt , t ) for all t (The state transition equation)
(iii) rx0 (ct +1 , xt +1 , t + 1) + λt +1 fx0 (ct +1 , xt +1 , t + 1) + λt +1 λt = 0 for
all t (The costate (λt ) equation)
(iv) xT +1 0, λT +1 0, xT +1 λT +1 = 0 (The transversality condition, or
the complementary-slackness condition)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 6 / 59

Example: Optimal Consumption-Saving Model

Suppose that a consumer has the utility function u (ct ) , where ct is

consumption at time t. The consumer’s utility is concave:

u 0 > 0, u 00 < 0, (6)

and u (ct ) satis…es the Inada conditions (They ensure that

consumption will always be interior):

lim u 0 (c ) = ∞ and lim u 0 (c ) = 0. (7)

c !0 c !∞

The consumer is also endowed with a0 = a (0) , and has income

stream derived from holding the asset: yt = rat , where r is the
interest rate. The consumer uses the income to purchase c. Any
income not consumed is added to the asset holdings as savings:

at + 1 = ( 1 + r ) at ct . (8)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 7 / 59

The Model Setup

The consumer’s lifetime utility maximization problem is to

T
max
fc ,a + g
t t 1
∑ β t u ( ct ) (9)
t =0

subject to

at +1 = Rat ct , t = 0, ,T, (10)

a0 = a ( 0 ) , aT + 1 0, (11)

where β is the consumer’s rate of time preference (β 1) and

R = 1 + r is the gross interest rate.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 8 / 59

(Conti.) Since limc !0 u 0 (c ) = ∞, c (t ) > 0 for all t T . Further, to
ensure that a solution exists, assets are constrained to be nonnegative
after the terminal period T (i.e., when the consumer dies), aT +1 0.
Note that this nonnegativity constraint must bind, i.e., aT +1 = 0. In
fact, this constraint serves two important purposes:
First, without this constraint aT +1 0, the consumer would want to
set aT +1 = ∞ and die with outstanding debts. This is clearly not
feasible.
Second, the fact that the constraint is binding in the optimal solution
guarantees that resources are not being thrown away after T .

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 9 / 59

Using Optimal Control to Solve the Model

Setting up the Lagrangian:

T
L= ∑ βt fu (ct ) + λt [Rat ct at +1 ]g + λT +1 aT +1 (12)
t =0

∂L
The FOC for ct is ∂ct = 0:

u 0 ( ct ) λt = 0 for t = 0, 1, ,T. (13)

∂L
The FOC for at +1 is ∂a t +1 = 0:

λt + βRλt +1 = 0 for t = 0, ,T 1. (14)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 10 / 59

(Conti.) The terminal condition aT +1 0 implies that the
transversality condition (the complementary slackness condition) is

λ T +1 0, aT +1 0, aT +1 λT +1 = 0, (15)

which means that either the asset holdings (a) must be exhausted on
the terminal date, or the shadow price of capital (λt ) must be 0 on
the terminal date. Since u 0 > 0, the marginal value of capital (λ)
cannot be 0 and thus the capital stock should optimally be exhausted
by the terminal date T + 1, i.e., aT +1 = 0.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 11 / 59

(Conti.) Combining (13) with (14) gives

u 0 (ct ) = βRu 0 (ct +1 ), (16)

which is called the Euler equation characterizing optimal consumption

dynamics.
Suppose that the utility function has an isoelastic form
1 γ
ct 1
u ( ct ) = , (17)
1 γ

where γ 0 (Note that when γ = 1, the function reduces to ln ct ).

We have
ct + 1
ct
γ
= βRct +γ1 =) = ( βR )1/γ . (18)
ct
Therefore, if βR > 1, the optimal consumption will rise over time; if
βR < 1, the optimal consumption will decline over time. When
βR = 1, ct = ct +1 .

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 12 / 59

(Conti.) (18) implies that the elasticity of intertemporal substitution
(EIS) is
d ctc+t 1 / ctc+t 1 d ln ctc+t 1 1
= = . (19)
dR/R d ln R γ

Hence, increasing γ i.e., reducing γ1 make the agent more unwilling

to postpone consumption (i.e., more unwilling to save). That’s why
we call this type of utility functions the isoelastic utility function.
The optimal problem is pinned down by a given initial condition (a0 )
and by a terminal condition (aT +1 = 0) . The sets of (T 1) Euler
equations (18) and constraints (10) then determine the time path of
c as well as a.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 13 / 59

Obtaining the Intertemporal Budget Constraint

The original budget constraint (10) implies that

a1 = Ra0 c0 ,
a2 = Ra1 c1 ,

aT +1 = RaT cT .

aT + 1 c c1
T
+ TT + + + c0 = Ra0 =)
R R R
T
ct
∑ Rt = Ra0 , (20)
t =0

where we use the fact that aT +1 = 0.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 14 / 59

(Conti.) For simplicity, assume that γ = 1. Combining (18) with (20)
gives
1 1
c0 + ( βRc0 ) + + β T R T c0 = Ra0 =)
R RT
R
c0 = a0 . (21)
1+β+ + βT
Hence, the time path of optimal consumption is

R t +1 β t
ct = a0 (22)
1+β+ + βT
where t = 0, , T . Given ct , it is straightforward to determine the
time path of at :

R t +1 β t
at +1 = Rat ct = Rat a0 , (23)
1+β+ + βT
where t = 0, , T and aT +1 = 0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 15 / 59
(Conti.) Consider an important special case, βR = 1. In this case,
(18) implies that c0 = c1 = = cT ,

1
c0 1 + + = Ra0 =)
RT
R 1
ct = a
(T +1 ) 0
. (24)
1 R
where t = 0, ,T.
Next, the optimal path of at can be derived by solving the following
di¤erence equation:
r
at +1 = Rat ct = Rat a
(T +1 ) 0
=)
1 R
1 1
at = 1 (T +1 )
a0 R t + a ,
(T +1 ) 0
(25)
1 R 1 R

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 16 / 59

Alternative Way to Solve Optimal Consumption

Instead of de…ning the Lagrange multiplier for each ‡ow budget

constraint, we may de…ne a Lagrange multiplier for the intertemporal
budget constraint (that is, the lifetime budget constraint):
T
ct
∑ Rt = Ra0 ,
t =0

where we have used the fact that aT +1 = 0. That is, we may write
the Lagrangian as follows
!
T T
ct
L = ∑ β u (ct ) + λ Ra0 ∑ t ,
t
t =0 t =0 R

where λ is the constant Lagrange multiplier for the lifetime budget

constraint.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 17 / 59

(Conti.) The FOCs for an optimum are then

1
β t u 0 ( ct ) = λ , where t = 0, ,T. (26)
Rt
Since λ is a constant, the above FOCs implies that the Euler
equations are

u 0 (ct ) = βRu 0 (ct +1 ), where t = 0, ,T 1,

which is identical to those Euler equations we derived above.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 18 / 59

The In…nite Horizon Case

There are some reasons to consider the in…nite horizon (T = ∞) case:

People don’t live forever, but they may care about their o¤spring (a
bequest motive).
Assuming an in…nite horizon eliminates the terminal date complications
(stop saving, consume their entire value, etc. There is no real world
counterpart for these actions because the real economy continue
forever).
Many economic models with a long time horizon tend to show very
similar results to IH models if the horizon is long enough.
IH models are stationary in nature: the remaining time horizon doesn’t
change as we move forward in time.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 19 / 59

The Model Setup

The consumer’s lifetime utility maximization problem

∞
max ∑ βt u (ct ) (27)
c (t ) t =0

subject to at +1 = Rat ct , 8 t 0, given the initial asset holdings

a0 = a ( 0 ) .
In addition, we need to impose the following institutional assumption:
no Ponzi game condition (nPg):
at + 1
lim 0, (28)
t !∞ Rt
means that in present value terms, the agent cannot engage in
borrowing and lending so that his terminal asset holdings are negative.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 20 / 59

(Conti.) (28) can help obtain an intertemporal budget constraint:
T
ct aT + 1
∑ Rt +
RT
= Ra0 .
t =0

Applying (28), we have

!
T T
ct aT + 1 ct aT + 1
lim
T !∞
∑ Rt + RT = lim
T !∞
∑ Rt + lim
T !∞ RT
t =0 t =0
∞
ct aT + 1
= ∑ Rt + lim
T !∞ RT
=)
t =0
∞
ct
∑ Rt Ra0 . (29)
t =0

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 21 / 59

(Conti.) In an optimum, (29) must be binding:
∞
ct
∑ Rt = Ra0 (30a)
t =0

because (28) is binding, i.e., limT !∞ aRt +t 1 = 0 (Otherwise, the

agent can consume the left amount and reach a higher utility level,
which contradicts optimum).
Set up the Lagrangian:
!
∞ ∞
c
L = ∑ βt u (ct ) + λ Ra0 ∑ t
t

t =0 t =0 R

The FOCs w.r.t. ct for any t are

1
β t u 0 ( ct ) λ = 0, 8t 0, (31)
Rt
where λ is the Lagrangian multiplier associated with the
intertemporal budget constraint.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 22 / 59
(Conti.) Note that the FOCs imply the following Euler equation
linking consumption in two consecutive periods

u 0 (ct ) = βRu 0 (ct +1 ) , 8t 0. (32)

Suppose that the utility function is log, u (ct ) = log ct . (31) implies
that ct = ( βR )t c0 , 8t 1. Substituting them into the (30a) gives
!
∞ ∞
( βR )t
∑ R t c0 = Ra0 =) ∑ βt c0 = Ra0 =)
t =0 t =0
c0 = R ( 1 β ) a0 , (33)

and consumption in periods t 1 can be recovered:

ct = ( βR )t c0 = ( βR )t R (1 β ) a0 , 8 t 1. (34)

Similarly, we can determine the optimal time path of asset holdings:

at +1 = Rat ( βR )t R (1 β ) a0 .

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 23 / 59

Dynamic Programming (DP): Application to the
Consumption-Savings Model
We use the same in…nite-horizon lifetime optimal
consumption-savings model to introduce DP and then provide more
general discussions on DP.
Consider the maximization problem
∞
Ut = ∑ βs t
u ( cs )
s =t
subject to as +1 = Ras cs , 8s t, given the initial asset holdings
at = a (t ). In addition, impose:
at + T + 1
lim 0.
T !∞ R t +T
Of course, DP is also applicable with a …nite horizon.
Assume that there is a function, called the value function, that gives
us the maximal constrained value of Ut as a function of the initial
wealth at . We write the value function as J (at ), and we assume it is
di¤erentiable.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 24 / 59
(Conti.) DP rests on a recursive equation involving the value
function. That equation, called the Bellman equation, characterizes
intertemporally optimizing plans:
A consumption plan optimal from the standpoint of t must maximize
Ut +1 subject to the future wealth at +1 produced by today’s
consumption plan ct . If not, the consumer could raise utility by
behaving di¤erently after t.
The foregoing property means that an optimizing consumer can behave
as if Ut = u (ct ) + βJ (at +1 ), where J (at +1 ) is the constrained
maximal value of Ut +1 .
If this is so, ct maximizing lifetime utility is the one maximizing
Ut = u (ct ) + βJ (at +1 ) subject to the same constraint.
Formally, the Bellman equation can be written as

J (at ) = max fu (ct ) + βJ (at +1 )g , (35)

ct
s.t. at +1 = Rat ct . (36)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 25 / 59

(conti.) (35) can be rewritten as:

J (at ) = max fu (ct ) + βJ (Rat ct )g . (37)

ct

The …rst-order condition is thus

u 0 ( ct ) βJ 0 (Rat ct ) = 0. (38)

Note that (38) gives optimal consumption as an implicit function of

current wealth, c = c (a). Substituting it into (37):

J (a) = u (c (a)) + βJ (Ra c (a)) . (39)

Taking di¤erentiation w.r.t. a gives

J 0 (a) = u 0 (c (a)) c 0 (a) + βJ 0 (Ra c (a)) R c 0 (a) , or

J 0 (a) = βRJ 0 (Ra c (a)) , or J 0 (a) = Ru 0 (c ) , (40)

which is called the Envelop theorem.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 26 / 59
(conti.) Implication: For an optimizing consumer, an increment to
wealth on any date has the same e¤ect on lifetime utility regardless of
how to put the wealth: consumption or saving.
Combining (38) with (38), we obtain the usual consumption Euler
equation:
u 0 (ct ) = βRu 0 (ct +1 ) . (41)
Following the same procedure used above, we can easily solve for
optimal consumption path.
Alternatively, we can solve for optimal consumption by using the
value function. Given the same utility function:
1 γ
c
u ( ct ) = t , (42)
1 γ
we …rst guess the form of the value function and then verify that the
guess was right. A natural guess is that the value function takes a
parallel form as u (ct )
a1 γ
J (a ) = A . (43)
1 γ
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 27 / 59
(conti.) Substituting this conjectured solution into
u 0 (ct ) βJ 0 (Rat ct ) = 0,

γ γ R
ct = βA (Rat ct ) , or ct = at . (44)
1 + ( βA)1/γ

Substituting it into J (a) = u (c (a)) + βJ (Ra c (a)) gives

a1 γ c (a )1 γ (Ra c (a))1 γ
A = + βA , (45)
1 γ 1 γ 1 γ
1
A = γ, (46)
1/γ 1 1/γ
β β R 1

ct = ( βR )1/γ β 1/γ
R1 1/γ
1 at , (47)

which is consistent with that we obtained using optimal control.

Finally, given A, we can obtain J (a).

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 28 / 59

Another Application to the Consumption-Savings Model
Consider the following consumption-saving problem with a nonlinear
saving technology:
∞
max U0 =
fct ,kt +1 g
∑ βt ln (ct )
t =0

subject to
kt +1 = θktα ct ,
ct 0, kt 0.
given the initial asset holdings k0 = k (0).
The Bellman equation can be written as:
n o n
V (k ) = max ln (c ) + βV ke or V (k ) = max ln θk α ke + βV
c ,ke ke

Guess that: V (k ) = E ln (k ) + F . It turns out that

V (k ) = 1 ααβ ln (k ) + F , c (k ) = (1 αβ) θk α , and ke (k ) = αβθk α .
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 29 / 59
Dynamic Programming (DP): General Setting
Choose an in…nite sequence of controls fct gt∞=0 to maximize
∞
∑ βt r (xt , ct ) (48)
t =0

subject to xt +1 = g (xt , ct ), with x0 given.

Assume that r (xt , ct ) is a concave function and the set
f(xt +1 , xt ) : xt +1 g (xt , ct )g is convex and compact. DP seeks a
time-invariant policy function c = h (x ) mapping the state (x ) into
the control (c ), such that the sequence fct gt∞=0 generated by iterating
the h and g functions starting from x0 solves the original problem.
To …nd h, we need to know the value function that is the optimal
value of the original problem:
∞
V (x0 ) = max
∞ ∑ βt r (xt , ct )
f c t g t =0 t = 0
(49)

subject to the same constraint.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 30 / 59
Note that if we knew V ( ), c can be obtained by solving for each x
the problem
max fr (x, c ) + βV (e
x )g (50)
c
subject to e
x = g (x, c ) with x given and e x denotes the state next
period.
Key point: Have exchanged the original problem of …nding fct gt∞=0
for the problem of …nding the optimal V (x ) and h (x ) that solves the
continuum of maximum problems (one maximum problem for each x).
The task is now to solve for (V (x ) , h (x )) linked by the Bellman
equation:
V (x ) = max fr (x, c ) + βV (g (x, c ))g . (51)
c
The maximizer of the right side of it is h (x ) that satis…es
V (x ) = r (x, h (x )) + βV (g (x, h (x ))) , (52)
which is a functional equation to be solved for unknown functions:
(V (x ) , h (x )). Note that the function equation must hold for every
value of x within the permitted domain and the solutions to it are
functions, V (x ) and h (x ).
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 31 / 59
Some Properties of the Bellman Equation

In the general setting, we encounter the same problems that we had

in the consumption-savings model:
We need to know the derivatives of the value function, V (x ).
The value function is unknown.
The envelop theorem of Benveniste and Scheinkman (1978) gives a
set of su¢ cient conditions under which one can obtain the derivative
of the value function:
State x 2 X , where X is a convex set with a nonempty interior.
Control c 2 C , where C is a convex set with a nonempty interior.
The period objective function, r , is concave and di¤erentiable.
The RHS of the state transition equation, g , is concave, di¤erentiable,
and invertible in c.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 32 / 59

Under the above particular assumptions about r , g , x, and c, there are
four main …ndings:
1 The Bellman equation, (51), has a unique strictly concave solution.
2 This solution can be approached in the limit as j ! ∞ by iterations on

Vj +1 (x ) = max fr (x, c ) + βVj (e

x )g (53)
c

subject to e
x = g (x, c ) with x given, starting from any bounded and
continuous initial V0 .
3 There is a unique and time-invariant optimal policy of the form
c = h (x ), where h is chosen to maximize the right side of (51).
4 The limiting value function V is di¤erentiable.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 33 / 59

Optimal Conditions

Since V is di¤erentiable, the FOC for (51) is

rc (x, c ) + βV 0 (g (x, c )) gc (x, c ) = 0. (54)

If we also assume that h is di¤erentiable, di¤erentiation of (52) gives

V 0 (x ) = rx (x, h (x )) + rc (x, h (x )) h0 (x ) (55)

0 0
+ βV (g [x, h (x )]) gx (x, h (x )) + gc (x, h (x )) h (x ) ,

which can be reduced to

V 0 (x ) = rx (x, h (x )) + βV 0 (g (x, h (x ))) gx (x, h (x )) . (56)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 34 / 59

Solution Methods

There are three main types of solution methods for solving DP.
1 Guess and verify. It involves guessing and verifying a solution V to
the Bellman equation.
2 Value function iteration. It proceeds by constructing a sequence of
value functions and associated policy functions. The sequence is
created by iterating on the following equation, starting from V0 = 0,
and continuing until Vj has converged:

Vj +1 (x ) = max fr (x, c ) + βVj (e

x )g (57)
c

subject to e
x = g (x, c ) with x given.
3 Policy function iteration (or Howard’s improvement algorithm).

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 35 / 59

Value Function Iteration
First, consider an initial guess for the value function V0 (x ). (It does
not matter very much and we usually guess it is 0.)
Second, one can then calculate an updated value function V1 (x )
using:
V1 (x ) = max fr (x, c ) + βV0 (g (x, c ))g , (58)
c
and doing the maximization numerically over the domain of x. This
maximization de…nes approximately the function V1 (x ).
Using this new function V1 (x ), one can update again and get a new
V2 (x ):
V2 (x ) = max fr (x, c ) + βV1 (g (x, c ))g , (59)
c
over the domain of x. Repeated this step until the sequence of the
approximate value functions converges to V (x ). Note that in the
repeated calculations of the value function, one is also calculating
repeated approximations to the policy function, c = h (x ).
Note that the subscript (0, 1, 2, ) represents the iteration on the
value function and does not have any link to time periods.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 36 / 59
Application: Value Function Iteration

Essentially, the value function iteration is searching for a …xed point,

at which point we have numerically solved the value function.
When applying to the consumption-saving problem, we can write the
process as:
n o
ΓV (k ) = max ln θk α ke + βV ke , 8k, (60)
ke

where Γ is an operator representing this process of iteration on the

value function until ΓV (k ) = V (k ).

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 37 / 59

 Value Function Iteration
10

5
Value Function, V

-5

-10

-15
0 10 20 30 40 50 60 70 80 90 100
Amount of Capital, k

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 38 / 59

Finite-horizon Dynamic Programming
The basic idea of …nite-horizon DP is to solve the optimization
problem period by period – starting from the terminal period, taking
the previous periods’solutions as given, and then working back
sequentially to the …rst period.
Having optimized the terminal period (T ), this solution is substituted
into the period T 1 problem and then period T 1 is optimized.
Substituting these previous solutions, the solutions for T 2, T 3,
, 0 are obtained in sequence in the same way.
Suppose that the problem is to maximize:
U (xt ) = r (xt , ct ) + βV (xt +1 ) , (61)
for t = 0, , T , subject to
xt +1 = f (xt , ct ) , (62)
with x0 given and xT +1 = x. Note that here we assume that the
value function, V (xt +1 ), is time invariant and later we can see that it
is correct.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 39 / 59
Consider the problem for period T . First, we maximize

U (xT ) = r (xT , cT ) + βV (xT +1 ) , (63)

with respect to cT , subject to xT +1 = f (xT , cT ) and taking xT as

given. Thus we maximize

U (xT ) = r (xT , cT ) + βV (f (xT , cT )) .

The FOC is
∂r (xT , cT ) ∂V (f (xT , cT ))
+β = 0,
∂cT ∂cT
which means that the solution for cT has the form:

cT = gT (xT ) . (64)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 40 / 59

(Conti.) Substituting this solution into U (xT ):

U (xT ) = r (xT , cT ) + βV (f (xT , cT ))

= r (f (xT 1 , cT 1 ) , gT (f (xT 1 , cT 1 )))
+ βV (f (f (xT 1 , cT 1 ) , gT (f (xT 1 , cT 1 ))))
, VT (xT 1 , cT 1 ) .

Turning to period T 1, we maximize

U (xT 1) = r (xT 1 , cT 1) + βV (xT ) ,

w.r.t. cT 1 , subject to xT = f (xT 1 , cT 1) and taking xT 1 as

given. The FOC is

∂r (xT 1 , cT 1) ∂VT (xT 1 , cT 1)

+β = 0, (65)
∂cT 1 ∂cT 1
from which we can write the solution for cT 1 as
cT 1 = gT 1 (xT 1 ) .
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 41 / 59
Substituting this into U (xT 1) gives

U (xT 1) = r (xT 1 , gT 1 (xT 1 )) + βV (f (xT 1 , gT 1 (xT 1 )))

, VT 1 (xT 2 , cT 2 ) . (66)

We can proceed similarly in periods T 2, , 0.

The general solution for period t has the FOC

∂r (xt , ct ) ∂V (f (xt , ct ))
+β = 0, (67)
∂ct ∂ct
and optimal control:
ct = gt (xt ) . (68)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 42 / 59

Application to the Same Consumption-Saving Model

Consider the same consumption-saving problem as above:

T
V ( a0 ) = max
fc ,a + g
t t 1
∑ β t u ( ct )
t =0
s.t.at +1 = Rat ct , f (at , ct ) , t = 0, ,T,

given a0 = a (0) and aT +1 = 0, where u (ct ) = ln (ct ).

We …rst consider the solution for period T . This requires to maximize

U (aT ) = u (cT ) + βV (aT +1 ) , (69)

with respect to cT , subject to aT +1 = f (aT , cT ), aT +1 = 0, and

taking aT as given. We thus have V (aT ) = u (cT ), i.e., no
maximization is required for the terminal period: cT = RaT , and

U (aT ) = u (RaT ) + βV (0) = ln (RaT ) + βV (0) . (70)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 43 / 59

(Conti.) The T 1 problem is to maximize:

U ( aT 1) = u ( cT 1) + βV (aT )
= u ( cT 1) + β [ln (RaT ) + βV (0)] (71)

with respect to cT 1 , subject to aT = RaT 1 cT 1 and taking

aT 1 as given. The FOC is:
1 1
β = 0,
cT 1 RaT 1 cT 1

which implies that

R
cT 1 = aT 1 (72)
1+β
and
R βR 2
V ( aT 1) = ln aT 1 + β ln aT 1 + βV (0) .
1+β 1+β
(73)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 44 / 59
(Conti.) Similarly, solving the T 2 problem yields the following
optimal solution:
R
cT 2 = aT 2. (74)
1 + β + β2
It is straightforward to show that the general solution for periods
0, 1, , T takes the following form:
R
ct = 2
a,
t t
(75)
1+β+β + + βT
which is identical to the corresponding solution we obtained using
optimal control (i.e., the Lagrange multiplier method). As T ! ∞,
we have
ct = (1 β) Rat . (76)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 45 / 59

From Discrete-time to Continuous-time

We may treat the continuous-time case as a limiting case of the

discrete-time case as the time interval (∆t ) goes to 0. (Note that in
the discrete-time models, ∆t = 1.)
Speci…cally, in discrete-time, the ‡ow constraint is

x (t + 1) x (t ) = f (c (t ) , x (t ) , t ) , (77)

which can be rewritten as

x (t + ∆t ) x (t ) = f (c (t ) , x (t ) , t ) ∆t (78)

if the time interval is ∆t.

Dividing both sides by ∆t and letting it goes to 0 gives

x (t + ∆t ) x (t )
xt0 = lim = f (c (t ) , x (t ) , t ) (79)
∆t !0 ∆t

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 46 / 59

(Conti.) Further, note that when the time interval is ∆t, the objective
function can be written as
T /∆t
∑ r (c (i ∆t ) , x (i ∆t ) , i ∆t ) ∆t; (80)
i =0

as ∆t ! 0,
T /∆t Z T
lim ∑
∆t !0 i =0
r (c (i ∆t ) , x (i ∆t ) , i ∆t ) ∆t =
0
r (c (t ) , x (t ) , t ) dt
(81)
Hence, the continuous-time version of the dynamic optimization
problem can be written as
Z T
max r (c (t ) , x (t ) , t ) dt,
c (t ) 0
dx (t )
s.t. xt0 = = f (c (t ) , x (t ) , t ) ,
dt
x0 = x (0) , and x (T ) 0.
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 47 / 59
Solving the Model

The Lagrange function for the above intertemporal problem is

Z T
L = r (ct , xt , t ) + λt f (ct , xt , t ) xt0 dt
0
Z T
= r (ct , xt , t ) + λt f (ct , xt , t ) + λt0 xt dt + λ0 x0 λ T xT
0

where we use the fact that the rule of integration by parts implies
Z T Z T
λt xt0 dt = λt0 xt dt + λ0 x0 λ T xT (82)
0 0

Now we can regard the integral as the sum and take …rst derivatives
w.r.t. ct and xt :

rc (ct , xt , t ) + λt fc (ct , xt , t ) = 0 (83)

rx (ct , xt , t ) + λt fx (ct , xt , t ) + λt0 = 0 (84)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 48 / 59

Hamiltonian Function
Similarly, we can de…ne a new function H, a Hamiltonian function,
H = r (ct , xt , t ) + λt [f (ct , xt , t )] . (85)
Hence, the FOC for ct , (83), can be rewritten as
∂Ht
= 0. (86)
∂ct

Theorem (The Maximum Principle)

The necessary FOCs for maximizing (81) subject to (79) are:
(i) Hc , ∂H∂ct = 0 for all t 2 [0, T ] .
t

0
(ii) xt = f (ct , xt , t ) for all t (The state transition equation)
(iii) Hx + λt0 = 0 rx (ct , xt , t ) + λt fx (ct , xt , t ) + λt0 = 0 for all t (The
costate (λt ) equation)
(iv) xT 0, λT 0, xT λT = 0 (The transversality condition, or the
complementary-slackness condition)
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 49 / 59
Continuous-time Version of the Consumption-Saving Model

Consider the following deterministic continuous-time optimal

consumption-saving problem:
Z T
max u (ct ) exp ( ρt ) dt, (87)
ct 0

subject to
at0 = rat ct , (88)
where ρ is called the discount rate, r is the instantaneous interest
rate, given a0 , and aT 0 (preventing the consumer from dying with
debts).
Here we assume that the utility function u (ct ) satis…es the usual
conditions (just like those discussed in the discrete-time case) and
ct > 0.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 50 / 59

The Hamiltonian for this problem is

H = u (ct ) exp ( ρt ) + λt (rat ct ) . (89)

Using the maximum principle,

Hc = 0 =) λt = u 0 (ct ) exp ( ρt ) , (90)

Ha + λt0 = 0 =) r λt + λt0 = 0, (91)
at0 = rat ct . (92)

In addition, aT = 0, i.e., it is optimal for the consumer to leave no

bequest after T . Note that if at optimum aT > 0, λT > 0, which
contradicts (90).

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 51 / 59

(Conti.) Using (91), we have

λt0
= r, (93)
λt
which can be easily solved for λt :

λt = λ0 exp ( rt ) . (94)
1 γ
ct 1
Suppose that u (ct ) = 1 γ . Then combining (90) with (94) gives

γ
ct = c0 γ exp ((ρ r ) t) . (95)

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 52 / 59

(Conti.) Di¤erentiating (91) gives

at00 = rat0 ct0 .

Since
dct u 0 ( ct ) r ρ
ct0 , = 00 (ρ r) = ct ,
dt u ( ct ) γ
we have
ρ r
at00 = rat0 + ct
γ
ρ r
= rat0 + rat at0
γ
ρ r r (ρ r )
= r at0 + at , (96)
γ γ

which is a second-order di¤erential equation (SODE) in terms of at .

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 53 / 59

 1
(Conti.) For r 6= γ (ρ r ), (i.e., no repeated roots for the
2 ρ r r (ρ r )
characteristic equation, λ r γ λ γ = 0,) this SODE
has the general solution:
ρ r
at = A exp (rt ) + B exp t , (97)
γ
where A and B can be pinned down by using the initial and terminal
conditions:
a0 = A + B, (98)
ρ r
= A exp (rT ) + B exp
aT T (= 0) . (99)
γ
h i
1
From (99), we have A = B exp γ (r ρ) r T .
Substituting it into (98):
r ρ
B = 1 exp r T a0 ,
γ
r ρ r ρ
A = exp r T 1 exp r T a0 .
γ γ
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 54 / 59
Optimal consumption:
ρ r
ct , rat at0 = r A exp (rt ) + B exp t
γ
ρ r ρ r
Ar exp (rt ) B exp t .
γ γ
Consider a special case in which r = ρ:
u 0 ( ct )
ct0 = (ρ r ) = 0,
u 00 (ct )
i.e., ct = c, independent of time t. In this case, at0 = rat ct
becomes at0 = rat c, whose solution is:
c
at = exp (rt ) a0 (1 exp ( rt )) . (100)
r
c
At T , we have aT = exp (rT ) a0 r (1 exp ( rT )) = 0:
r
c= a0 . (101)
1 exp ( rT )
Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 55 / 59
Continuous-time Bellman Equation (Optional)

Consider the same deterministic continuous-time consumption-saving

problem: Z ∞
J (a0 ) = max u (ct ) exp ( ρt ) dt (102)
ct 0

subject to at0 = rat ct , given a0 , and aT 0. Here J (a0 ) is de…ned

as the value function.
(102) can be rewritten as the following recursive form:
Z ∆t Z ∞
J (a0 ) = max u (ct ) exp ( ρt ) dt + u (ct ) exp ( ρt ) dt
c 0 ∆t
= max fu (c0 ) ∆t + exp ( ρ∆t ) J (a∆t )g (103)
c0

for a small interval, ∆t.

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 56 / 59

(Conti.) Here we use the facts:
Z ∆t Z ∆t
u (ct ) exp ( ρt ) dt = u (c0 ) exp ( ρt ) dt = u (c0 ) ∆t,
Z0 ∞ 0

u (ct ) exp ( ρt ) dt
∆t
Z ∞
= exp ( ρ∆t ) u (ct ) exp ( ρ (t ∆t )) d (t ∆t )
0
, exp ( ρ∆t ) J (a∆t ) . (104)

Linearizing J (a∆t ) around a0 :

J (a∆t ) = J (a0 ) + J 0 (a0 ) (a∆t a0 ) .

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 57 / 59

(Conti.) Substituting it back into (103):

J ( a0 ) = max u (c0 ) ∆t + (1 ρ∆t ) J (a0 ) + J 0 (a0 ) (a∆t a0 )

c0
u (c0 ) ∆t + J 0 (a0 ) (a∆t a0 )
() ρ∆tJ (a0 ) = max ,(105
c0 ρ∆tJ 0 (a0 ) (a∆t a0 )

where we use the fact that exp ( ρ∆t ) = 1 ρ∆t.

Dividing ∆t on both sides:
a ∆t a 0
u (c0 ) + J 0 (a0 ) lim∆t !0 ∆t
ρJ (a0 ) = max
c0 ρJ 0 (a0 ) lim∆t !0 (a∆t a0 )
= max u (c0 ) + J 0 (a0 ) a00
c0
= max u (c0 ) + J 0 (a0 ) (ra0 c0 ) (106)
c0

and taking derivative w.r.t. the control c0 : u 0 (c0 ) = J 0 (a0 ), which

implies that c0 = h (a0 ).

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 58 / 59

(Conti.) Substituting c0 = h (a0 ) into the Bellman equation (106):

ρJ (a0 ) = u (h (a0 )) + J 0 (a0 ) [ra0 h (a0 )] . (107)

The envelop theorem implies that

ρJ 0 (a0 ) = u 0 (h (a0 )) h0 (a0 ) + J 00 (a0 ) [ra0 h (a0 )]

0 0
+ J ( a0 ) r h ( a0 )
= J 00 (a0 ) [ra0 h (a0 )] + J 0 (a0 ) r ,

which means that

dc0
(ρ r ) u 0 (c0 ) = u 00 (c0 ) [ra0 h (a0 )] ,
da0
c00 ( ρ r ) u 0 ( c0 ) r ρ
= 00
, , (108)
c0 c0 u ( c0 ) γ
dc0
where we use the fact that u 00 (c0 ) da 0
= J 00 (a0 ).

Luo, Y. (FBE, HKU) ECON6012: Macro Theory September 5, 2018 59 / 59