1 Introduction
1.1 Previous literature
1.2 Structure of the paper
-
In Sect. 2, we present the basic setup, and in Sect. 3, we discuss the concept of equilibrium. This replaces in our setting the optimality concept for a standard stochastic control problem, and in Definition 3.4, we give a precise definition of the equilibrium control and the equilibrium value function.
-
Since the equilibrium concept in continuous time is quite delicate, we build the continuous-time theory on the discrete-time theory previously developed in [5]. In Sect. 4, we start to study the continuous-time problem by going to the limit for a discretized problem, and using the results from [5]. This leads to an extension of the standard HJB equation to a system of equations with an embedded static optimization problem. The limiting procedure described above is done in an informal manner. It is largely heuristic, and it thus remains to clarify precisely how the derived extended HJB system is related to the precisely defined equilibrium problem under consideration.
-
The needed clarification is in fact delivered in Sect. 5. In Theorem 5.2, which is the main theoretical result of the paper, we give a precise statement and proof of a verification theorem. This theorem says that a solution to the extended HJB system does indeed deliver the equilibrium control and equilibrium value function to our original problem.
-
Section 7 treats the infinite-horizon case.
-
In Sect. 8, we study a time-inconsistent version of the linear-quadratic regulator to illustrate how the theory works in a concrete case.
-
Section 9 is devoted to a rather detailed study of a general equilibrium model for a production economy with time-inconsistent preferences.
-
In Sect. 10, we review some remaining open problems.
2 The model
-
For any fixed \(u \in{{\mathbb{R}^{k}}}\), the functions \(\mu^{u}\), \(\sigma^{u}\) and \(C^{u}\) are defined by$$\begin{aligned} \mu^{u} (t,x) =&\mu(t,x,u),\quad\sigma^{u} (t,x)=\sigma(t,x,u), \\ C^{u}(t,x) =&\sigma(t,x,u)\sigma(t,x,u)'. \end{aligned}$$
-
For any admissible control law \({\mathbf{u}}\), the functions \(\mu^{\mathbf{u}}\), \(\sigma^{\mathbf{u}}\), \(C^{\mathbf{u}}(t,x)\) are defined by$$\begin{aligned} \mu^{\mathbf{u}} (t,x) =&\mu\big(t,x,{\mathbf{u}}(t,x)\big),\quad \sigma^{\mathbf{u}} (t,x)=\sigma\big(t,x,{\mathbf{u}}(t,x)\big), \\ C^{\mathbf{u}}(t,x) =&\sigma\big(t,x,{\mathbf{u}}(t,x)\big)\sigma \big(t,x,{\mathbf{u}}(t,x)\big)'. \end{aligned}$$
-
For any fixed \(u \in{{\mathbb{R}^{k}}}\), the operator \({{\mathbf{A}}} ^{u}\) is defined by$${{\mathbf{A}}}^{u}=\frac{\partial}{\partial t} +\sum_{i=1}^{n}\mu _{i}^{u}(t,x)\frac{\partial}{\partial x_{i}}+ \frac{1}{2}\sum_{i,j=1} ^{n}C_{ij}^{u}(t,x)\frac{{\partial}^{2} }{\partial{x_{i}}\partial {x_{j}}}. $$
-
For any admissible control law \({\mathbf{u}}\), the operator \({{\mathbf{A}}}^{\mathbf{u}}\) is defined by$${{\mathbf{A}}}^{\mathbf{u}}=\frac{\partial}{\partial t} + \sum_{i=1} ^{n}\mu_{i}^{\mathbf{u}}(t,x)\frac{\partial}{\partial x_{i}}+ \frac{1}{2}\sum_{i,j=1}^{n}C_{ij}^{\mathbf{u}}(t,x)\frac{{\partial} ^{2} }{\partial{x_{i}}\partial{x_{j}}}. $$
3 Problem formulation
-
The present state \(x\) appears in the function \(F\).
-
In the second term, we have (even apart from the appearance of the present state \(x\)) a nonlinear function \(G\) operating on the expected value \(E_{t,x}[ X_{T}^{{\mathbf{u}}} ]\).
-
Consider a non-cooperative game where we have one player for each point in time \(t\). We refer to this player as “Player \(t\)”.
-
For each fixed \(t\), Player \(t\) can only control the process \(X\) exactly at time \(t\). He/she does that by choosing a control function \({\mathbf{u}}(t, \cdot)\); so the action taken at time \(t\) with state \(X_{t}\) is given by \({\mathbf{u}}(t, X_{t})\).
-
Gluing together the control functions for all players, we thus have a feedback control law \({{\mathbf{u}}}:{[0,T] \times\mathbb{R}^{n}} \rightarrow{\mathbb{R}^{k}}\).
-
Given the feedback law \({\mathbf{u}}\), the reward to Player \(t\) is given by the reward functional$$J(t, x,{\mathbf{u}})=E_{t,x}[ F(x,X_{T}^{{\mathbf{u}}}) ] + G( {x, E _{t,x}[ X_{T}^{{\mathbf{u}}} ]} ). $$
-
If for each \(s > t\), Player \(s\) chooses the control \(\hat{{\mathbf{u}}}(s, \cdot)\), then it is optimal for Player \(t\) to choose \(\hat{{\mathbf{u}}}(t, \cdot)\).
4 An informal derivation of the extended HJB equation
-
We discretize (to some extent) the continuous-time problem. We then use our results from discrete-time theory to obtain a discretized recursion for \(\hat{{\mathbf{u}}}\), and we then let the time step tend to zero.
-
In the limit, we obtain our continuous-time extension of the HJB equation. Not surprisingly, it will in fact be a system of equations.
-
In the discretizing and limiting procedure, we mainly rely on informal heuristic reasoning. In particular, we do not claim that the derivation is rigorous. The derivation is, from a logical point of view, only of motivational value.
-
In Sect. 5, we then go on to show that our (informally derived) extended HJB equation is in fact the “correct” one, by proving a rigorous verification theorem.
4.1 Deriving the equation
-
Choose an arbitrary initial point \((t,x)\). Also choose a “small” time increment \(h >0\) and an arbitrary admissible control \({\mathbf{u}}\).
-
Define the control law \({\mathbf{u}}_{h}\) on the time interval \([t,T]\) by$${\mathbf{u}}_{h}(s,y)= \left\{ \textstyle\begin{array}{ccl} {\mathbf{u}}(s,y)&& \mbox{for} \ t \leq s < t+h, y \in\mathbb{R}^{n}, \\ \hat{{\mathbf{u}}}(s,y)&& \mbox{for} \ t+h \leq s \leq T, y \in \mathbb{R}^{n}. \end{array}\displaystyle \right. $$
-
If now \(h\) is “small enough”, we expect to haveand in the limit as \(h \rightarrow0\), we should have equality if \({\mathbf{u}}(t,x)=\hat{{\mathbf{u}}}(t,x)\).$$J(t,x,{\mathbf{u}}_{h}) \leq J(t,x,\hat{{\mathbf{u}}}), $$
-
For any fixed \(y \in\mathbb{R}^{n}\), the mapping \({f^{y}}:{[0,T] \times\mathbb{R}^{n}}\rightarrow{\mathbb{R}}\) is defined by$$f^{y}(t,x)=E_{t,x}[ F( {y,X_{T}^{\hat{{\mathbf{u}}}}} ) ]. $$
-
The function \({f}:{[0,T] \times\mathbb{R}^{n} \times\mathbb{R}^{n}} \rightarrow{\mathbb{R}}\) is defined byWe sometimes also, with a slight abuse of notation, denote the entire family of functions \(\left\{ {f^{y}: y\in\mathbb{R}^{n}} \right\} \) by \(f\).$$f(t,x,y)=f^{y}(t,x). $$
-
For any function \(k(t,x)\), the operator \({\mathbf{A}}_{h}^{{\mathbf{u}}}\) is defined by$$( {{\mathbf{A}}_{h}^{{\mathbf{u}}}k} )(t,x)= E_{t,x}[ k(t+h, X_{t+h} ^{\mathbf{u}}) ]-k(t,x). $$
-
The function \({g}:{[0,T] \times\mathbb{R}^{n}}\rightarrow{\mathbb{R} ^{n}}\) is defined by$$g(t,x)=E_{t,x}[ X_{T}^{\hat{{\mathbf{u}}}} ]. $$
-
The function \(G \diamond g\) is defined by$$\left( {G \diamond g} \right) (t,x)=G\big( {x,g(t,x)} \big). $$
-
The term \({\mathbf{H}}_{h}^{{\mathbf{u}}}g\) is defined by$$( {{\mathbf{H}}_{h}^{{\mathbf{u}}}g} )(t,x)=G\big(x,E_{t,x}[ g(t+h,X _{t+h}^{{\mathbf{u}}}) ]\big)- G\big(x,g(t,x)\big). $$
-
In the expressions above, \(\hat{{\mathbf{u}}}\) always denotes the control law which realizes the supremum in the first equation.
-
In order to solve the \(V\)-equation, we need to know \(f\) and \(g\), but these are determined by the equilibrium control law \(\hat{{\mathbf{u}}}\), which in turn is determined by the sup-part of the \(V\)-equation.
-
We have used the notation$$\begin{aligned} f(t,x,y) =&f^{y}(t,x),\quad\left( {G \diamond g} \right) (t,x)=G \big(x,g(t,x)\big), \\ {\mathbf{H}}^{u}g(t,x) =&G_{y}\big(x,g(t,x)\big) {\mathbf{A}}^{u}g(t,x), \quad G_{y}(x,y)=\frac{\partial G}{\partial y}(x,y). \end{aligned}$$
-
The operator \({\mathbf{A}}^{u}\) only operates on variables within parentheses. So for instance, the expression \(\left( {{\mathbf{A}} ^{u}f} \right) (t,x,x)\) is interpreted as \(\left( {{\mathbf{A}}^{u}h} \right) (t,x)\) with \(h\) defined by \(h(t,x)=f(t,x,x)\). In the expression \(\left( {{\mathbf{A}}^{u}f^{y}} \right) (t,x)\) the operator does not act on the upper case index \(y\), which is viewed as a fixed parameter. Similarly, in the expression \(\left( {{\mathbf{A}}^{u}f ^{x}} \right) (t,x)\), the operator only acts on the variables \(t,x\) within the parentheses, and does not act on the upper case index \(x\).
-
If \(F(x,y)\) does not depend on \(x\) and there is no \(G\)-term, the problem trivializes to a standard time-consistent problem. The terms \(\left( {{\mathbf{A}}^{u}f} \right) (t,x,x)+\left( {{\mathbf{A}}^{u}f ^{x}} \right) (t,x)\) in the \(V\)-equation cancel, and the system reduces to the standard Bellman equation$$( {{\mathbf{A}}^{u}V} )(t,x)=0,\quad V(T,x)=F(x). $$
4.2 Existence and uniqueness
5 A verification theorem
-
In the second step, we then prove that \(\hat{{\mathbf{u}}}\) is indeed an equilibrium control law.
6 The general case
7 Infinite horizon
8 Example: the time-inconsistent linear-quadratic regulator
-
The value functional for Player \(t\) is given bywhere \(\gamma\) is a positive constant.$$E_{t,x}\left[ \frac{1}{2}\int_{t}^{T}u_{s}^{2}ds \right] + \frac{ \gamma}{2}E_{t,x}[ \left( {X_{T}-x} \right) ^{2} ], $$
-
The state process \(X\) is scalar with dynamicswhere \(a\), \(b\) and \(\sigma\) are given constants.$$dX_{t}=(aX_{t} + bu_{t})dt+\sigma dW_{t}, $$
-
The control \(u\) is scalar with no constraints.
9 Example: a Cox–Ingersoll–Ross production economy with time-inconsistent preferences
9.1 The model
9.2 Equilibrium definitions
-
intrapersonal equilibrium;
-
market equilibrium.
9.2.1 Intrapersonal equilibrium
9.2.2 Market equilibrium
9.3 Main goals of the study
9.4 The extended HJB equation
9.5 Determining market equilibrium
-
The equilibrium short rate is given by$$ r(t,x)=\alpha+ \sigma^{2} \frac{xf_{xx}(t,x,t,x)}{f_{x}(t,x,t,x)}. $$(9.3)
-
The equilibrium Girsanov kernel \({\varphi}\) is given by$$ {\varphi}(t,x)=\sigma\frac{xf_{xx}(t,x,t,x)}{f_{x}(t,x,t,x)}. $$(9.4)
-
The extended equilibrium HJB system has the form$$ \textstyle\begin{array}{rcl} \displaystyle U(t,x,t,\hat{c}) +f_{t}+ ( {\alpha x -\hat{c}} )f_{x}+\frac{1}{2}x ^{2} \sigma^{2} f_{xx}&=&0, \\ {\mathbf{A}}^{\hat{{\mathbf{c}}}}f^{sy}(t,x)+U\left( {s,y,t,\hat{c}(t,x)} \right) &=&0. \end{array} $$(9.5)
-
The equilibrium consumption \(\hat{c}\) is determined by the first order condition$$U_{c}(t,x,t,\hat{c})=f_{x}(t,x,t,x). $$
-
The term \({\mathbf{A}}^{\hat{{\mathbf{c}}}}f^{tx}(t,x)\) is given by$${\mathbf{A}}^{\hat{{\mathbf{c}}}}f^{tx}(t,x)=f_{t}+ x\left( {\alpha-r} \right) f_{x}+ (rx-\hat{c})f_{x}+\frac{1}{2}x^{2} \sigma^{2} f_{xx}. $$
-
The equilibrium dynamics for \(X\) are given by$$dX_{t}=( {\alpha X_{t} -\hat{c}_{t}} )dt + X_{t}\sigma dW_{t}. $$
9.6 Recap of standard results
9.7 The stochastic discount factor
9.7.1 A representation formula for \(M\)
9.8 Production economy with non-exponential discounting
9.8.1 Generalities
-
The equilibrium short rate is given by$$r(x)=\alpha+ \sigma^{2} \frac{xV_{xx}(x)}{V_{x}(x)}. $$
-
The equilibrium Girsanov kernel \({\varphi}\) is given by$${\varphi}(x)=\sigma\frac{xV_{xx}(x)}{V_{x}(x)}. $$
-
The extended equilibrium HJB system has the form$$\begin{aligned} U(\hat{c}) +g_{t}(0,x)+ ( {\alpha x -\hat{c}} )g_{x}(0,x)+\frac{1}{2}x ^{2} \sigma^{2} g_{xx}(0,x) =&0, \\ {\mathbf{A}}^{\hat{{\mathbf{c}}}}g(t,x)+\beta(t)U\big( {\hat{c}(x)} \big) =&0. \end{aligned}$$
-
The function \(g\) has the representation$$g(t,x)=E_{0,x}\left[ \int_{0}^{\infty}\beta(t+ s)U( {\hat{c}_{s}} )ds \right] . $$
-
The equilibrium consumption \(\hat{c}\) is determined by the first order condition$$ U_{c}(\hat{c})=g_{x}(0,x). $$(9.17)
-
The term \({\mathbf{A}}^{\hat{{\mathbf{c}}}}g(t,x)\) is given by$$ {\mathbf{A}}^{\hat{{\mathbf{c}}}}g(t,x)=g_{t}(t,x)+ \left( {\alpha x - \hat{c}(x)} \right) g_{x}(t,x)+ \frac{1}{2}x^{2} \sigma^{2} g_{xx}(t,x). $$
-
The equilibrium dynamics of \(X\) are given by$$ dX_{t}=( {\alpha X_{t} -\hat{c}_{t}} )dt + X_{t}\sigma dW_{t}. $$(9.18)
9.8.2 Power utility
9.8.3 Checking the verification theorem conditions for power utility
10 Conclusion and open problems
-
A theorem proving convergence of the discrete-time theory to the continuous-time limit. For the quadratic case, this is done in [8], but the general problem is open.
-
An open and difficult problem is to provide conditions on primitives which guarantee that the functions \(V\) and \(f\) are regular enough to satisfy the extended HJB system.
-
A related (hard) open problem is to prove existence and/or uniqueness for solutions of the extended HJB system.
-
Another related problem is to give conditions on primitives which guarantee that the assumptions of the verification theorem are satisfied.
-
The present theory depends critically on the Markovian structure. It would be interesting to see what can be done without this assumption.