Existence and Uniqueness of Q-Functions

Suppose that $$\left\{ {{X_n},n = 0,1,...} \right\}$$ is a discrete time Markov chain with discrete state space E and stationary transition probabilities. Then the reader is well aware that such a stochastic process is uniquely determined by the one-step transition matrix P whose i,jth component is $${p_{ij = }}\Pr \left\{ {{X_{n + 1}} = j\left| {{X_n} = i} \right.} \right\},$$, and an initial distribution vector p, whose ith component is $${p_i} = \Pr \left\{ {{X_0} = i} \right\}.$$. Every probability involving the random variables of this chain can be determined from the finite-dimensional distributions $$\Pr \left\{ {{X_{{n_1}}} = {i_1},{X_{{n_2}}} = {i_2},...,{X_{{n_k}}} = {i_k}} \right\}$$, and the latter can be expressed in the form $$\sum\limits_i p iPi{i_1}\left( {{n_1}} \right)P{i_1}{i_2}\left( {{n_2} - {n_1}} \right)...{P_{{i_{k - 1}}}}_{{i_k}}\left( {{n_k} - {n_{k - 1}}} \right),$$, where P _ij (n) is the i,jth element of the n-step transition matrix Pn, which is the nth power of the one-step transition matrix P. Furthermore, the one-step transition probabilities P _ij have very obvious meaning in terms of the process being modeled by the Markov chain, and are easily estimated from observations of the process.