1991 | OriginalPaper | Chapter
Existence and Uniqueness of Q-Functions
Author : William J. Anderson
Published in: Continuous-Time Markov Chains
Publisher: Springer New York
Included in: Professional Book Archive
Activate our intelligent search to find suitable subject content or patents.
Select sections of text to find matching patents with Artificial Intelligence. powered by
Select sections of text to find additional relevant content using AI-assisted search. powered by
Suppose that $$\left\{ {{X_n},n = 0,1,...} \right\}$$ is a discrete time Markov chain with discrete state space E and stationary transition probabilities. Then the reader is well aware that such a stochastic process is uniquely determined by the one-step transition matrix P whose i,jth component is $${p_{ij = }}\Pr \left\{ {{X_{n + 1}} = j\left| {{X_n} = i} \right.} \right\},$$, and an initial distribution vector p, whose ith component is $${p_i} = \Pr \left\{ {{X_0} = i} \right\}.$$. Every probability involving the random variables of this chain can be determined from the finite-dimensional distributions $$\Pr \left\{ {{X_{{n_1}}} = {i_1},{X_{{n_2}}} = {i_2},...,{X_{{n_k}}} = {i_k}} \right\}$$, and the latter can be expressed in the form $$\sum\limits_i p iPi{i_1}\left( {{n_1}} \right)P{i_1}{i_2}\left( {{n_2} - {n_1}} \right)...{P_{{i_{k - 1}}}}_{{i_k}}\left( {{n_k} - {n_{k - 1}}} \right),$$, where P ij (n) is the i,jth element of the n-step transition matrix Pn, which is the nth power of the one-step transition matrix P. Furthermore, the one-step transition probabilities P ij have very obvious meaning in terms of the process being modeled by the Markov chain, and are easily estimated from observations of the process.