Skip to main content

Über dieses Buch

This new edition of Markov Chains: Models, Algorithms and Applications has been completely reformatted as a text, complete with end-of-chapter exercises, a new focus on management science, new applications of the models, and new examples with applications in financial risk management and modeling of financial data.

This book consists of eight chapters. Chapter 1 gives a brief introduction to the classical theory on both discrete and continuous time Markov chains. The relationship between Markov chains of finite states and matrix theory will also be highlighted. Some classical iterative methods for solving linear systems will be introduced for finding the stationary distribution of a Markov chain. The chapter then covers the basic theories and algorithms for hidden Markov models (HMMs) and Markov decision processes (MDPs).

Chapter 2 discusses the applications of continuous time Markov chains to model queueing systems and discrete time Markov chain for computing the PageRank, the ranking of websites on the Internet. Chapter 3 studies Markovian models for manufacturing and re-manufacturing systems and presents closed form solutions and fast numerical algorithms for solving the captured systems. In Chapter 4, the authors present a simple hidden Markov model (HMM) with fast numerical algorithms for estimating the model parameters. An application of the HMM for customer classification is also presented.

Chapter 5 discusses Markov decision processes for customer lifetime values. Customer Lifetime Values (CLV) is an important concept and quantity in marketing management. The authors present an approach based on Markov decision processes for the calculation of CLV using real data.

Chapter 6 considers higher-order Markov chain models, particularly a class of parsimonious higher-order Markov chain models. Efficient estimation methods for model parameters based on linear programming are presented. Contemporary research results on applications to demand predictions, inventory control and financial risk measurement are also presented. In Chapter 7, a class of parsimonious multivariate Markov models is introduced. Again, efficient estimation methods based on linear programming are presented. Applications to demand predictions, inventory control policy and modeling credit ratings data are discussed. Finally, Chapter 8 re-visits hidden Markov models, and the authors present a new class of hidden Markov models with efficient algorithms for estimating the model parameters. Applications to modeling interest rates, credit ratings and default data are discussed.

This book is aimed at senior undergraduate students, postgraduate students, professionals, practitioners, and researchers in applied mathematics, computational science, operational research, management science and finance, who are interested in the formulation and computation of queueing networks, Markov chain models and related topics. Readers are expected to have some basic knowledge of probability theory, Markov processes and matrix theory.



Chapter 1. Introduction

Markov chains are named after Prof. Andrei A. Markov (1856–1922). He was born on June 14, 1856 in Ryazan, Russia and died on July 20, 1922 in St. Petersburg, Russia. Markov enrolled at the University of St. Petersburg, where he earned a master’s degree and a doctorate degree. He was a professor at St. Petersburg and also a member of the Russian Academy of Sciences.

Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu

Chapter 2. Queueing Systems and the Web

In this chapter, we first discuss some more Markovian queueing systems. The queueing system is a classical application of continuous Markov chains. We then present an important numerical algorithm based on the computation of Markov chains for ranking webpages. This is a modern application of Markov chains though the numerical methods used are classical.

Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu

Chapter 3. Manufacturing and Re-manufacturing Systems

In this chapter, we consider the application of the Markovian queueing systems discussed in Chap. 2 in modeling manufacturing systems and re-manufacturing systems. We adopt

Hedging Point Production

(HPP) policy as a production control policy. We note that in a queueing system, there are servers, customers, and waiting spaces. To model a


manufacturing system by a queueing system, one may regard a server as a machine. The customers can be regarded as the inventory of product or the jobs to be processed respectively; see for instance Buzacott and Shanthikumar [34]. In a


system, a certain amount of inventory (called the hedging point) is kept to cope with the fluctuation of demand and therefore production control is necessary. The system will stop production when this level of inventory is attained.

Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu

Chapter 4. A Hidden Markov Model for Customer Classification

In this chapter, a new simple Hidden Markov Model (HMM) is proposed. The process of the proposed HMM can be explained by the following example.

Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu

Chapter 5. Markov Decision Processes for Customer Lifetime Value

In this chapter a stochastic dynamic programming model with a Markov chain is proposed to capture customer behavior. The advantage of using Markov chains is that the model can take into account the customers switching between the company and its competitors. Therefore customer relationships can be described in a probabilistic way, see for instance Pfeifer and Carraway [170]. Stochastic dynamic programming is then applied to solve the optimal allocation of the promotion budget for maximizing the Customer Lifetime Value (CLV). The proposed model is then applied to practical data in a computer services company.

Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu

Chapter 6. Higher-Order Markov Chains

Data sequences or time series occur frequently in many real world applications. One of the most important steps in analyzing a data sequence (or time series) is the selection of an appropriate mathematical model for the data. This is because it helps in predictions, hypothesis testing and rule discovery.

Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu

Chapter 7. Multivariate Markov Chains

By making use of the transition probability matrix in, a categorical data sequence of


states can be modeled by an


-state Markov chain model. In this chapter, we extend this idea to model multiple categorical data sequences. One would expect categorical data sequences generated by similar sources or the same source to be correlated to each other. Therefore, by exploring these relationships, one can develop better models for the categorical data sequences and hence better prediction rules.

Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu

Chapter 8. Hidden Markov Chains

Hidden Markov models (HMMs) have been applied to many real-world applications. Usually HMMs only deal with the first-order transition probability distribution among the hidden states, see for instance Sect.1.4. Moreover, the observable states are affected by the hidden states but not vice versa. In this chapter, we study both higher-order hidden Markov models and interactive HMMs in which the hidden states are directly affected by the observed states. We will also develop estimation methods for the model parameters in both cases.

Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu


Weitere Informationen

Premium Partner

Stellmach & BröckersBBL | Bernsau BrockdorffMaturus Finance GmbHPlutahww hermann wienberg wilhelm

BranchenIndex Online

Die B2B-Firmensuche für Industrie und Wirtschaft: Kostenfrei in Firmenprofilen nach Lieferanten, Herstellern, Dienstleistern und Händlern recherchieren.



Künstliche Intelligenz und der Faktor Arbeit - Implikationen für Unternehmen und Wirtschaftspolitik

Künstliche Intelligenz und ihre Auswirkung auf den Faktor Arbeit ist zum Modethema in der wirtschaftswissenschaftlichen Politikberatung avanciert. Studien, die alarmistisch die baldige Verdrängung eines Großteils konventioneller Jobprofile beschwören, leiden jedoch unter fragwürdiger Datenqualität und Methodik. Die Unternehmensperspektive zeigt, dass der Wandel der Arbeitswelt durch künstliche Intelligenz weitaus langsamer und weniger disruptiv ablaufen wird. Jetzt gratis downloaden!