Skip to main content
SpringerLink
Log in

A minimization problem of the risk probability in first passage semi-Markov decision processes with loss rates

  • Articles
  • Published:
Science China Mathematics Aims and scope Submit manuscript

Abstract

This paper is the first attempt to investigate the risk probability criterion in semi-Markov decision processes with loss rates. The goal is to find an optimal policy with the minimum risk probability that the total loss incurred during a first passage time to some target set exceeds a loss level. First, we establish the optimality equation via a successive approximation technique, and show that the value function is the unique solution to the optimality equation. Second, we give suitable conditions, under which we prove the existence of optimal policies and develop an algorithm for computing ε-optimal policies. Finally, we apply our main results to a business system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Boda K, Filar J A, Lin Y, et al. Stochastic target hitting time and the problem of early retirement. IEEE Trans Automat Control, 2004, 49: 409–419

    Article  MathSciNet  Google Scholar 

  2. Bouakiz M, Kebir Y. Target-level criterion in Markov decision processes. J Optim Theory Appl, 1995, 86: 1–15

    Article  MathSciNet  MATH  Google Scholar 

  3. Guo X P, Hernández-Lerma O. Continuous-Time Markov Decision Processes: Theory and Applications. Berlin: Springer-Verlag, 2009

    Book  Google Scholar 

  4. Guo X P, Hernández-Lerma O. New optimality conditions for average-payoff continuous-time Markov games in Polish spaces. Sci China Math, 2011, 54: 793–816

    Article  MathSciNet  MATH  Google Scholar 

  5. Hernández-Lerma O, Lasserre J B. Discrete-Time Markov Control Processes: Basic Optimality Criteria. New York: Springer-Verlag, 1996

    Book  Google Scholar 

  6. Huang Y H, Guo X P. Optimal risk probability for first passage models in semi-Markov decision processes. J Math Anal Appl, 2009, 359: 404–420

    Article  MathSciNet  MATH  Google Scholar 

  7. Huang Y H, Guo X P, Li Z F. Minimum risk probability for finite horizon semi-Markov decision processes. J Math Anal Appl, 2013, 402: 378–391

    Article  MathSciNet  MATH  Google Scholar 

  8. Huang Y H, Guo X P, Song X Y. Performance analysis for controlled semi-Markov systems with application to maintenance. J Optim Theory Appl, 2011, 150: 395–415

    Article  MathSciNet  MATH  Google Scholar 

  9. Limnios N, Oprisan G. Semi-Markov Processes and Reliability. Boston: Birkhäuser, 2001

    Book  MATH  Google Scholar 

  10. Love C E, Zhang Z G, Zitron M A, et al. A discrete semi-Markov decision model to determine the optimal repair/replacement policy under general repairs. European J Oper Res, 2000, 125: 398–409

    Article  MathSciNet  Google Scholar 

  11. Mamer J W. Successive approximations for finite horizon, semi-Markov decision processes with application to asset liquidation. Oper Res, 1986, 34: 638–644

    Article  MathSciNet  MATH  Google Scholar 

  12. Ohtsubo Y. Risk minimization in optimal stopping problem and applications. J Oper Res Soc Japan, 2003, 46: 342–352

    MathSciNet  MATH  Google Scholar 

  13. Ohtsubo Y. Minimizing risk models in stochastic shortest path problems. Math Methods Oper Res, 2003, 57: 79–88

    Article  MathSciNet  MATH  Google Scholar 

  14. Ohtsubo Y. Optimal threshold probability in undiscounted Markov decision processes with a target set. Appl Math Comput, 2004, 149: 519–532

    Article  MathSciNet  MATH  Google Scholar 

  15. Ohtsubo Y, Toyonaga K. Equivalence classes for optimizing risk models in Markov decision processes. Math Methods Oper Res, 2004, 60: 239–250

    Article  MathSciNet  MATH  Google Scholar 

  16. Puterman M L. Markov Decision Processes: Discrete Stochastic Dynamic Programming. New York: John Wiley & Sons, 1994

    Book  Google Scholar 

  17. Sakaguchi M, Ohtsubo Y. Optimal threshold probability and expectation in semi-Markov decision processes. Appl Math Comput, 2010, 216: 2947–2958

    Article  MathSciNet  MATH  Google Scholar 

  18. Singh S S, Tadić V B, Doucet A. A policy gradient method for semi-Markov decision processes with application to call admission control. European J Oper Res, 2007, 178: 808–818

    Article  MathSciNet  MATH  Google Scholar 

  19. Wei Q D, Guo X P. New average optimality conditions for semi-Markov decision processes in Borel spaces. J Optim Theory Appl, 2012, 153: 709–732

    Article  MathSciNet  MATH  Google Scholar 

  20. White D J. Minimising a threshold probability in discounted Markov decision processes. J Math Anal Appl, 1993, 173: 634–646

    Article  MathSciNet  MATH  Google Scholar 

  21. Wu R, Fang K. A risk model with delay in claim settlement. Acta Math Appl Sin Engl Ser, 1999, 15: 352–360

    Article  MathSciNet  MATH  Google Scholar 

  22. Wu Y H. Bounds for the ruin probability under a Markovian modulated risk model. Comm Statist Stoch Models, 1999, 15: 125–136

    Article  MATH  Google Scholar 

  23. Yu S X, Lin Y L, Yan P F. Optimization models for the first arrival target distribution function in discrete time. J Math Anal Appl, 1998, 225: 193–223

    Article  MathSciNet  MATH  Google Scholar 

  24. Zhang W Z, Guo X P. Nonzero-sum games for continuous-time Markov chains with unbounded transition and average payoff rates. Sci China Math, 2012, 55: 2405–2416

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mathematics and Computational Science, Sun Yat-Sen University, Guangzhou, 510275, China

    XiangXiang Huang, XiaoLong Zou & XianPing Guo

Authors
  1. XiangXiang Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. XiaoLong Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. XianPing Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to XianPing Guo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, X., Zou, X. & Guo, X. A minimization problem of the risk probability in first passage semi-Markov decision processes with loss rates. Sci. China Math. 58, 1923–1938 (2015). https://doi.org/10.1007/s11425-015-5029-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11425-015-5029-x

Keywords

MSC(2010)

Search

Navigation