6. Markov Decision Processes for Screening and Treatment of Chronic Diseases

In recent years, Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs) have found important applications to medical decision making in the context of prevention, screening, and treatment of diseases. In this chapter, we provide a review of state-of-the-art models and methods that have been applied to chronic diseases. We provide a tutorial about how to formulate and solve these important problems emphasizing some of the challenges specific to chronic diseases such as diabetes, heart disease, and cancer. Then, we illustrate important considerations for model formulation and solution methods through two examples. The first example is an MDP model for optimal control of drug treatment decisions for managing the risk of heart disease and stroke in patients with type 2 diabetes. The second example is a POMDP model for optimal design of biomarker-based screening policies in the context of prostate cancer. We end the chapter with a discussion of the challenges of using MDPs and POMDPs for medical contexts and describe some important future directions for research.