4. Continuous-Time Markov Chains

The c-variable approach is extended in this chapter by adding a new linear constraint for continuous time. This method’s benefit is that it transforms the continuous-time Markov decision process problem into a discrete-time Markov decision process, where the linear constraints make the problem computationally tractable. Chemical reaction networks, where the concentration dynamics is described as a continuous-time Markov chain, serve as an example of the method’s use. Using a state-discrete continuous-time Markov decision process, we provide a mathematical optimization method for resolving chemical processes. The first application is a single reversible reaction that produces the amidogen radical, and we were able to determine the ideal temperature that reduces a linear functional of interest. The second is a chemical reaction network that involves the proton transfer, hydration, and tautomeric reaction of anthocyanin pigments, in this case we found an optimal strategy over a set of values of pH that minimizes the corresponding linear functional.