Abstract
A numerical scheme for solving high-dimensional stochastic control problems on an infinite time horizon that appear relevant in the context of molecular dynamics is outlined. The scheme rests on the interpretation of the corresponding Hamilton–Jacobi–Bellman equation as a nonlinear eigenvalue problem that, using a logarithmic transformation, can be recast as a linear eigenvalue problem, for which the principal eigenvalue and its eigenfunction are sought. The latter can be computed efficiently by approximating the underlying stochastic process with a coarse-grained Markov state model for the dominant metastable sets. We illustrate our method with two numerical examples, one of which involves the task of maximizing the population of α-helices in an ensemble of small biomolecules (alanine dipeptide), and discuss the relation to the large deviation principle of Donsker and Varadhan.
Similar content being viewed by others
References
Fleming W.H., Soner H.M.: Controlled Markov Processes and Viscosity Solutions. Springer, New York (2006)
Kushner H.J., Dupuis P.G.: Numerical Methods for Stochastic Control Problems in Continuous Time. Springer, New York (2001)
Barles G., Souganidis P.: Convergence of approximation schemes for fully nonlinear equations. Asymptot. Anal. 4, 271–283 (1991)
Barles G., Jakobsen E.: Error bounds for monotone approximation schemes for parabolic Hamilton–Jacobi–Bellman equations. Math. Comput. 76, 1861–1893 (2007)
Carlini E., Falcone M., Ferretti R.: An efficient algorithm for Hamilton Jacobi equations in high dimension. Comput. Vis. Sci. 7, 15–29 (2004)
Kappen H.J.: Path integrals and symmetry breaking for optimal control theory. J. Stat. Mech. Theor. Exp. 2005(11), P11011 (2005)
Isralewitz B., Gao M., Schulten K.: Steered molecular dynamics and mechanical functions of proteins. Curr. Opin. Struct. Biol. 11, 224–230 (2001)
Schlitter J., Engels M., Krüger P., Jacoby E., Wollmer A.: Targeted molecular dynamics. Mol. Simul. 10, 291–309 (1993)
Rabitz H., de Vivie-Riedle R., Motzkus M., Kompa K.: Whither the future of controlling quantum phenomena?. Science 288(5467), 824–828 (2000)
Kühn O., Wöste L.: Analysis and control of ultrafast photoinduced reactions. In: Kühn, O., Wöste, L.(eds) Series in Chemical Physics, vol. 87, Springer, New York (2007)
Engel A., Müller D.J.: Observing single biomolecules at work with the atomic force microscope. Nat. Struct. Mol. Biol. 7, 715–718 (2000)
Fleming W.H., McEneaney W.M.: Risk-sensitive control on an infinite time horizon. SIAM J. Control Opt. 33, 1881–1915 (1995)
Fleming W.H., Sheu S.J.: Asymptotics for the principal eigenvalue and eigenfunction of a nearly first-order operator with large potential. Ann. Probab. 25, 1953–1994 (1997)
Freidlin M., Wentzell A.D.: Random Perturbations of Dynamical Systems. Springer, New York (1998)
Jäger M., Zhang Y., Bieschke J., Nguyen H., Dendle M., Bowman M.E., Noel J.P., Gruebele M., Kelly J.W.: Structure-function-folding relationship in a WW domain. Proc. Natl. Acad. Sci. USA. 103, 10648–10653 (2006)
Kobitski A.Y., Nierth A., Helm M., Jäschke A., Nienhaus G.U.: Mg2+ dependent folding of a Diels-Alderase ribozyme probed by single-molecule FRET analysis. Nucleic Acids Res. 35, 2047–2059 (2007)
Fischer S., Windshuegel B., Horak D., Holmes K.C., Smith J.C.: Structural mechanism of the recovery stroke in the myosin molecular motor. Proc. Natl. Acad. Sci. USA. 102, 6873–6878 (2005)
Noé F., Krachtus D., Smith J.C., Fischer S.: Transition networks for the comprehensive characterization of complex conformational change in proteins. J. Chem. Theor. Comp. 2, 840–857 (2006)
Ostermann A., Waschipky R., Parak F.G., Nienhaus U.G.: Ligand binding and conformational motions in myoglobin. Nature 404, 205–208 (2000)
Schaeffer D.D., Fersht A., Daggett V.: Combining experiment and simulation in protein folding: closing the gap for small model systems. Curr. Opin. Struct. Biol. 18(1), 4–9 (2008)
Noé F., Schütte C., Vanden-Eijnden E., Reich L., Weikl T.R.: Constructing the full ensemble of folding pathways from short off-equilibrium simulations. Proc. Natl. Acad. Sci. USA 106, 19011–19016 (2009)
van Gunsteren W., Dolenc J., Mark A.: Molecular simulation as an aid to experimentalists. Curr. Opin. Struct. Biol. 18(2), 149–153 (2008)
Schütte, C.: Conformational dynamics: modelling, theory, algorithm, and applications to biomolecules. Habilitation thesis, Fachbereich Mathematik und Informatik, FU Berlin (1999)
Deuflhard P., Huisinga W., Fischer A., Schütte C.: Identification of almost invariant aggregates in reversible nearly uncoupled Markov chains. Linear Algebra Appl. 315, 39–59 (2000)
Schütte C., Fischer A., Huisinga W., Deuflhard P.: A direct approach to conformational dynamics based on hybrid Monte Carlo. J. Comp. Phys. (Special issue on Comput. Biophys.) 151, 146–168 (1999)
Schütte C., Huisinga W.: Biomolecular conformations can be identified as metastable sets of molecular dynamics. In: Le Bris, C.(ed) Handbook of Numerical Analysis, pp. 699–744. Elsevier, New York (2003)
Bovier A., Eckhoff M., Gayrard V., Klein M.: Metastability and low lying spectra in reversible markov chains. Commun. Math. Phys. 228, 219–255 (2002)
Noé F., Horenko I., Schütte C., Smith J.: Hierarchical analysis of conformational dynamics in biomolecules: transition networks of metastable states. J. Chem. Phys. 126, 155102 (2007)
Chodera J., Singhal N., Pande V.S., Dill K., Swope W.: Automatic discovery of metastable states for the construction of Markov models of macromolecular conformational dynamics. J. Chem. Phys. 126, 155101 (2007)
Buchete N.V., Hummer G.: Coarse master equations for peptide folding dynamics. J. Phys. Chem. B. 112, 6057–6069 (2008)
Pan A.C., Roux B.: Building Markov state models along pathways to determine free energies and rates of transitions. J. Chem. Phys. 129, 064107 (2008)
Sarich M., Noé F., Schütte C.: On the approximation quality of Markov state models. Multiscale Model. Simul. 8(4), 1154–1177 (2010)
Voter A.: Introduction to the kinetic Monte Carlo method. In: Sickafus, K., Kotomin, E., Uberuaga, B.(eds) Radiation Effects in Solids, Springer, NATO Publishing Unit, Dordrecht (2005)
Huisinga, W.: Metastability of markovian systems a transfer operator based approach in application to molecular dynamics. Ph.D. thesis, Fachbereich Mathematik und Informatik, FU Berlin (2001)
Schütte C., Noe F., Lu J., Sarich M., Vanden-Eijnden E.: Markov state models based on milestoning. J. Chem. Phys. 134, 204105–204120 (2011)
Borkar V.S., Ghosh M.K.: Ergodic control of multidimensional diffusions, II: adaptive control. Appl. Math. Opt. 21, 191–220 (1990)
Bensoussan A., Nagai H.: An ergodic control problem arising from the principal eigenvalue of an elliptic operator. J. Math. Soc. Jpn. 43, 49–65 (1990)
Huisinga W., Meyn S.P., Schütte C.: Phase transitions & metastability in Markovian and molecular systems. Ann. Appl. Probab. 14, 419–458 (2004)
James M.: Asymptotic analysis of nonlinear stochastic risk-sensitive control and differential games. Math. Control Signals Syst. 5, 401–417 (1992)
Holland C.J.: A minimum principle for the principal eigenvalue for second-order linear elliptic equations with natural boundary conditions. Commun. Pure Appl. Math. 31, 509–519 (1978)
Whittle P.: Risk-sensitivity, large deviations and stochastic control. Eur. J. Oper. Res. 73, 295–303 (1994)
Donsker M.D., Varadhan S.S.: Asymptotic evaluation of certain Markov process expectations for large time, i. Commun. Pure Appl. Math. 28, 1–47 (1975)
Ellis R.S.: Entropy, Large Deviations and Statistical Mechanics. Springer, Berlin (1985)
Kurchan J.: Fluctuation theorem for stochastic dynamics. J. Phys. A. Math. Gen. 31, 3719–3729 (1998)
Lebowitz J.L., Spohn H.: A Gallavotti–Cohen-type symmetry in the large deviation functional for stochastic dynamics. J. Stat. Phys. 95, 333–365 (1999)
Pham H.: Continuous-time Stochastic Control and Optimization with Financial Applications. Springer, Berlin (2009)
Robin M.: Long-term average cost control problems for continuous time Markov processes: a survey. Acta Appl. Math. 1, 281–299 (1983)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Schütte, C., Winkelmann, S. & Hartmann, C. Optimal control of molecular dynamics using Markov state models. Math. Program. 134, 259–282 (2012). https://doi.org/10.1007/s10107-012-0547-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-012-0547-6