Skip to main content
Erschienen in: Water Resources Management 9/2020

27.06.2020

Deep Reinforcement Learning for Cascaded Hydropower Reservoirs Considering Inflow Forecasts

verfasst von: Wei Xu, Xiaoli Zhang, Anbang Peng, Yue Liang

Erschienen in: Water Resources Management | Ausgabe 9/2020

Einloggen

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

This paper develops a deep reinforcement learning (DRL) framework for intelligence operation of cascaded hydropower reservoirs considering inflow forecasts, in which two key problems of large discrete action spaces and uncertainty of inflow forecasts are addressed. In this study, a DRL framework is first developed based on a newly defined knowledge sample form and a deep Q-network (DQN). Then, an aggregation-disaggregation model is used to reduce the multi-dimension spaces of state and action for cascaded reservoirs. Following, three DRL models are developed respectively to evaluate the performance of the newly defined decision value functions and modified decision action selection approach. In this paper, the DRL methodologies are tested on China’s Hun River cascade hydropower reservoirs system. The results show that the aggregation-disaggregation model can effectively reduce the dimensions of state and action, which also makes the model structure simpler and has higher learning efficiency. The Bayesian theory in the decision action selection approach is useful to address the uncertainty of inflow forecasts, which can improve the performance to reduce spillages during the wet season. The proposed DRL models outperform the comparison models (i.e., stochastic dynamic programming) in terms of annual hydropower generation and system reliability. This study suggests that the DRL has the potential to be implemented in practice to derive optimal operation strategies.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Literatur
Zurück zum Zitat Archibald TW, Marshall SE (2018) Review of mathematical programming applications in water resource management under uncertainty. Environ Model Assess 23(6):753–777CrossRef Archibald TW, Marshall SE (2018) Review of mathematical programming applications in water resource management under uncertainty. Environ Model Assess 23(6):753–777CrossRef
Zurück zum Zitat Celeste AB, Billib M (2009) Evaluation of stochastic reservoir operation optimization models. Adv Water Resour 32(9):1429–1443CrossRef Celeste AB, Billib M (2009) Evaluation of stochastic reservoir operation optimization models. Adv Water Resour 32(9):1429–1443CrossRef
Zurück zum Zitat Chen P, He Z, Chen C, Xu J (2018) Control strategy of speed servo systems based on deep reinforcement learning. Algorithms. 11(5):65CrossRef Chen P, He Z, Chen C, Xu J (2018) Control strategy of speed servo systems based on deep reinforcement learning. Algorithms. 11(5):65CrossRef
Zurück zum Zitat Doltsinis S, Ferreira P, Lohse N (2014) An MDP model-based reinforcement learning approach for production station ramp-up optimization: Q-learning analysis. IEEE Trans Syst Man Cybern Syst 44(9):1125–1138CrossRef Doltsinis S, Ferreira P, Lohse N (2014) An MDP model-based reinforcement learning approach for production station ramp-up optimization: Q-learning analysis. IEEE Trans Syst Man Cybern Syst 44(9):1125–1138CrossRef
Zurück zum Zitat Dressler OJ, Howes PD, Choo J, deMello AJ (2018) Reinforcement learning for dynamic microfluidic control. ACS Omega 3(8):10084–10091CrossRef Dressler OJ, Howes PD, Choo J, deMello AJ (2018) Reinforcement learning for dynamic microfluidic control. ACS Omega 3(8):10084–10091CrossRef
Zurück zum Zitat Dulac-Arnold G, Evans R, van Hasselt H, Sunehag P, Lillicrap T, Hunt J, Mann T, Weber T, Degris T, Coppin B (2015) Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679 Dulac-Arnold G, Evans R, van Hasselt H, Sunehag P, Lillicrap T, Hunt J, Mann T, Weber T, Degris T, Coppin B (2015) Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679
Zurück zum Zitat Gao Y, Chen J, Robertazzi T, Brown KA (2019) Reinforcement learning based schemes to manage client activities in large distributed control systems. Phys Rev Accel Beams 22(1):014601CrossRef Gao Y, Chen J, Robertazzi T, Brown KA (2019) Reinforcement learning based schemes to manage client activities in large distributed control systems. Phys Rev Accel Beams 22(1):014601CrossRef
Zurück zum Zitat Hashimoto T, Stedinger JR, Loucks DP (1982) Reliability, resiliency, and vulnerability criteria for water resource system performance evaluation. Water Resour Res 18(1):14–20CrossRef Hashimoto T, Stedinger JR, Loucks DP (1982) Reliability, resiliency, and vulnerability criteria for water resource system performance evaluation. Water Resour Res 18(1):14–20CrossRef
Zurück zum Zitat Kim S, Lim H (2018) Reinforcement learning based energy management algorithm for smart energy buildings. Energies. 11(8):2010CrossRef Kim S, Lim H (2018) Reinforcement learning based energy management algorithm for smart energy buildings. Energies. 11(8):2010CrossRef
Zurück zum Zitat Li M, Deng CH, Tan J, Yang W, Zheng L (2016) Research on small hydropower generation forecasting method based on improved BP neural network. 3rd international conference on materials engineering, manufacturing technology and control. Atlantis Press Li M, Deng CH, Tan J, Yang W, Zheng L (2016) Research on small hydropower generation forecasting method based on improved BP neural network. 3rd international conference on materials engineering, manufacturing technology and control. Atlantis Press
Zurück zum Zitat Li H, Cai R, Liu N, Lin X, Wang Y (2018) Deep reinforcement learning: algorithm, applications, and ultra-low-power implementation. Nano Commun Netw 16:81–90CrossRef Li H, Cai R, Liu N, Lin X, Wang Y (2018) Deep reinforcement learning: algorithm, applications, and ultra-low-power implementation. Nano Commun Netw 16:81–90CrossRef
Zurück zum Zitat Lu H, Hu B, Ma Z, Wen S (2014) Reinforcement learning optimization for energy-efficient cellular networks with coordinated multipoint communications. Math Probl Eng 2014:1–9 Lu H, Hu B, Ma Z, Wen S (2014) Reinforcement learning optimization for energy-efficient cellular networks with coordinated multipoint communications. Math Probl Eng 2014:1–9
Zurück zum Zitat Ming B, Liu P, Chang J, Wang Y, Huang Q (2017) Deriving operating rules of pumped water storage using multiobjective optimization: case study of the Han to Wei interbasin water transfer project, China. J Water Resour Plan Manage 143(10):05017012CrossRef Ming B, Liu P, Chang J, Wang Y, Huang Q (2017) Deriving operating rules of pumped water storage using multiobjective optimization: case study of the Han to Wei interbasin water transfer project, China. J Water Resour Plan Manage 143(10):05017012CrossRef
Zurück zum Zitat Mnih V, Kavukcuoglu K, Silver,D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 Mnih V, Kavukcuoglu K, Silver,D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602
Zurück zum Zitat Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature. 518(7540):529–533CrossRef Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature. 518(7540):529–533CrossRef
Zurück zum Zitat Mujumdar PP, Nirmala B (2007) A Bayesian stochastic optimization model for a multi-reservoir hydropower system. Water Resour Manag 21(9):1465–1485CrossRef Mujumdar PP, Nirmala B (2007) A Bayesian stochastic optimization model for a multi-reservoir hydropower system. Water Resour Manag 21(9):1465–1485CrossRef
Zurück zum Zitat Niroui F, Zhang K, Kashino Z, Nejat G (2019) Deep reinforcement learning robot for search and rescue applications: exploration in unknown cluttered environments. IEEE Robot Autom Lett 4(2):610–617CrossRef Niroui F, Zhang K, Kashino Z, Nejat G (2019) Deep reinforcement learning robot for search and rescue applications: exploration in unknown cluttered environments. IEEE Robot Autom Lett 4(2):610–617CrossRef
Zurück zum Zitat Peng A, Peng Y, Zhou H, Zhang C (2014) Multi-reservoir joint operating rule in inter-basin water transfer-supply project. Sci China-Technol Sci 58(1):123–137CrossRef Peng A, Peng Y, Zhou H, Zhang C (2014) Multi-reservoir joint operating rule in inter-basin water transfer-supply project. Sci China-Technol Sci 58(1):123–137CrossRef
Zurück zum Zitat Pineau J, Bellemare MG, Islam R, Henderson P, François-Lavet V (2018) An introduction to deep reinforcement learning. Found Trends Mach Learn 11(3–4):219–354 Pineau J, Bellemare MG, Islam R, Henderson P, François-Lavet V (2018) An introduction to deep reinforcement learning. Found Trends Mach Learn 11(3–4):219–354
Zurück zum Zitat Rodriguez-Ramos A, Sampedro C, Bavle H, de la Puente P, Campoy P (2018) A deep reinforcement learning strategy for UAV autonomous landing on a moving platform. J Intell Robot Syst 93(1–2):351–366 Rodriguez-Ramos A, Sampedro C, Bavle H, de la Puente P, Campoy P (2018) A deep reinforcement learning strategy for UAV autonomous landing on a moving platform. J Intell Robot Syst 93(1–2):351–366
Zurück zum Zitat Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nature. 529(7587):484–489CrossRef Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nature. 529(7587):484–489CrossRef
Zurück zum Zitat Tang G, Zhou H, Li N, Wang F, Wang Y, Jian D (2010) Value of medium-range precipitation forecasts in inflow prediction and hydropower optimization. Water Resour Manag 24(11):2721–2742CrossRef Tang G, Zhou H, Li N, Wang F, Wang Y, Jian D (2010) Value of medium-range precipitation forecasts in inflow prediction and hydropower optimization. Water Resour Manag 24(11):2721–2742CrossRef
Zurück zum Zitat Turgeon A (1980) Optimal operation of multireservoir power systems with stochastic inflows. Water Resour Res 16(2):275–283CrossRef Turgeon A (1980) Optimal operation of multireservoir power systems with stochastic inflows. Water Resour Res 16(2):275–283CrossRef
Zurück zum Zitat Xu W, Zhang C, Peng Y, Fu G, Zhou H (2014) A two stage Bayesian stochastic optimization model for cascaded hydropower systems considering varying uncertainty of flow forecasts. Water Resour Res 50(12):9267–9286CrossRef Xu W, Zhang C, Peng Y, Fu G, Zhou H (2014) A two stage Bayesian stochastic optimization model for cascaded hydropower systems considering varying uncertainty of flow forecasts. Water Resour Res 50(12):9267–9286CrossRef
Zurück zum Zitat Zhang X, Peng Y, Xu W, Wang B (2018) An optimal operation model for hydropower stations considering inflow forecasts with different Lead-times. Water Resour Manag 33(1):173–188CrossRef Zhang X, Peng Y, Xu W, Wang B (2018) An optimal operation model for hydropower stations considering inflow forecasts with different Lead-times. Water Resour Manag 33(1):173–188CrossRef
Zurück zum Zitat Zhao T, Zhao J, Liu P, Lei X (2015) Evaluating the marginal utility principle for long-term hydropower scheduling. Energy Conv Manag 106:213–223CrossRef Zhao T, Zhao J, Liu P, Lei X (2015) Evaluating the marginal utility principle for long-term hydropower scheduling. Energy Conv Manag 106:213–223CrossRef
Metadaten
Titel
Deep Reinforcement Learning for Cascaded Hydropower Reservoirs Considering Inflow Forecasts
verfasst von
Wei Xu
Xiaoli Zhang
Anbang Peng
Yue Liang
Publikationsdatum
27.06.2020
Verlag
Springer Netherlands
Erschienen in
Water Resources Management / Ausgabe 9/2020
Print ISSN: 0920-4741
Elektronische ISSN: 1573-1650
DOI
https://doi.org/10.1007/s11269-020-02600-w

Weitere Artikel der Ausgabe 9/2020

Water Resources Management 9/2020 Zur Ausgabe