Top

Published in:

2020 | OriginalPaper | Chapter

Optimizing Q-Learning with K-FAC Algorithm

Author : Roman Beltiukov

Published in: Analysis of Images, Social Networks and Texts

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In this work, we present intermediate results of the application of Kronecker-factored Approximate curvature (K-FAC) algorithm to Q-learning problem. Being more expensive to compute than plain stochastic gradient descent, K-FAC allows the agent to converge a bit faster in terms of epochs compared to Adam on simple reinforcement learning tasks and tend to be more stable and less strict to hyperparameters selection. Considering the latest results we show that DDQN with K-FAC learns more quickly than with other optimizers and improves constantly in contradiction to similar with Adam or RMSProp.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

next chapter Evolutionary Algorithms for Constructing an Ensemble of Decision Trees

https://github.com/maybe-hello-world/qfac.

Mnih, V., et al.: Playing atari with deep reinforcement learning, December 2013. http://arxiv.org/abs/1312.5602

van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning, September 2015. http://arxiv.org/abs/1509.06461

Bellemare, M.G., Dabney, W., Munos, R.: A distributional perspective on reinforcement learning, July 2017. http://arxiv.org/abs/1707.06887

Fortunato, M., et al.: Noisy networks for exploration, June 2017. http://arxiv.org/abs/1706.10295

Amari, S.I.: Natural gradient works efficiently in learning. Neural Comput. 10(2), 251–276 (1998). https://doi.org/10.1162/089976698300017746. http://www.mitpressjournls.org/10.1162/089976698300017746CrossRef

Knight, E., Lerner, O.: Natural gradient deep Q-learning, March 2018. http://arxiv.org/abs/1803.07482

Desjardins, G., Simonyan, K., Pascanu, R., Kavukcuoglu, K.: Natural neural networks, July 2015. http://arxiv.org/abs/1507.00210

Schulman, J., Levine, S., Moritz, P., Jordan, M.I., Abbeel, P.: Trust region policy optimization, February 2015. https://arxiv.org/abs/1502.05477

Wu, Y., Mansimov, E., Liao, S., Grosse, R., Ba, J.: Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation, August 2017. http://arxiv.org/abs/1708.05144

10.

Sutton, R.S., Barto, A.G., et al.: Introduction to Reinforcement Learning, vol. 135. MIT Press, Cambridge (1998)MATH

11.

Martens, J., Grosse, R.: Optimizing neural networks with Kronecker-factored approximate curvature, March 2015. http://arxiv.org/abs/1503.05671

12.

Martens, J., Tankasali, V., Duckworth, D., Johnson, M., Zhang, G., Koonce, B.: tensorflow\(\backslash \)kfac. https://github.com/tensorflow/kfac. Accessed 25 Mar 2019

13.

Dhariwal, P., et al.: OpenAI baselines (2017). https://github.com/openai/baselines

14.

Brockman, G., et al.: OpenAI Gym, June 2016. http://arxiv.org/abs/1606.01540

15.

Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization, December 2014. http://arxiv.org/abs/1412.6980

16.

Hinton, G.: RMSProp. https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf. Accessed 25 Mar 2019

17.

Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K.O., Clune, J.: Go-Explore: a new approach for hard-exploration problems, January 2019. http://arxiv.org/abs/1901.10995

18.

Resnick, C., Raileanu, R., Kapoor, S., Peysakhovich, A., Cho, K., Bruna, J.: Backplay: “Man muss immer umkehren”, July 2018. http://arxiv.org/abs/1807.06919

19.

Hessel, M., et al.: Rainbow: combining improvements in deep reinforcement learning, October 2017. http://arxiv.org/abs/1710.02298

Title: Optimizing Q-Learning with K-FAC Algorithm
Author: Roman Beltiukov
Publisher: Springer International Publishing
Book: Analysis of Images, Social Networks and Texts
Print ISBN: 978-3-030-39574-2

Electronic ISBN: 978-3-030-39575-9

Copyright Year: 2020
DOI: https://doi.org/10.1007/978-3-030-39575-9_1

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner