Skip to main content
Top
Published in:
Cover of the book

2019 | OriginalPaper | Chapter

Learning to Run Faster in a Humanoid Robot Soccer Environment Through Reinforcement Learning

Authors : Miguel Abreu, Luis Paulo Reis, Nuno Lau

Published in: RoboCup 2019: Robot World Cup XXIII

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Reinforcement learning techniques bring a new perspective to enduring problems. Developing skills from scratch is not only appealing due to the artificial creation of knowledge. It can also replace years of work and refinement in a matter of hours. From all the developed skills in the RoboCup 3D Soccer Simulation League, running is still considerably relevant to determine the winner of any match. However, current approaches do not make full use of the robotic soccer agents’ potential. To narrow this gap, we propose a way of leveraging the Proximal Policy Optimization using the information provided by the simulator for official RoboCup matches. To do this, our algorithm uses a mix of raw, computed and internally generated data. The final result is a sprinting and a stopping behavior that work in tandem to bring the agent from point a to point b in a very short time. The sprinting speed stabilizes at around 2.5 m/s, which is a great improvement over current solutions. Both the sprinting and stopping behaviors are remarkably stable.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Abreu, M., Lau, N., Sousa, A., Reis, L.P.: Learning low level skills from scratch for humanoid robot soccer using deep reinforcement learning. In: 19th IEEE International Conference on Autonomous Robot Systems and Competitions (IEEE ICARSC 2019), Gondomar, Porto, Portugal, 24–26 April 2019 Abreu, M., Lau, N., Sousa, A., Reis, L.P.: Learning low level skills from scratch for humanoid robot soccer using deep reinforcement learning. In: 19th IEEE International Conference on Autonomous Robot Systems and Competitions (IEEE ICARSC 2019), Gondomar, Porto, Portugal, 24–26 April 2019
2.
go back to reference Noda, I., Suzuki, S.J., Matsubara, H., Asada, M., Kitano, H.: RoboCup-97: the first robot world cup soccer games and conferences. AI Mag. 19(3), 49 (1998) Noda, I., Suzuki, S.J., Matsubara, H., Asada, M., Kitano, H.: RoboCup-97: the first robot world cup soccer games and conferences. AI Mag. 19(3), 49 (1998)
7.
go back to reference MacAlpine, P., Depinet, M., Liang, J., Stone, P.: UT Austin Villa: RoboCup 2014 3D simulation league competition and technical challenge champions. In: Bianchi, R.A.C., Akin, H.L., Ramamoorthy, S., Sugiura, K. (eds.) RoboCup 2014. LNCS (LNAI), vol. 8992, pp. 33–46. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18615-3_3CrossRef MacAlpine, P., Depinet, M., Liang, J., Stone, P.: UT Austin Villa: RoboCup 2014 3D simulation league competition and technical challenge champions. In: Bianchi, R.A.C., Akin, H.L., Ramamoorthy, S., Sugiura, K. (eds.) RoboCup 2014. LNCS (LNAI), vol. 8992, pp. 33–46. Springer, Cham (2015). https://​doi.​org/​10.​1007/​978-3-319-18615-3_​3CrossRef
8.
go back to reference Snafii, N., Abdolmaleki, A., Lau, N., Reis, L.P.: Development of an omnidirectional walk engine for soccer humanoid robots. Int. J. Adv. Rob. Syst. 12(12), 193 (2015) Snafii, N., Abdolmaleki, A., Lau, N., Reis, L.P.: Development of an omnidirectional walk engine for soccer humanoid robots. Int. J. Adv. Rob. Syst. 12(12), 193 (2015)
9.
go back to reference Moradi, K., Fathian, M., Ghidary, S.S.: Omnidirectional walking using central pattern generator. Int. J. Mach. Learn. Cybernet. 7(6), 1023–1033 (2016)CrossRef Moradi, K., Fathian, M., Ghidary, S.S.: Omnidirectional walking using central pattern generator. Int. J. Mach. Learn. Cybernet. 7(6), 1023–1033 (2016)CrossRef
10.
go back to reference Abdolmaleki, A., Lau, N., Reis, L.P., Peters, J., Neumann, G.: Contextual policy search for linear and nonlinear generalization of a humanoid walking controller. J. Intell. Rob. Syst. 83(3), 393–408 (2016)CrossRef Abdolmaleki, A., Lau, N., Reis, L.P., Peters, J., Neumann, G.: Contextual policy search for linear and nonlinear generalization of a humanoid walking controller. J. Intell. Rob. Syst. 83(3), 393–408 (2016)CrossRef
11.
go back to reference Abdolmaleki, A., Lau, N., Reis, L.P., Peters, J., Neumann, G.: Contextual policy search for generalizing a parameterized biped walking controller. In: 2015 IEEE International Conference on Autonomous Robot Systems and Competitions, pp. 17–22. IEEE (2015) Abdolmaleki, A., Lau, N., Reis, L.P., Peters, J., Neumann, G.: Contextual policy search for generalizing a parameterized biped walking controller. In: 2015 IEEE International Conference on Autonomous Robot Systems and Competitions, pp. 17–22. IEEE (2015)
12.
go back to reference Shafii, N., Lau, N., Reis, L.P.: Learning to walk fast: optimized hip height movement for simulated and real humanoid robots. J. Intell. Rob. Syst. 80(3), 555–571 (2015)CrossRef Shafii, N., Lau, N., Reis, L.P.: Learning to walk fast: optimized hip height movement for simulated and real humanoid robots. J. Intell. Rob. Syst. 80(3), 555–571 (2015)CrossRef
15.
16.
go back to reference Sugiura, N.: Further analysis of the data by Akaike’s information criterion and the finite corrections. Commun. Stat. Theory Methods 7(1), 13–26 (1978)CrossRef Sugiura, N.: Further analysis of the data by Akaike’s information criterion and the finite corrections. Commun. Stat. Theory Methods 7(1), 13–26 (1978)CrossRef
17.
go back to reference Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. CoRR, vol. abs/1707.06347 (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. CoRR, vol. abs/1707.06347 (2017)
20.
go back to reference Adelaar, R.S.: The practical biomechanics of running. Am. J. Sports Med. 14(6), 497–500 (1986)CrossRef Adelaar, R.S.: The practical biomechanics of running. Am. J. Sports Med. 14(6), 497–500 (1986)CrossRef
21.
go back to reference Novacheck, T.F.: The biomechanics of running. Gait Posture 7(1), 77–95 (1998)CrossRef Novacheck, T.F.: The biomechanics of running. Gait Posture 7(1), 77–95 (1998)CrossRef
Metadata
Title
Learning to Run Faster in a Humanoid Robot Soccer Environment Through Reinforcement Learning
Authors
Miguel Abreu
Luis Paulo Reis
Nuno Lau
Copyright Year
2019
DOI
https://doi.org/10.1007/978-3-030-35699-6_1

Premium Partner