Skip to main content

2021 | OriginalPaper | Buchkapitel

Convergence Dynamics of Generative Adversarial Networks: The Dual Metric Flows

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Fitting neural networks often resorts to stochastic (or similar) gradient descent which is a noise-tolerant (and efficient) resolution of a gradient descent dynamics. It outputs a sequence of networks parameters, which sequence evolves during the training steps. The gradient descent is the limit, when the learning rate is small and the batch size is infinite, of this set of increasingly optimal network parameters obtained during training. In this contribution, we investigate instead the convergence in the Generative Adversarial Networks used in machine learning. We study the limit of small learning rate, and show that, similar to single network training, the GAN learning dynamics tend, for vanishing learning rate to some limit dynamics. This leads us to consider evolution equations in metric spaces (which is the natural framework for evolving probability laws) that we call dual flows. We give formal definitions of solutions and prove the convergence. The theory is then applied to specific instances of GANs and we discuss how this insight helps understand and mitigate the mode collapse.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Anhänge
Nur mit Berechtigung zugänglich
Fußnoten
1
See Lemma 1 in appendix for information on the relationship between explicit and implicit numerical schemes.
 
Literatur
2.
Zurück zum Zitat Ambrosio, L., Gigli, N.: A user’s guide to optimal transport. In: Piccoli, B., Rascle, M. (eds.) Modelling and Optimisation of Flows on Networks: Cetraro, Italy 2009, pp. 1–155. Springer, Heidelberg (2013) Ambrosio, L., Gigli, N.: A user’s guide to optimal transport. In: Piccoli, B., Rascle, M. (eds.) Modelling and Optimisation of Flows on Networks: Cetraro, Italy 2009, pp. 1–155. Springer, Heidelberg (2013)
3.
Zurück zum Zitat Ambrosio, L., Gigli, N., Savaré, G.: Gradient Flows in Metric Spaces and in the Space of Probability Measures, 2nd edn. Birkhäuser, Basel (2008) Ambrosio, L., Gigli, N., Savaré, G.: Gradient Flows in Metric Spaces and in the Space of Probability Measures, 2nd edn. Birkhäuser, Basel (2008)
9.
Zurück zum Zitat Dukler, Y., Li, W., Lin, A.T., Montúfar, G.: Wasserstein of Wasserstein loss for learning generative models. In: Chaudhuri, K. (ed.) Proceedings of the 36th International Conference on Machine Learning, 9–15 June 2019, Long Beach, California, USA, Proceedings of machine learning research, vol. 97, pp. 1716–1725. PMLR, Long Beach, California (2019) Dukler, Y., Li, W., Lin, A.T., Montúfar, G.: Wasserstein of Wasserstein loss for learning generative models. In: Chaudhuri, K. (ed.) Proceedings of the 36th International Conference on Machine Learning, 9–15 June 2019, Long Beach, California, USA, Proceedings of machine learning research, vol. 97, pp. 1716–1725. PMLR, Long Beach, California (2019)
11.
Zurück zum Zitat Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017) Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)
17.
18.
Zurück zum Zitat Kolouri, S., Pope, P.E., Martin, C.E., Rohde, G.K.: Sliced-Wasserstein autoencoder: an embarrassingly simple generative model. arXiv preprint arXiv:1804.01947 (2018) Kolouri, S., Pope, P.E., Martin, C.E., Rohde, G.K.: Sliced-Wasserstein autoencoder: an embarrassingly simple generative model. arXiv preprint arXiv:​1804.​01947 (2018)
19.
Zurück zum Zitat Kopfer, E.: Gradient flow for the Boltzmann entropy and Cheeger’s energy on time-dependent metric measure spaces. ArXiv e-prints, November 2016 Kopfer, E.: Gradient flow for the Boltzmann entropy and Cheeger’s energy on time-dependent metric measure spaces. ArXiv e-prints, November 2016
24.
Zurück zum Zitat Rossi, R., Mielke, A., Savaré, G.: A metric approach to a class of doubly nonlinear evolution equations and applications. Ann. Sc. Norm. Super. Pisa Cl. Sci. (5) 7(1), 97–169 (2008) Rossi, R., Mielke, A., Savaré, G.: A metric approach to a class of doubly nonlinear evolution equations and applications. Ann. Sc. Norm. Super. Pisa Cl. Sci. (5) 7(1), 97–169 (2008)
29.
Zurück zum Zitat Wu, J., Huang, Z., Acharya, D., Li, W., Thoma, J., Paudel, D.P., Gool, L.V.: Sliced Wasserstein generative models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3713–3722 (2019) Wu, J., Huang, Z., Acharya, D., Li, W., Thoma, J., Paudel, D.P., Gool, L.V.: Sliced Wasserstein generative models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3713–3722 (2019)
30.
Zurück zum Zitat Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2223–2232 (2017) Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2223–2232 (2017)
Metadaten
Titel
Convergence Dynamics of Generative Adversarial Networks: The Dual Metric Flows
verfasst von
Gabriel Turinici
Copyright-Jahr
2021
DOI
https://doi.org/10.1007/978-3-030-68763-2_47