Skip to main content

2017 | OriginalPaper | Buchkapitel

Sparsity of Shallow Networks Representing Finite Mappings

verfasst von : Věra Kůrková

Erschienen in: Engineering Applications of Neural Networks

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Limitations of capabilities of shallow networks to represent sparsely real-valued functions on finite domains is investigated. Influence of sizes of function domains and of sizes dictionaries of computational units on sparsity of networks computing finite mappings is explored. It is shown that when dictionary is not sufficiently large with respect to the size of the finite domain, then almost any uniformly randomly chosen function on the domain either cannot be sparsely represented or its computation is unstable.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Ito, Y.: Finite mapping by neural networks and truth functions. Math. Sci. 17, 69–77 (1992)MathSciNetMATH Ito, Y.: Finite mapping by neural networks and truth functions. Math. Sci. 17, 69–77 (1992)MathSciNetMATH
3.
Zurück zum Zitat Fine, T.L.: Feedforward Neural Network Methodology. Springer, Heidelberg (1999)MATH Fine, T.L.: Feedforward Neural Network Methodology. Springer, Heidelberg (1999)MATH
4.
Zurück zum Zitat Bengio, Y., LeCun, Y.: Scaling learning algorithms towards AI. In: Bottou, L., Chapelle, O., DeCoste, D., Weston, J. (eds.) Large-Scale Kernel Machines. MIT Press (2007) Bengio, Y., LeCun, Y.: Scaling learning algorithms towards AI. In: Bottou, L., Chapelle, O., DeCoste, D., Weston, J. (eds.) Large-Scale Kernel Machines. MIT Press (2007)
5.
Zurück zum Zitat Ba, L.J., Caruana, R.: Do deep networks really need to be deep? In: Ghahrani, Z., et al. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 1–9 (2014) Ba, L.J., Caruana, R.: Do deep networks really need to be deep? In: Ghahrani, Z., et al. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 1–9 (2014)
6.
Zurück zum Zitat Kainen, P.C., Kůrková, V., Sanguineti, M.: Dependence of computational models on input dimension: tractability of approximation and optimization tasks. IEEE Trans. Inf. Theory 58, 1203–1214 (2012)MathSciNetCrossRefMATH Kainen, P.C., Kůrková, V., Sanguineti, M.: Dependence of computational models on input dimension: tractability of approximation and optimization tasks. IEEE Trans. Inf. Theory 58, 1203–1214 (2012)MathSciNetCrossRefMATH
7.
Zurück zum Zitat Maiorov, V.E., Pinkus, A.: Lower bounds for approximation by MLP neural networks. Neurocomputing 25, 81–91 (1999)CrossRefMATH Maiorov, V.E., Pinkus, A.: Lower bounds for approximation by MLP neural networks. Neurocomputing 25, 81–91 (1999)CrossRefMATH
8.
Zurück zum Zitat Maiorov, V.E., Meir, R.: On the near optimality of the stochastic approximation of smooth functions by neural networks. Adv. Comput. Math. 13, 79–103 (2000)MathSciNetCrossRefMATH Maiorov, V.E., Meir, R.: On the near optimality of the stochastic approximation of smooth functions by neural networks. Adv. Comput. Math. 13, 79–103 (2000)MathSciNetCrossRefMATH
9.
Zurück zum Zitat Bengio, Y., Delalleau, O., Roux, N.L.: The curse of highly variable functions for local kernel machines. In: Advances in Neural Information Processing Systems, vol. 18, pp. 107–114. MIT Press (2006) Bengio, Y., Delalleau, O., Roux, N.L.: The curse of highly variable functions for local kernel machines. In: Advances in Neural Information Processing Systems, vol. 18, pp. 107–114. MIT Press (2006)
10.
Zurück zum Zitat Bianchini, M., Scarselli, F.: On the complexity of neural network classifiers: a comparison between shallow and deep architectures. IEEE Trans. Neural Netw. Learn. Syst. 25, 1553–1565 (2014)CrossRef Bianchini, M., Scarselli, F.: On the complexity of neural network classifiers: a comparison between shallow and deep architectures. IEEE Trans. Neural Netw. Learn. Syst. 25, 1553–1565 (2014)CrossRef
11.
Zurück zum Zitat Kůrková, V., Sanguineti, M.: Model complexities of shallow networks representing highly varying functions. Neurocomputing 171, 598–604 (2016)CrossRef Kůrková, V., Sanguineti, M.: Model complexities of shallow networks representing highly varying functions. Neurocomputing 171, 598–604 (2016)CrossRef
12.
Zurück zum Zitat Kůrková, V.: Lower bounds on complexity of shallow perceptron networks. In: Jayne, C., Iliadis, L. (eds.) EANN 2016. CCIS, vol. 629, pp. 283–294. Springer, Heidelberg (2016)CrossRef Kůrková, V.: Lower bounds on complexity of shallow perceptron networks. In: Jayne, C., Iliadis, L. (eds.) EANN 2016. CCIS, vol. 629, pp. 283–294. Springer, Heidelberg (2016)CrossRef
14.
Zurück zum Zitat Kůrková, V., Sanguineti, M.: Approximate minimization of the regularized expected error over kernel models. Math. Oper. Res. 33, 747–756 (2008)MathSciNetCrossRefMATH Kůrková, V., Sanguineti, M.: Approximate minimization of the regularized expected error over kernel models. Math. Oper. Res. 33, 747–756 (2008)MathSciNetCrossRefMATH
15.
Zurück zum Zitat Barron, A.R.: Neural net approximation. In: Narendra, K.S. (ed.) Proceedings of the 7th Yale Workshop on Adaptive and Learning Systems, pp. 69–72. Yale University Press (1992) Barron, A.R.: Neural net approximation. In: Narendra, K.S. (ed.) Proceedings of the 7th Yale Workshop on Adaptive and Learning Systems, pp. 69–72. Yale University Press (1992)
16.
Zurück zum Zitat Kůrková, V.: Dimension-independent rates of approximation by neural networks. In: Warwick, K., Kárný, M. (eds.) Computer-Intensive Methods in Control and Signal Processing, The Curse of Dimensionality, pp. 261–270. Birkhäuser, Boston (1997) Kůrková, V.: Dimension-independent rates of approximation by neural networks. In: Warwick, K., Kárný, M. (eds.) Computer-Intensive Methods in Control and Signal Processing, The Curse of Dimensionality, pp. 261–270. Birkhäuser, Boston (1997)
17.
Zurück zum Zitat Barron, A.R.: Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inf. Theory 39, 930–945 (1993)MathSciNetCrossRefMATH Barron, A.R.: Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inf. Theory 39, 930–945 (1993)MathSciNetCrossRefMATH
18.
Zurück zum Zitat Kůrková, V.: Complexity estimates based on integral transforms induced by computational units. Neural Netw. 33, 160–167 (2012)CrossRefMATH Kůrková, V.: Complexity estimates based on integral transforms induced by computational units. Neural Netw. 33, 160–167 (2012)CrossRefMATH
19.
Zurück zum Zitat Kůrková, V., Savický, P., Hlaváčková, K.: Representations and rates of approximation of real-valued Boolean functions by neural networks. Neural Netw. 11, 651–659 (1998)CrossRef Kůrková, V., Savický, P., Hlaváčková, K.: Representations and rates of approximation of real-valued Boolean functions by neural networks. Neural Netw. 11, 651–659 (1998)CrossRef
20.
Zurück zum Zitat Ball, K.: An elementary introduction to modern convex geometry. In: Levy, S. (ed.) Flavors of Geometry, pp. 1–58. Cambridge University Press (1997) Ball, K.: An elementary introduction to modern convex geometry. In: Levy, S. (ed.) Flavors of Geometry, pp. 1–58. Cambridge University Press (1997)
22.
Zurück zum Zitat Schläfli, L.: Theorie der Vielfachen Kontinuität. Zürcher & Furrer, Zürich (1901) Schläfli, L.: Theorie der Vielfachen Kontinuität. Zürcher & Furrer, Zürich (1901)
23.
Zurück zum Zitat Cover, T.M.: Geometrical and statistical properties of systems of linear inequalities with applictions in pattern recognition. IEEE Trans. Electron. Comput. 14, 326–334 (1965)CrossRefMATH Cover, T.M.: Geometrical and statistical properties of systems of linear inequalities with applictions in pattern recognition. IEEE Trans. Electron. Comput. 14, 326–334 (1965)CrossRefMATH
24.
Zurück zum Zitat Candès, E.J.: The restricted isometric property and its implications for compressed sensing. C. R. Acad. Sci. Paris I 346, 589–592 (2008)CrossRefMATH Candès, E.J.: The restricted isometric property and its implications for compressed sensing. C. R. Acad. Sci. Paris I 346, 589–592 (2008)CrossRefMATH
25.
Zurück zum Zitat Roychowdhury, V., Siu, K.Y., Orlitsky, A.: Neural models and spectral methods. In: Roychowdhury, V., Siu, K., Orlitsky, A. (eds.) Theoretical Advances in Neural Computation and Learning, pp. 3–36. Springer, New York (1994)CrossRef Roychowdhury, V., Siu, K.Y., Orlitsky, A.: Neural models and spectral methods. In: Roychowdhury, V., Siu, K., Orlitsky, A. (eds.) Theoretical Advances in Neural Computation and Learning, pp. 3–36. Springer, New York (1994)CrossRef
26.
Zurück zum Zitat Laughlin, S.B., Sejnowski, T.J.: Communication in neural networks. Science 301, 1870–1874 (2003)CrossRef Laughlin, S.B., Sejnowski, T.J.: Communication in neural networks. Science 301, 1870–1874 (2003)CrossRef
Metadaten
Titel
Sparsity of Shallow Networks Representing Finite Mappings
verfasst von
Věra Kůrková
Copyright-Jahr
2017
DOI
https://doi.org/10.1007/978-3-319-65172-9_29