Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Bücher
Zeitschriften
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
SLX-Digitalkonferenz: Zukunftswerkstatt 2024

Mehr Umsatz mit Top-Kontakten

Am 7. Mai erfahren Sie, wie Vertriebsteams Leadgenerierung, Leadnurturing und Leadstrategien effizient steuern können.
Erweiterte Suche
Anmelden
nach oben
Erschienen in:
Titan Science Return Quantification Kapitel lesen Erstes Kapitel lesen

2015 | OriginalPaper | Buchkapitel

Sparse hidden units activation in Restricted Boltzmann Machine

verfasst von : Jakub M. Tomczak, Adam Gonczarek

Erschienen in: Progress in Systems Engineering

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Sparsity has become a concept of interest in machine learning for many years. In deep learning sparse solutions play crucial role in obtaining robust and discriminative features. In this paper, we study a new regularization term for sparse hidden units activation in the context of Restricted Boltzmann Machine (RBM). Our proposition is based on the symmetric Kullback-Leibler divergence applied to compare the actual and the desired distribution over the active hidden units. We compare our method against two other enforcing sparsity regularization terms by evaluating the empirical classification error using two datasets: (i) for image classification (MNIST), (ii) for document classification (20-newsgroups).

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren
Vorheriges Kapitel Intelligent system concept for high-energy performance and adaptable user comfort
Nächstes Kapitel Accelerated learning for Restricted Boltzmann Machine with momentum term
Fußnoten
1
In [5] such approach is called selectivity.
 
2
A i⋅  denotes the i th row of matrix A, A ⋅ j denotes the j th column of matrix A, and A ij is the element of matrix A.
 
4
In the experiments we used the small version of the original dataset: http://​www.​cs.​nyu.​edu/​~roweis/​data.​html.
 
Literatur
1.
Bengio, Y.: Learning Deep Architectures for AI. Foundations and Trends® in Machine Learning 2(1):1-127. (2009).
2.
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer New York. (2006).
3.
Cho, K., Ilin, A., & Raiko, T.: Tikhonov-Type regularization for Restricted Boltzmann Machines. In Artificial Neural Networks and Machine Learning (ICANN 2012). pp. 81-88. Springer Berlin Heidelberg. (2012).
4.
Glorot, X., Bordes, A., & Bengio, Y.: Deep Sparse Rectifier Networks. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS 2011). Journal of Machine Learning Research Workshop & Conference Proceedings 15:315-323. (2011).
5.
Goh, H., Thome, N., & Cord, M.: Biasing restricted Boltzmann machines to manipulate latent selectivity and sparsity. In NIPS workshop on deep learning and unsupervised feature learning. (2010).
6.
Hinton, G. E.: Training products of experts by minimizing contrastive divergence. Neural Comput 14:1771-1800. (2002)CrossRefMATH
7.
Hinton, G. E.: A practical guide to training Restricted Boltzmann Machines. In Neural Networks: Tricks of the Trade. pp. 599-619. Springer Berlin Heidelberg. (2012).
8.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. (2012).
9.
Larochelle, H., & Bengio, Y.: Classification using discriminative restricted Boltzmann machines. In Proceedings of the 25th International Conference on Machine learning (ICML 2008). pp. 536-543. (2008, July).
10.
Lee, H., Ekanadham, C., & Ng, A.: Sparse deep belief net model for visual area V2. In Advances in Neural Information Processing Systems (NIPS 2007). pp. 873-880. (2007).
11.
Le, Q.V., Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., & Ng, A.: On optimization methods for deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML 2011). pp. 265-272. (2011).
12.
Le Roux, N., & Bengio, Y.: Representational power of Restricted Boltzmann Machines and deep belief networks. Neural Computation 20(6):1631-1649. (2008).MathSciNetCrossRefMATH
13.
Lewicki, M. S., & Sejnowski, T. J.: Learning overcomplete representations. Neural Computation 12(2):337-365. (2000).CrossRef
14.
Marlin, B.M., Swersky, K., Chen, B., & Freitas, N.D.: Inductive principles for restricted Boltzmann machine learning. In International Conference on Artificial Intelligence and Statistics (ICML 2010). pp. 509-516. (2010).
15.
Martens, J., Chattopadhya, A., Pitassi, T., & Zemel, R.: On the Expressive Power of Restricted Boltzmann Machines. In Advances in Neural Information Processing Systems (NIPS 2013). pp. 2877-2885. (2013).
16.
Nair, V., & Hinton, G. E.: 3D object recognition with deep belief nets. In Advances in Neural Information Processing Systems (NIPS 2009). pp. 1339-1347. (2009).
17.
Nair, V., & Hinton, G. E.: Rectified linear units improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on Machine Learning (ICML 2010). pp. 807-814. (2010).
18.
Smolensky, P.: Information processing in dynamical systems: foundations of harmony theory. In: Parallel distributed processing: explorations in the microstructure of cognition, Vol. 1: Foundations, pp. 194281. MIT Press, Cambridge, MA, USA. (1986)
Metadaten
Titel
Sparse hidden units activation in Restricted Boltzmann Machine
verfasst von
Jakub M. Tomczak
Adam Gonczarek
Copyright-Jahr
2015
Buch
Progress in Systems Engineering

Print ISBN: 978-3-319-08421-3

Electronic ISBN: 978-3-319-08422-0

Copyright-Jahr: 2015

DOI
https://doi.org/10.1007/978-3-319-08422-0_27

Neuer Inhalt

    Bildnachweise