Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Information Theory, Machine Learning, and Reproducing Kernel Hilbert Spaces Read chapter Read first chapter

2010 | OriginalPaper | Chapter

8. Self-Organizing ITL Principles for Unsupervised Learning

Authors : Sudhir Rao, Deniz Erdogmus, Dongxin Xu, Kenneth Hild II

Published in: Information Theoretic Learning

Publisher: Springer New York

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Chapter 1 presented a synopsis of information theory to understand its foundations and how it affected the field of communication systems. In a nutshell, mutual information characterizes the fundamental compromise of maximum rate for error-free information transmission (the channel capacity theorem) as well as the minimal information that needs to be sent for a given distortion (the rate distortion theorem). In essence given the statistical knowledge of the data and these theorems the optimal communication system emerges, or self-organizes from the data.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Clustering with ITL Principles
next chapter A Reproducing Kernel Hilbert Space Framework for ITL
Literature
1.
Aczél J., Daróczy Z., On measures of information and their characterizations, Mathematics in Science and Engineering, vol. 115, Academic Press, New York, 1975.
5.
Amari S., Cichocki A., Yang H., A new learning algorithm for blind signal separation. Advances in Neural Information Processing Systems, vol. 8 pp. 757–763, MIT Press, Cambridge, MA, 1996.
9.
Atick J., Redlich A., Towards a theory of early visual processing, Neural Comput., 2:308–320, 1990.CrossRef
10.
Attneave F., Some informational aspects of visual perception, Psychol. Rev., 61; 183–193, 1954.CrossRef
16.
17.
Barlow H., Kausal T., Mitchison G., Finding minimum entropy codes, Neural Comput., 1(3):412–423, 1989.CrossRef
24.
Becker S., Hinton G., A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355:161–163, 1992CrossRef
25.
Becker. S., Unsupervised learning with global objective functions. In M. A. Arbib, (Ed.), The Handbook of Brain Theory and Neural Networks, pp. 997–1000. MIT Press, Cambridge, MA, 1998c
26.
Bell A., Sejnowski T., An information-maximization approach to blind separation and blind deconvolution. Neural Comput., 7(6):1129–1159, 1995.CrossRef
33.
Benveniste A., Goursat M., Ruget G., Robust identification of a non-minimum phase system: Blind adjustment of a linear equalizer in data communications, IEEE Trans. Autom. Control, 25(3):385–399, 1980.CrossRefMATHMathSciNet
38.
Bishop C., Neural Networks for Pattern Recognition, Clarendon Press, Oxford, 1995.
47.
Cardoso J., Blind signal separation: Statistical principles, Proc. IEEE, 86(10):2009–2025, 1998.CrossRef
52.
Chechik G., Tishby N., Temporal dependent plasticity: An information theoretic account, Proc. Neural Inf. Process. Syst., 13:110–116, 2001.
54.
Chen Z., Haykin S., Eggermont J., Bekker S., Correlative learning: A basis for brain and adaptive systems, John Wiley, Hoboken, NJ, 2007.CrossRef
59.
Choi S., Cichocki A., Amari S., Flexible independent component analysis, J. VLSI Signal Process., 26:25–38, 2000.CrossRefMATH
62.
Comon P., Independent component analysis, a new concept?, Signal Process., 36(3):287–314, 1994.CrossRefMATH
77.
Donoho D., On minimum entropy deconvolution, in Applied Time Series Analysis II, Academic Press, New York, 1981, pp. 565–609.CrossRef
89.
Erdogmus D., Principe J., Hild II K., Do Hebbian synapses estimate entropy?, Proc. IEEE Workshop on Neural Networks for Signal Process., Martigni, Switzerland, pp. 199–208, 2002.
93.
Erdogmus D., Hild II K., Lazaro M., Santamaria I., Principe J., Adaptive blind deconvolution of linear channels using Renyi’s nntropy with Parzen estimation, IEEE Trans. Signal Process., 52(6)1489–1498, 2004.CrossRefMathSciNet
96.
Erdogmus D., Ozertem U., Self-consistent locally defined principal surfaces. In Proc. Int. Conf. Acoustic, Speech and Signal Processing, volume 2, pp. 15–20, April 2007.
110.
Gersho A., Gray R. Vector Quantization and Signal Compression. Springer, New York, 1991
141.
Haykin S., Neural Networks: A Comprehensive Foundation, Prentice Hall, Upper Saddle River, NJ, 1999.MATH
144.
Hebb D., Organization of Behavior: A Neurophysiology Theory, John Wiley, NY, New York, 1949.
146.
Heskes T., Energy functions for self-organizing maps. In E. Oja and S. Kaski, editors, Kohonen Maps, Elsevier, Amsterdam, 1999, pp. 303–316.CrossRef
148.
Hild II K., Erdogmus D., Principe J., Blind source separation using Renyi’s mutual information, IEEE Signal Process. Lett., 8:174–176, 2001.CrossRef
150.
Hild II K., Erdogmus D., Principe J., An analysis of entropy estimators for blind source separation, Signal Process., 86(1):182–194, 2006.CrossRefMATH
152.
Hinton G. and Sejnowski T., Unsupervised learning: Foundations of neural computation, MIT Press, Cambridge, MA, 1999.
155.
Hyvarinen A., Fast and Robust Fixed-Point Algorithms for Independent Component Analysis, IEEE Trans. Neural Netw., 10(3):626–634, 1999.CrossRef
157.
161.
Jaynes E., Probability Theory, the Logic of Science, Cambridge University Press, Cambridge, UK, 2003.CrossRefMATH
171.
181.
Kegl B., Krzyzak A., Piecewise linear skeletonization using principal curves. IEEE Trans. Pattern Anal. Mach. Intell., 24(1):59–74, 2002.CrossRef
196.
LeCun Y., Chopra S., Hadsell R., Ranzato M., Huang F., A tutorial on energy-based learning, in Predicting Structured Data, Bakir, Hofman, Scholkopf, Smola, Taskar (Eds.), MIT Press, Boston, 2006.
197.
Lehn-Schieler T., Hegde H., Erdogmus D., and Principe J., Vector-quantization using information theoretic concepts. Natural Comput., 4:39–51, Jan. 2005.CrossRef
199.
Linsker R., Towards an organizing principle for a layered perceptual network. In D. Z. Anderson (Ed.), Neural Information Processing Systems - Natural and Synthetic. American Institute of Physics, New York, 1988.
207.
MacKay D., Information Theory, Inference and Learning Algorithms. Cambridge University Press, Cambridge, UK, 2003.MATH
213.
Marossero D., Erdogmus D., Euliano N., Principe J., Hild II, K., Independent components analysis for fetal electrocardiogram extraction: A case for the data efficient mermaid algorithm, Proceedings of NNSP’03, pp. 399–408, Toulouse, France, Sep 2003.
228.
Nadal J., Parga N., Nonlinear neurons in the low noise limit: a factorial code maximizes information transfer, Network, 5:561–581, 1994.CrossRef
233.
235.
Papoulis A., Probability, Random Variables and Stochastic Processes, McGraw-Hill, New York, 1965.MATH
245.
Pereira F., Tishby N., Lee L., Distributional clustering of english words. In Meeting of the Association for Computational Linguistics, pp. 183–190, 1993.
246.
Pham D., Vrins, F., Verleysen, M., On the risk of using Renyi’s entropy for blind source separation, IEEE Trans. Signal Process., 56(10):4611–4620, 2008.CrossRefMathSciNet
251.
Principe J., Xu D., Information theoretic learning using Renyi’s quadratic entropy, in Proc. ICA’99, 407–412, Aussois, France, 1999.
252.
Principe, J., Xu D., Fisher J., Information theoretic learning, in unsupervised adaptive filtering, Simon Haykin (Ed.), pp. 265–319, Wiley, New York, 2000.
253.
Principe J., Euliano N., Lefebvre C., Neural Systems: Fundamentals through Simulations, CD-ROM textbook, John Wiley, New York, 2000.
259.
Rao S., Unsupervised Learning: An Information Theoretic Learning Approach, Ph.D. thesis, University of Florida, Gainesville, 2008.
274.
Roweis S., Saul L., Nonlinear dimensionality reduction by locally linear embedding. Science, 290:2323–2326, 2000.CrossRef
305.
Slonim N. Tishby N., The power of word clusters for text classification. In 23rd European Colloquium on Information Retrieval Research, 2001.
316.
Tishby N., Pereira F., and Bialek W., The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, pp. 368–377, 1999.
330.
Watanabe S., Pattern Recognition: Human and Mechanical. Wiley, New York, 1985.
335.
Wu H., Principe J., Simultaneous diagonalization in the frequency domain for source separation, Proc. First Int. Workshop on Ind. Comp. Anal. ICA’99, 245–250, Aussois, France, 1999.
337.
Wyszecki G., Stiles W., Color Science: Concepts and Methods, Quantitative Data and Formulae. Wiley, New York, 1982.
339.
Xu D., Principe J., Fisher J., Wu H., A novel measure for independent component analysis (ICA), in Proc. of ICASSP’98, vol. 2, pp. 1161–1164, 1998
348.
Zemel R., Hinton G., Learning population codes by minimizing the description length, in Unsupervised Learning, Hinton and Sejnowski (Eds.), pp. 261–276, MIT Press, Cambridge, MA, 1999.
Metadata
Title
Self-Organizing ITL Principles for Unsupervised Learning
Authors
Sudhir Rao
Deniz Erdogmus
Dongxin Xu
Kenneth Hild II
Copyright Year
2010
Publisher
Springer New York
Book
Information Theoretic Learning

Print ISBN: 978-1-4419-1569-6

Electronic ISBN: 978-1-4419-1570-2

Copyright Year: 2010

DOI
https://doi.org/10.1007/978-1-4419-1570-2_8

Premium Partner

    Image Credits