Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
DE
EN
Books
Journals
Events
Topic Page
Marketing
Start single access now
Access for companies
12th International Engine Congress 2025 | 25. + 26. February 2025

Meeting Place for the Powertrain and Sustainable Fuels Community

The congress will examine the future role of the internal combustion engine and its non-fossil fuels from a global perspective with leading experts. Be part of it! More information and booking options are available on our website.
EXTENDED SEARCH
Log in
Top
Foundations of Computational Mathematics
Published in:

11-04-2022

Free Component Analysis: Theory, Algorithms and Applications

Authors: Raj Rao Nadakuditi, Hao Wu

Published in: Foundations of Computational Mathematics | Issue 3/2023

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

We describe a method for unmixing mixtures of freely independent random variables in a manner analogous to the independent component analysis (ICA)-based method for unmixing independent random variables from their additive mixtures. Random matrices play the role of free random variables in this context so the method we develop, which we call free component analysis (FCA), unmixes matrices from additive mixtures of matrices. Thus, while the mixing model is standard, the novelty and difference in unmixing performance comes from the introduction of a new statistical criteria, derived from free probability theory, that quantify freeness analogous to how kurtosis and entropy quantify independence. We describe the theory, the various algorithms, and compare FCA to vanilla ICA which does not account for spatial or temporal structure. We highlight why the statistical criteria make FCA also vanilla despite its matricial underpinnings and show that FCA performs comparably to, and sometimes better than, (vanilla) ICA in every application, such as image and speech unmixing, where ICA has been known to succeed. Our computational experiments suggest that not-so-random matrices, such as images and short-time Fourier transform matrix of waveforms are (closer to being) freer “in the wild” than we might have theoretically expected.
previous article A Unified Approach to Uniform Signal Recovery From Nonlinear Observations
next article Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularities

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
Appendix
Available only for authorised users
Footnotes
1
Here \({{\widehat{F}}}(\cdot )\) is either the (self-adjoint or rectangular) free kurtosis, the free entropy or a higher (than fourth)-order (even-valued) free cumulant. See Table 2.
 
Literature
1.
Almeida, L.B.: MISEP–Linear and nonlinear ICA based on mutual information. Journal of Machine Learning Research 4(Dec), 1297–1318 (2003)MATH
2.
Anderson, G.W., Farrell, B.: Asymptotically liberating sequences of random unitary matrices. Advances in Mathematics 255, 381–413 (2014)MathSciNetCrossRefMATH
3.
Arora, S., Ge, R., Moitra, A., Sachdeva, S.: Provable ica with unknown gaussian noise, with implications for gaussian mixtures and autoencoders. In: Advances in Neural Information Processing Systems, pp. 2375–2383 (2012)
4.
Barry, D., Coyle, E., Fitzgerald, D., Lawlor, R.: Single channel source separation using short-time independent component analysis. In: Audio Engineering Society Convention 119 (2005). Audio Engineering Society
5.
Bell, A.J., Sejnowski, T.J.: The “independent components” of natural scenes are edge filters. Vision research 37(23), 3327–3338 (1997)CrossRef
6.
Benaych-Georges, F.: Rectangular random matrices, entropy, and fisher’s information. Journal of Operator Theory, 371–419 (2009a)
7.
Benaych-Georges, F.: Rectangular random matrices, related convolution. Probability Theory and Related Fields 144(3-4), 471–515 (2009b)MathSciNetCrossRefMATH
8.
Benaych-Georges, F.: Rectangular random matrices, related convolution. Probability Theory and Related Fields 144(3-4), 471–515 (2009c)MathSciNetCrossRefMATH
9.
Bofill, P., Zibulevsky, M.: Underdetermined blind source separation using sparse representations. Signal processing 81(11), 2353–2362 (2001)CrossRefMATH
10.
Boumal, N., Mishra, B., Absil, P.-A., Sepulchre, R.: Manopt, a Matlab toolbox for optimization on manifolds. Journal of Machine Learning Research 15, 1455–1459 (2014)MATH
11.
12.
Cardoso, J.-F.: High-order contrasts for independent component analysis. Neural computation 11(1), 157–192 (1999)MathSciNetCrossRef
13.
Casey, M.A., Westner, A.: Separation of mixed audio sources by independent subspace analysis. In: ICMC, pp. 154–161 (2000)
14.
15.
16.
Chissom, B.S.: Interpretation of the kurtosis statistic. The American Statistician 24(4), 19–22 (1970)
17.
18.
Comon, P.: Independent component analysis, a new concept? Signal processing 36(3), 287–314 (1994)CrossRefMATH
19.
Comon, P., Jutten, C.: Handbook of Blind Source Separation: Independent Component Analysis and Applications. Academic press, Cambridge, MA (2010)
20.
Cornish, E.A., Fisher, R.A.: Moments and cumulants in the specification of distributions. Revue de l’Institut international de Statistique, 307–320 (1938)
21.
Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley & Sons, Hoboken, NJ (2012)MATH
22.
Cruces, S., Castedo, L., Cichocki, A.: Robust blind source separation algorithms using cumulants. Neurocomputing 49(1-4), 87–118 (2002)CrossRefMATH
23.
Davies, M.E., James, C.J.: Source separation using single channel ica. Signal Processing 87(8), 1819–1832 (2007)CrossRefMATH
24.
De Lathauwer, L., Castaing, J., Cardoso, J.-F.: Fourth-order cumulant-based blind identification of underdetermined mixtures. IEEE Transactions on Signal Processing 55(6), 2965–2973 (2007)MathSciNetCrossRefMATH
25.
26.
Eriksson, J., Koivunen, V.: Blind identifiability of class of nonlinear instantaneous ICA models. In: 2002 11th European Signal Processing Conference, pp. 1–4 (2002). IEEE
27.
Eriksson, J., Koivunen, V.: Identifiability, separability, and uniqueness of linear ica models. IEEE signal processing letters 11(7), 601–604 (2004)CrossRef
28.
Frieze, A., Jerrum, M., Kannan, R.: Learning linear transformations. In: Proceedings of 37th Conference on Foundations of Computer Science, pp. 359–368 (1996). IEEE
29.
Gao, P., Chang, E.-C., Wyse, L.: Blind separation of fetal ecg from single mixture using svd and ica. In: Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia. Proceedings of the 2003 Joint, vol. 3, pp. 1418–1422 (2003). IEEE
30.
Griffin, G., Holub, A., Perona, P.: Caltech-256 object category dataset (2007)
31.
Haykin, S., Chen, Z.: The cocktail party problem. Neural computation 17(9), 1875–1902 (2005)CrossRef
32.
Hiai, F., Petz, D.: The Semicircle Law, Free Random Variables and Entropy. Mathematical Surveys and Monographs, vol. 77, p. 376. American Mathematical Society, Providence, RI (2000)
33.
Hoyer, P.O., Hyvärinen, A.: Independent component analysis applied to feature extraction from colour and stereo images. Network: computation in neural systems 11(3), 191–210 (2000)CrossRefMATH
34.
Hyvarinen, A.J., Morioka, H.: Nonlinear ICA of temporally dependent stationary sources. (2017). Proceedings of Machine Learning Research
35.
Hyvarinen, A.: A family of fixed-point algorithms for independent component analysis. In: 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 5, pp. 3917–3920 (1997a). IEEE
36.
Hyvarinen, A.: One-unit contrast functions for independent component analysis: A statistical analysis. In: Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop, pp. 388–397 (1997b). IEEE
37.
Hyvarinen, A.: Fast and robust fixed-point algorithms for Independent Component Analysis. IEEE transactions on Neural Networks 10(3), 626–634 (1999)CrossRef
38.
Hyvarinen, A., Morioka, H.: Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. In: Advances in Neural Information Processing Systems, pp. 3765–3773 (2016)
39.
Hyvärinen, A., Oja, E.: Independent component analysis: algorithms and applications. Neural networks 13(4-5), 411–430 (2000)CrossRef
40.
Hyvärinen, A., Karhunen, J., Oja, E.: Independent Component Analysis vol. 46. John Wiley & Sons, Hoboken, NJ (2004)
41.
Hyvarinen, A., Sasaki, H., Turner, R.E.: Nonlinear ICA using auxiliary variables and generalized contrastive learning. arXiv preprint arXiv:​1805.​08651 (2018)
42.
Ilmonen, P., Nordhausen, K., Oja, H., Ollila, E.: A new performance index for ica: properties, computation and asymptotic analysis. In: International Conference on Latent Variable Analysis and Signal Separation, pp. 229–236 (2010). Springer
43.
Lee, T.-W.: Independent Component Analysis. In: Independent Component Analysis, pp. 27–66. Springer, Boston (1998)
44.
Lee, T.-W., Girolami, M., Bell, A.J., Sejnowski, T.J.: A unifying information-theoretic framework for independent component analysis. Computers & Mathematics with Applications 39(11), 1–21 (2000)MathSciNetCrossRefMATH
45.
Lehner, F.: Cumulants in noncommutative probability theory i. noncommutative exchangeability systems. Mathematische Zeitschrift 248(1), 67–100 (2004)MathSciNetCrossRefMATH
46.
Male, C.: Traffic distributions and independence: permutation invariant random matrices and the three notions of independence. arXiv preprint arXiv:​1111.​4662 (2011)
47.
Meyer, C.D., Stewart, G.W.: Derivatives and perturbations of eigenvectors. SIAM Journal on Numerical Analysis 25(3), 679–691 (1988)MathSciNetCrossRefMATH
48.
Mika, D., Budzik, G., Jozwik, J.: Single channel source separation with ica-based time-frequency decomposition. Sensors 20(7), 2019 (2020)CrossRef
49.
Mingo, J.A., Speicher, R.: Free Probability and Random Matrices vol. 35. Springer, New York (2017)MATH
50.
Mitsui, Y., Kitamura, D., Takamichi, S., Ono, N., Saruwatari, H.: Blind Source Separation based on independent low-rank matrix analysis with sparse regularization for time-series activity. In: Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference On, pp. 21–25 (2017). IEEE
51.
52.
53.
Nica, A., Speicher, R.: Lectures on the Combinatorics of Free Probability vol. 13. Cambridge University Press, Cambridge (2006)CrossRefMATH
54.
Oja, E., Yuan, Z.: The fastica algorithm revisited: Convergence analysis. IEEE Transactions on Neural Networks 17(6), 1370–1381 (2006)CrossRef
55.
Pearson, K.: Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2(11), 559–572 (1901)CrossRefMATH
56.
Pourazad, M., Moussavi, Z., Farahmand, F., Ward, R.: Heart sounds separation from lung sounds using independent component analysis. In: 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, pp. 2736–2739 (2006). IEEE
57.
Smith, P.J.: A recursive formulation of the old problem of obtaining moments from cumulants and vice versa. The American Statistician 49(2), 217–218 (1995)MathSciNet
58.
Speicher, R.: Multiplicative functions on the lattice of non-crossing partitions and free convolution. Mathematische Annalen 298(1), 611–628 (1994)MathSciNetCrossRefMATH
59.
60.
Voiculescu, D.: The analogues of entropy and of fisher’s information measure in free probability theory, i. Communications in mathematical physics 155(1), 71–92 (1993)MathSciNetCrossRefMATH
61.
Voiculescu, D.: The analogues of entropy and of fisher’s information measure in free probability theory, ii. Inventiones mathematicae 118(1), 411–440 (1994)MathSciNetCrossRefMATH
62.
Voiculescu, D.: Operations on certain non-commutative operator-valued random variables, in recent advances in operator algebras. Astérisque 232, 243–275 (1995)MATH
63.
Voiculescu, D.: The analogues of entropy and of fisher’s information measure in free probability theory, iv: maximum entropy and freeness, in free probability theory. Fields Inst. Commun. 12, 293–302 (1997)MATH
Metadata
Title
Free Component Analysis: Theory, Algorithms and Applications
Authors
Raj Rao Nadakuditi
Hao Wu
Publication date
11-04-2022
Publisher
Springer US
Published in
Foundations of Computational Mathematics / Issue 3/2023
Print ISSN: 1615-3375
Electronic ISSN: 1615-3383
DOI
https://doi.org/10.1007/s10208-022-09564-w

Other articles of this Issue 3/2023

Continuity of Formal Power Series Products in Nonlinear Control Theory

  • Open Access

A Generalized Conditional Gradient Method for Dynamic Inverse Problems with Optimal Transport Regularization

  • Open Access

Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularities

  • Open Access

A Unified Approach to Uniform Signal Recovery From Nonlinear Observations

The Canny–Emiris Conjecture for the Sparse Resultant

  • Open Access

Premium Partner

    Image Credits