Top

Neural Computing and Applications

Published in:

14-06-2021 | Original Article

A wavelet-based neural network scheme for supervised and unsupervised learning

Authors: Manuel L. Esquível, Nadezhda P. Krasii

Published in: Neural Computing and Applications | Issue 20/2021

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

We introduce a scheme for supervised and unsupervised learning based on successive decomposition of random inputs by means of wavelet basis. To each successive layer corresponds a different wavelet basis with more null moments than its predecessor. We consider two types of operations in the scheme: firstly, a input signal treatment phase—the awake phase—and, secondly, a reorganizing phase of the random wavelet coefficients obtained in the previous awake phase—the asleep phase. The next awake phase input treatment will include a feedback derived from the previous asleep phase. The set of random wavelet coefficients of the deepest layer—at each stage of the learning process—is supposed to be Gaussian distributed, and the corresponding sequence of Gaussian distributions constitutes the inner representation of the world in the scheme. We show that in the case of a constant average value of the inputs in each successive awake phases, the mean value of the feedback converges to a multiple of the constant expected value of the inputs. We show a general result on the stabilization of the Gaussian distribution corresponding to the deepest layer, and we show that when the estimated means and covariances converge, then the sequence of Gaussian distributions of the inner representation of the world in the scheme also converges. We present an example of a neural network scheme for supervised learning corresponding to extracting a signal from data corrupted with Gaussian noise.

previous article Dual attentive graph neural network for metro passenger flow prediction

next article Real-time analysis of adaptive fuzzy predictive controller for chaotification under varying payload and noise conditions

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Adamowski K, Prokoph A, Adamowski J (2009) Development of a new method of wavelet aided trend detection and estimation. Hydrol Process 23(18):2686–2696. https://doi.org/10.1002/hyp.7260CrossRef

Alexandridis AK, Zapranis AD (2013) Wavelet neural networks: a practical guide. Neural Netw 42:1–27. https://doi.org/10.1016/j.neunet.2013.01.008CrossRefMATH

Anderson TW (2003) An introduction to multivariate statistical analysis, 3rd edn. Wiley series in probability and statistics. Wiley, HobokenMATH

Benassi A, Jaffard S, Roux D (1997) Elliptic Gaussian random processes. Rev Mat Iberoam 13(1):19–90MathSciNetCrossRef

Billingsley P (1995) Probability and measure, 3rd edn. Wiley series in probability and mathematical statistics. Wiley, New YorkMATH

Billingsley P (1999) Convergence of probability measures, 2nd edn. Wiley series in probability and statistics: probability and statistics. Wiley, New York. https://doi.org/10.1002/9780470316962CrossRefMATH

Borkar VS, Dwivedi R, Sahasrabudhe N (2016) Gaussian approximations in high dimensional estimation. Syst Control Lett 92:42–45. https://doi.org/10.1016/j.sysconle.2016.03.001MathSciNetCrossRefMATH

Bouny LE, Khalil M, Adib A (2020) An end-to-end multi-level wavelet convolutional neural networks for heart diseases diagnosis. Neurocomputing 417:187–201. https://doi.org/10.1016/j.neucom.2020.07.056CrossRef

Byrnes JS, Byrnes JL, Hargreaves KA (eds) (1994) Wavelets and their applications. In: Proceedings of the NATO ASI conference, 16–29 August 1992, Il Ciocco, Italy., vol 442. Kluwer Academic Publishers, Dordrecht

10.

Chariker L, Shapley R, Young LS (2016) Orientation selectivity from very sparse lgn inputs in a comprehensive model of macaque v1 cortex. J Neurosci 36(49):12368–12384. https://doi.org/10.1523/JNEUROSCI.2603-16.2016CrossRef

11.

Chariker L, Shapley R, Young LS (2018) Rhythm and synchrony in a cortical network model. J Neurosci 38(40):8621–8634. https://doi.org/10.1523/JNEUROSCI.0675-18.2018CrossRef

12.

Dacunha-Castelle D, Revuz D, Schreiber M (1970) Recueil de problèmes de calcul des probabilités. Deuxième édition, revue et augmentée. Préfaces de A. Tortrat. Masson et Cie, Éditeurs, Paris

13.

Daubechies I (1992) Ten lectures on wavelets, vol 61. SIAM, PhiladelphiaCrossRef

14.

Donoho DL, Johnstone IM (1994) Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3):425–455MathSciNetCrossRef

15.

Donoho DL, Johnstone IM (1999) Asymptotic minimaxity of wavelet estimators with sampled data. Stat Sin 9(1):1–32MathSciNetMATH

16.

Fujieda S, Takayama K, Hachisuka T (2018) Wavelet convolutional neural networks. arXiv:1907.03128

17.

Ghiasi MM, Bahadori A, Zendehboudi S, Chatzis I (2015) Rigorous models to optimise stripping gas rate in natural gas dehydration units. Fuel 140:421–428. https://doi.org/10.1016/j.fuel.2014.09.084CrossRef

18.

Grim J, Hora J (2008) Iterative principles of recognition in probabilistic neural networks. Neural Netw 21(6):838–846. https://doi.org/10.1016/j.neunet.2008.03.002 (Computational and Biological Inspired Neural Networks, selected papers from ICANN 2007)CrossRefMATH

19.

Hinton G, Dayan P, Frey B, Neal R (1995) The wake-sleep algorithm for unsupervised neural networks. Science 268(5214):1158–1161CrossRef

20.

Huang M, Cui B (2005) A novel learning algorithm for wavelet neural networks. In: Advances in Natural Computation. ICNC 2005. Lecture Notes in Computer Science, Springer, Berlin, Heidelberg, vol 3610, pp 1–7

21.

Jaffard S (1994) Regularity analysis of functions and random processes using wavelets. In: Proceedings of the NATO ASI conference on wavelets and their applications, 16–29 August 1992, Il Ciocco, Italy, Dordrecht: Kluwer Academic Publishers, pp 51–82

22.

Jaffard S, Meyer Y (1996) Wavelet methods for pointwise regularity and local oscillations of functions, vol 587. American Mathematical Society (AMS), ProvidenceMATH

23.

Kallenberg O (2002) Foundations of modern probability, 2nd edn. Probability and its applications. Springer-Verlag, New York. https://doi.org/10.1007/978-1-4757-4015-8CrossRefMATH

24.

Kamari A, Mohammadi AH, Bahadori A, Zendehboudi S (2014) Prediction of air specific heat ratios at elevated pressures using a novel modeling approach. Chem Eng Technol 37(12):2047–2055. https://doi.org/10.1002/ceat.201400261CrossRef

25.

Kiseľák J, Lu Y, Švihra J, Szépe P, Stehlík M (2020) SPOCU: scaled polynomial constant unit activation function. Neural Comput Appl. https://doi.org/10.1007/s00521-020-05182-1CrossRef

26.

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds). Advances in neural information processing systems, Curran Associates, Inc., vol 25, pp 1097–1105. https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf

27.

Kumar R, Srivastava S, Gupta J, Mohindru A (2018) Self-recurrent wavelet neural network-ai based identification and adaptive predictive control of nonlinear dynamical systems. Int J Adapt Control Signal Process 32(9):1326–1358. https://doi.org/10.1002/acs.2916CrossRefMATH

28.

Lecun Y (1989) Generalization and network design strategies. In: Pfeifer R, Schreter Z, Fogelman F, Steels L (eds) Connectionism in perspective, Elsevier, chap 8, pp 143–156

29.

LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551. https://doi.org/10.1162/neco.1989.1.4.541CrossRef

30.

Li W, Yao J (2018) On structure testing for component covariance matrices of a high dimensional mixture. J R Stat Soc Ser B (Stat Methodol) 80(2):293–318. https://doi.org/10.1111/rssb.12248MathSciNetCrossRefMATH

31.

Liu P, Zhang H, Lian W, Zuo W (2019) Multi-level wavelet convolutional neural networks. IEEE Access 7:74973–74985. https://doi.org/10.1109/ACCESS.2019.2921451CrossRef

32.

Meyer Y (1990) Ondelettes et opérateurs. I: Ondelettes. Paris: Hermann, Éditeurs des Sciences et des Arts

33.

Meyer Y (1993) Wavelets: algorithms and applications. Transl. and rev. by Robert D. Ryan. Philadelphia, PA: SIAM

34.

Morlet J, Arens G, Fourgeau E, Giard D (1982) Wave propagation and sampling theory; Part II, Sampling theory and complex waves. Geophysics 47(2):222–236. https://doi.org/10.1190/1.1441329CrossRef

35.

Morlet J, Arens G, Fourgeau E, Glard D (1982) Wave propagation and sampling theory-part i: complex signal and scattering in multilayered media. Geophysics 47(2):203–221. https://doi.org/10.1190/1.1441328CrossRef

36.

Narayanaswamy CR, Raghavarao D (1991) Principal component analysis of large dispersion matrices. J R Stat Soc Ser C (Appl Stat) 40(2):309–316. https://doi.org/10.2307/2347595MathSciNetCrossRefMATH

37.

Pati YC, Krishnaprasad PS (1993) Analysis and synthesis of feedforward neural networks using discrete affine wavelet transformations. IEEE Trans Neural Netw 4(1):73–85. https://doi.org/10.1109/72.182697CrossRef

38.

Rowe ACH, Abbott PC (1995) Daubechies wavelets and mathematica. Comput Phys 9(6):635–648. https://doi.org/10.1063/1.168556CrossRef

39.

Ruskai MB, Beylkin G, Coifman R, Daubechies I, Mallat S, Meyer Y, Raphael L (eds) (1992) Wavelets and their applications. Jones and Bartlett Publishers, BostonMATH

40.

Salazar L, Nicolis O, Ruggeri F, Kiseľák J, Stehlík M (2019) Predicting hourly ozone concentrations using wavelets and ARIMA models. Neural Comput Appl 31(8):4331–4340. https://doi.org/10.1007/s00521-018-3345-0CrossRef

41.

Shiryaev AN (2012) Problems in probability. Problem books in mathematics. Springer, New York. https://doi.org/10.1007/978-1-4614-3688-1CrossRefMATH

42.

Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations, pp 1–14. arXiv:1409.1556

43.

Suykens JAK, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300. https://doi.org/10.1023/A:1018628609742CrossRef

44.

Talebi R, Ghiasi MM, Talebi H, Mohammadyian M, Zendehboudi S, Arabloo M, Bahadori A (2014) Application of soft computing approaches for modeling saturation pressure of reservoir oils. J Natural Gas Sci Eng 20:8–15. https://doi.org/10.1016/j.jngse.2014.04.023CrossRef

45.

Thiele J, Diehl PU, Cook M (2017) A wake-sleep algorithm for recurrent, spiking neural networks. CoRR. arXiv:1703.06290

46.

Torrence C, Compo GP (1998) A practical guide to wavelet analysis. Bull Am Meteorol Soc 79(1):61–78. https://doi.org/10.1175/1520-0477(1998)0790061:APGTWA2.0.CO;2CrossRef

47.

Towell GG, Shavlik JW (1993) Extracting refined rules from knowledge-based neural networks. Mach Learn 13(1):71–101. https://doi.org/10.1023/A:1022683529158CrossRef

48.

Vardi MY (2019) The long game of research. Commun ACM 62(9):7. https://doi.org/10.1145/3352489CrossRef

49.

Veitch D (2005) Wavelet neural networks and their application in the study of dynamical systems. Master’s thesis, Department of Mathematics, University of York, Helsington, UK. MSc. in Data Analysis, Networks and Nonlinear Dynamics

50.

Williams T, Li R (2018) Wavelet pooling for convolutional neural networks. In: International conference on learning representations, pp 1–12. https://openreview.net/forum?id=rkhlb8lCZ

51.

Zendehboudi S, Rezaei N, Lohi A (2018) Applications of hybrid models in chemical, petroleum, and energy systems: a systematic review. Appl Energy 228:2539–2566. https://doi.org/10.1016/j.apenergy.2018.06.051CrossRef

52.

Zhang Q, Benveniste A (1992) Wavelet networks. IEEE Trans Neural Netw 3(6):889–898. https://doi.org/10.1109/72.165591CrossRef

53.

Zhang Z, Shi Y, Toda H, Akiduki T (2017) A study of a new wavelet neural network for deep learning. In: 2017 international conference on wavelet analysis and pattern recognition (ICWAPR), pp 127–131. https://doi.org/10.1109/ICWAPR.2017.8076676

Title: A wavelet-based neural network scheme for supervised and unsupervised learning
Authors: Manuel L. Esquível
Nadezhda P. Krasii
Publication date: 14-06-2021
Publisher: Springer London
Published in: Neural Computing and Applications / Issue 20/2021
Print ISSN: 0941-0643
Electronic ISSN: 1433-3058
DOI: https://doi.org/10.1007/s00521-021-05968-x

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 20/2021

Wavelet neural learning-based type-2 fuzzy PID controller for speed regulation in BLDC motor

Revisiting convolutional neural network on graphs with polynomial approximations of Laplace–Beltrami spectral filtering

Robust color image hashing using convolutional stacked denoising auto-encoders for image authentication

Text mining using nonnegative matrix factorization and latent semantic analysis

A novel measure to evaluate generative adversarial networks based on direct analysis of generated images

Deep proximal support vector machine classifiers for hyperspectral images classification

Premium Partner