Pointwise and uniform approximation by multivariate neural network operators of the max-product type
Introduction
The theory of neural network (NN) operators has been introduced in last years as a constructive approach for approximating functions by a neural process. The latter topic, other than the “classical” theory of artificial neural networks (NNs), is strictly related to the Approximation Theory. Indeed, many papers have been written concerning applications of NNs to the above subject, see e.g. Barron (1993), Cao and Chen, 2009, Cao and Chen, 2012, Cybenko (1989), Di Marco, Forti, Grazzini, and Pancioni (2014), Di Marco, Forti, Nistri, and Pancioni (2015), Gnecco and Sanguineti (2011), Gripenberg (2003), Ismailov (2014), Ito (2001), Kainen and Kurková (2009), Llanas and Sainz (2006) and Sontag (1992). The main difficulty arising in order to prove approximation results by means of NNs was to become able to construct concretely NNs which approximate a given function defined on a fixed bounded set of (). This problem has been solved, e.g., for functions of one variable by various authors (see e.g., Cheang, 2010, Maiorov, 2006 and Makovoz, 1998) which gave constructive proofs, for instance, in the space of continuous functions. Much more difficulties arise when dealing with functions of several variables. Some results have been obtained by (quite non standard) approaches based on convolution, or resorting to the theory of ridge functions, see e.g. Cheney, Light, and Xu (1993). The constructive approach for approximating functions by NNs based on the study of a family of operators of NN type, has been introduced by Anastassiou in Anastassiou (1997) and was inspired by a paper of Cardaliaguet and Euvrard Cardaliaguet and Euvrard (1992).
The theory of NN operators was largely investigated in a number of articles, recently collected in the monograph (Anastassiou, 2011). The approximation results mainly proved in Anastassiou (2011), both for functions of one and several variables, involve NN operators activated by logistic and hyperbolic tangent sigmoidal functions only. The results proved in Anastassiou (2011) have been successfully improved in the paper (Costarelli & Spigler, 2013), both for what concerns the order of approximation, and for the activation functions used for the NN operators. More precisely, in Costarelli and Spigler (2013), convergence results have been proved for NN operators activated by any sigmoidal functions satisfying suitable, not restrictive, assumptions.
Recently, an interesting procedure was introduced by Coroianu and Gal, which allows to improve the order of approximation that can be achieved by a family of linear operators, and consists in the so-called max-product approach, see e.g. Coroianu and Gal (2010), Coroianu and Gal (2011) and Coroianu and Gal (2012). In general, discrete linear operators are defined by finite sums or series, with respect to certain indexes. The max-product operators are defined by replacing the sums or the series by a maximum or a supremum, computed over the same sets of indexes.
The above procedure, allows to convert linear operators into nonlinear ones, which are able to achieve an higher order of approximation with respect to their linear counterparts, see e.g., Coroianu and Gal (2010), Coroianu and Gal (2011) and Coroianu and Gal (2012).
In Anastassiou, Coroianu, and Gal (2010) the max-product approach is applied to NN operators. The activation functions used to define the above max-product NN operators were centered bell-shaped functions, with compact support, and satisfying some other quite restrictive assumptions.
The results proved in Anastassiou et al. (2010) have been successfully extended and improved in Costarelli and Vinti (2016), in case of approximation of functions of one variable. Here, the assumptions required on the activation functions have been weakened, considering bell-shaped functions, not necessarily with compact support, generated by a suitable finite linear combination of sigmoidal functions.
Since in general, neurocomputing process involve high dimensional data, it becomes interesting to obtain a multivariate extension of the results proved in Costarelli and Vinti (2016). For this reason, in this paper, we provide results in this direction.
Indeed, pointwise and uniform approximation results for a family of multivariate NN operators of the max-product type are proved. Moreover, some estimates for the order of approximation are derived.
In general, by means of NN operators we are able to approximate functions defined on bounded domains only, and in the present paper, in order to approximate functions defined on the whole space , a class of quasi-interpolation operators is studied.
Finally, we give a brief description of the plan of the present article. In Section 2, some notations are firstly introduced, then some preliminary results concerning the activation functions used for the max-product NN operators are proved. In Section 3 the NN and quasi-interpolation operators are introduced and studied. In both cases, pointwise and uniform convergence theorem are proved, when continuous functions are approximated. Moreover, estimates concerning the order of approximation are derived in both the previous cases. Finally, in Section 4 many examples of sigmoidal functions for which the previous theory can be applied are shown, while in Section 5 the main conclusions of the paper are pointed out.
Section snippets
Auxiliary results and notations
In the present paper, we will denote by a multivariate box-domain in . By means of the symbols and we will denote the spaces of continuous functions and , respectively. Moreover, we denote by the space of all uniformly continuous functions . In all the above settings, we consider the usual sup-norm .
Further, by the symbols and , we will denote respectively the “ceiling” and the “integral part” of a given number.
The multivariate neural network and quasi-interpolation operators of the max-product type
Now, we are able to introduce the operators that we will study in the rest of the paper. Definition 3.1 Let be a bounded function, with , and such that , . The multivariate max-product neural network (NN) operators activated by are defined by: where denotes the set of indexes , such that , , for every sufficiently large.
In general, when is sufficiently large, it
Examples of sigmoidal activation functions
In this section, some examples of sigmoidal functions for which the above results can be applied will be shown.
First of all, we present some typical examples of sigmoidal functions used as activation functions in neural network approximation processes. We consider the logistic and hyperbolic tangent sigmoidal functions, defined respectively by: Clearly, and are smooth functions and they satisfy all the assumptions of Section 2. In particular, due
Conclusions
The max-product NN operators represent neuroprocessing models in which the global behavior of the network is mainly determined by one of the artificial neuron of the network.
The multivariate NN operators of the max-product type, allow us to obtain a constructive, nonlinear approximation formula based on a kind of neural networks which allow us to achieve more accurate approximations than the corresponding linear counterparts studied in Costarelli and Spigler (2013).
More precisely, we point out
Acknowledgments
The authors are members of the Gruppo Nazionale per l’Analisi Matematica, la Probabilitá e le loro Applicazioni (GNAMPA) of the Istituto Nazionale di Alta Matematica (INdAM).
The authors of the present paper have been supported by the Department of Mathematics and Computer Science of the University of Perugia (Italy). Moreover, the first author of the paper has been partially supported within the GNAMPA-INdAM Project “Metodi di approssimazione e applicazioni al Signal e Image Processing”;
References (43)
Rate of convergence of some neural network operators to the unit-univariate case
Journal of Mathematical Analysis and Applications
(1997)- et al.
Approximation of a function and its derivative with a neural network
Neural Networks
(1992) Approximation with neural networks activated by ramp sigmoids
Journal of Approximation Theory
(2010)- et al.
Enhancement of thermographic images as tool for structural analysis in earthquake engineering
NDT & E International
(2015) Neural network operators: constructive interpolation of multivariate functions
Neural Networks
(2015)- et al.
Multivariate neural network operators with sigmoidal activation functions
Neural Networks
(2013) - et al.
Convergence of a family of neural network operators of the Kantorovich type
Journal of Approximation Theory
(2014) - et al.
Max-product neural network and quasi interpolation operators activated by sigmoidal functions
Journal of Approximation Theory
(2016) - et al.
Necessary and sufficient condition for multistability of neural networks evolving on a closed hypercube
Neural Networks
(2014) Approximation by neural network with a bounded number of nodes at each level
Journal of Approximation Theory
(2003)