Elsevier

Neural Networks

Volume 81, September 2016, Pages 81-90
Neural Networks

Pointwise and uniform approximation by multivariate neural network operators of the max-product type

https://doi.org/10.1016/j.neunet.2016.06.002Get rights and content

Abstract

In this article, the theory of multivariate max-product neural network (NN) and quasi-interpolation operators has been introduced. Pointwise and uniform approximation results have been proved, together with estimates concerning the rate of convergence. At the end, several examples of sigmoidal activation functions have been provided.

Introduction

The theory of neural network (NN) operators has been introduced in last years as a constructive approach for approximating functions by a neural process. The latter topic, other than the “classical” theory of artificial neural networks (NNs), is strictly related to the Approximation Theory. Indeed, many papers have been written concerning applications of NNs to the above subject, see e.g. Barron (1993), Cao and Chen, 2009, Cao and Chen, 2012, Cybenko (1989), Di Marco, Forti, Grazzini, and Pancioni (2014), Di Marco, Forti, Nistri, and Pancioni (2015), Gnecco and Sanguineti (2011), Gripenberg (2003), Ismailov (2014), Ito (2001), Kainen and Kurková (2009), Llanas and Sainz (2006) and Sontag (1992). The main difficulty arising in order to prove approximation results by means of NNs was to become able to construct concretely NNs which approximate a given function f defined on a fixed bounded set of Rs (sN+). This problem has been solved, e.g., for functions of one variable by various authors (see e.g., Cheang, 2010, Maiorov, 2006 and Makovoz, 1998) which gave constructive proofs, for instance, in the space of continuous functions. Much more difficulties arise when dealing with functions of several variables. Some results have been obtained by (quite non standard) approaches based on convolution, or resorting to the theory of ridge functions, see e.g. Cheney, Light, and Xu (1993). The constructive approach for approximating functions by NNs based on the study of a family of operators of NN type, has been introduced by Anastassiou in Anastassiou (1997) and was inspired by a paper of Cardaliaguet and Euvrard Cardaliaguet and Euvrard (1992).

The theory of NN operators was largely investigated in a number of articles, recently collected in the monograph (Anastassiou, 2011). The approximation results mainly proved in Anastassiou (2011), both for functions of one and several variables, involve NN operators activated by logistic and hyperbolic tangent sigmoidal functions only. The results proved in Anastassiou (2011) have been successfully improved in the paper (Costarelli & Spigler, 2013), both for what concerns the order of approximation, and for the activation functions used for the NN operators. More precisely, in Costarelli and Spigler (2013), convergence results have been proved for NN operators activated by any sigmoidal functions satisfying suitable, not restrictive, assumptions.

Recently, an interesting procedure was introduced by Coroianu and Gal, which allows to improve the order of approximation that can be achieved by a family of linear operators, and consists in the so-called max-product approach, see e.g. Coroianu and Gal (2010), Coroianu and Gal (2011) and Coroianu and Gal (2012). In general, discrete linear operators are defined by finite sums or series, with respect to certain indexes. The max-product operators are defined by replacing the sums or the series by a maximum or a supremum, computed over the same sets of indexes.

The above procedure, allows to convert linear operators into nonlinear ones, which are able to achieve an higher order of approximation with respect to their linear counterparts, see e.g., Coroianu and Gal (2010), Coroianu and Gal (2011) and Coroianu and Gal (2012).

In Anastassiou, Coroianu, and Gal (2010) the max-product approach is applied to NN operators. The activation functions used to define the above max-product NN operators were centered bell-shaped functions, with compact support, and satisfying some other quite restrictive assumptions.

The results proved in Anastassiou et al. (2010) have been successfully extended and improved in Costarelli and Vinti (2016), in case of approximation of functions of one variable. Here, the assumptions required on the activation functions have been weakened, considering bell-shaped functions, not necessarily with compact support, generated by a suitable finite linear combination of sigmoidal functions.

Since in general, neurocomputing process involve high dimensional data, it becomes interesting to obtain a multivariate extension of the results proved in Costarelli and Vinti (2016). For this reason, in this paper, we provide results in this direction.

Indeed, pointwise and uniform approximation results for a family of multivariate NN operators of the max-product type are proved. Moreover, some estimates for the order of approximation are derived.

In general, by means of NN operators we are able to approximate functions defined on bounded domains only, and in the present paper, in order to approximate functions defined on the whole space Rs, a class of quasi-interpolation operators is studied.

Finally, we give a brief description of the plan of the present article. In Section  2, some notations are firstly introduced, then some preliminary results concerning the activation functions used for the max-product NN operators are proved. In Section  3 the NN and quasi-interpolation operators are introduced and studied. In both cases, pointwise and uniform convergence theorem are proved, when continuous functions are approximated. Moreover, estimates concerning the order of approximation are derived in both the previous cases. Finally, in Section  4 many examples of sigmoidal functions for which the previous theory can be applied are shown, while in Section  5 the main conclusions of the paper are pointed out.

Section snippets

Auxiliary results and notations

In the present paper, we will denote by R[a1,b1]××[as,bs]Rs a multivariate box-domain in Rs. By means of the symbols C+(R) and C+(Rs) we will denote the spaces of continuous functions f:RR0+ and f:RsR0+, respectively. Moreover, we denote by UC+(Rs) the space of all uniformly continuous functions f:RsR0+. In all the above settings, we consider the usual sup-norm .

Further, by the symbols and , we will denote respectively the “ceiling” and the “integral part” of a given number.

The multivariate neural network and quasi-interpolation operators of the max-product type

Now, we are able to introduce the operators that we will study in the rest of the paper.

Definition 3.1

Let f:RR be a bounded function, with R[a1,b1]××[as,bs]Rs, and nN+ such that nainbi, i=1,,s. The multivariate max-product neural network (NN) operators activated by σ are defined by: Fn(M)(f,x¯)k¯Jnf(k¯/n)Ψσ(nx¯k¯)k¯JnΨσ(nx¯k¯),x¯R, where Jn denotes the set of indexes k¯Zs, such that naikinbi, i=1,,s, for every nN+ sufficiently large.

In general, when nN+ is sufficiently large, it

Examples of sigmoidal activation functions

In this section, some examples of sigmoidal functions for which the above results can be applied will be shown.

First of all, we present some typical examples of sigmoidal functions used as activation functions in neural network approximation processes. We consider the logistic and hyperbolic tangent sigmoidal functions, defined respectively by: σ(x)(1+ex)1,σh(x)21(tanh(x)+1),xR. Clearly, σ and σh are smooth functions and they satisfy all the assumptions of Section  2. In particular, due

Conclusions

The max-product NN operators represent neuroprocessing models in which the global behavior of the network is mainly determined by one of the artificial neuron of the network.

The multivariate NN operators of the max-product type, allow us to obtain a constructive, nonlinear approximation formula based on a kind of neural networks which allow us to achieve more accurate approximations than the corresponding linear counterparts studied in Costarelli and Spigler (2013).

More precisely, we point out

Acknowledgments

The authors are members of the Gruppo Nazionale per l’Analisi Matematica, la Probabilitá e le loro Applicazioni (GNAMPA) of the Istituto Nazionale di Alta Matematica (INdAM).

The authors of the present paper have been supported by the Department of Mathematics and Computer Science of the University of Perugia (Italy). Moreover, the first author of the paper has been partially supported within the GNAMPA-INdAM Project “Metodi di approssimazione e applicazioni al Signal e Image Processing”;

References (43)

  • V.E. Ismailov

    On the approximation by neural networks with bounded number of neurons in hidden layers

    Journal of Mathematical Analysis and Applications

    (2014)
  • B. Llanas et al.

    Constructive approximate interpolation by neural networks

    Journal of Computational and Applied Mathematics

    (2006)
  • V. Maiorov

    Approximation by neural networks and learning theory

    Journal of Complexity

    (2006)
  • Y. Makovoz

    Uniform approximation by neural networks

    Journal of Approximation Theory

    (1998)
  • R. Schaback et al.

    Construction techniques for highly accurate quasi-interpolation operators

    Journal of Approximation Theory

    (1997)
  • E.D. Sontag

    Feedforward nets for interpolation and classification

    Journal of Computer and System Sciences

    (1992)
  • G.A. Anastassiou
  • G.A. Anastassiou et al.

    Approximation by a nonlinear Cardaliaguet-Euvrard neural network operator of max-product kind

    Journal of Computational Analysis and Applications

    (2010)
  • C. Bardaro et al.

    On pointwise convergence of linear integral operators with homogeneous kernels

    Integral Transforms and Special Functions

    (2008)
  • Bardaro, C., Musielak, J., & Vinti, G. (2003). Nonlinear integral operators and applications, New York, Berlin, De...
  • A.R. Barron

    Universal approximation bounds for superpositions of a sigmoidal function

    IEEE Transaction on Information Theory

    (1993)
  • Cited by (0)

    View full text