“SPOCU”: scaled polynomial constant unit activation function

Neural computing and activations in neural networks are multidisciplinary topics, which range from neuroscience to theoretical statistical physics. Also, importantly enough, one of the most important theoretical and practical topics in neural networks and neural computing is choice of the appropriate activation function. Activation function and its nonlinear ability represent core of the neural networks, both deep and shallow, with various architectures. In standard problems with a reasonable nonlinearity, the sigmoidal function is well justified; however, one faces the inverse problem of specification of the underlying parameters; otherwise, network may perform slowly. Recently, several papers related to making a more adequate activation function have appeared, mainly comparing activation functions on large datasets. To illustrate some of important contributions, the influence of the activation function in the convolutional neural network (CNN) model is studied in [24], improving a ReLU activation by construction of a novel surrogate. Theoretical analysis about gradient instability as well as the fundamental explanation for the exploding/vanishing gradient and the performances of different activation functions are given in [13].

The main contribution of this paper is the construction and testing of novel scaled polynomial constant unit (SPOCU) activation function. Such a novel activation function relates to complexity patterns through phenomenon of percolation, and thus, it can overcome already introduced activation functions, e.g., SELU and ReLU. In statistical physics and mathematics, percolation theory describes the behavior of a network when nodes or links are removed, or complex patterns are embedded to learning process. Thus, SPOCU well contributes to fill the gap in the theories and it is “picking up” the appropriate properties for activation function directly from training of classification on complex patterns, e.g., cancer images. Such efforts can contribute to many applications. One example is the generation of the artificial networks applied in the field of the Touring pattern generators networks, since they require the two-layer coupling (two “diffusion” mechanisms having different diffusion coefficients) of resistor couplings. Turing patterns are nonlinear phenomena which appear in the reaction–diffusion systems based on the memristive cell. For Turing patterns in the simplest memristive cellular nonlinear networks (MCNNs), see [2], where it is proposed a new MCNN model consisting in a two-dimensional array of cells made by only two components: a linear passive capacitor and a nonlinear active memristor. In fact, cellular nonlinear networks are perfectly suited to fit the structure of reaction–diffusion systems, since they can be used to map partial differential equations [8]. MCNNs are intrinsically related to percolation; link to this relation is outlined in Sect. 3; for more, see, e.g., Chapter 18 in [6].

The paper is organized as follows. In Sect. 2.1, we introduce random fractal construction useful for our purposes. We give an overview of the problem of generating of random type of Sierpiński carpet, related to the [pppq] model introduced by [9]. We emphasize importance of using Kronecker product, which gives a simple and quick possibility to generate random fractals. We also present several useful results concerning fractal geometry and “average” fractal dimension. In Sect. 3, we deal with percolation threshold, which is an important instrument for SPOCU. By using of direct approach and logistic regression, we derive estimations for this critical value in the case of specific random models. In Sect. 4, we use basic generator from random Sierpiński carpet as a new activation function SPOCU for self-normalizing neural network. We study general activation functions, but with main focus on SPOCU. As normalizing the output of layers is known to be a very efficient way to improve the performance of neural networks, we can conclude that SPOCU activation function behaves very well. Further we provide theoretical justification of the fact that SPOCU outperforms both SELU and ReLU in several desirable properties, necessary for the correct classification of complex images. The SPOCU also gets uniformly better performance with respect to generic properties on large MNIST database. This is well illustrated in the last Sect. 6 on image-based discrimination for cancer diagnostics. Therein we apply the developed methodology to cancer discrimination problems; namely, we consider image-based discrimination for mammary cancer versus mastopathy and benign prostatic hyperplasia or normal prostate versus prostate cancer. A comparison of SPOCU, ReLU and SELU on the Wisconsin Diagnostic Breast Cancer (WDBC) dataset justifies SPOCU qualities. Technicalities and proofs are given in "Appendix A."

Many of fractal constructs have random analogues; see Chapter 15 in [6]. Here, we start with the natural motivation of the random Sierpiński carpet (SC). We introduce simple computation of matrix expressing the form of fractals, including random SC.

Percolation itself is a physically well-motivated phenomenon and it is intrinsically related to memristor models (see [22]); thus, it can be well involved in neural modeling of memristive cellular nonlinear networks (MCNNs). The combination of percolation theory and Monte Carlo simulation provides a possible solution to model the ion migration and electron transport in an amorphous system even from the hardware perspective. The most known mathematical percolation model is to take some regular lattice, e.g., a square lattice \(N\times N\), and make it into a random network by randomly “occupying” sites (vertices) or bonds (edges) with a statistically independent random variables. For example, each of the lattice sites is either occupied (with probability p) or vacant (with probability \(1-p\)). This is commonly known as the site percolation problem. There exists also a related bond percolation problem. It can be posed in terms of whether or not the edges between neighboring sites are open or closed. It is well known that there is a well-defined range of p for which the probability of percolation decreases rapidly from one to zero. It is centered on a critical value \(p_c\) called percolation threshold. Percolation clusters become self-similar precisely at the threshold density \(p_c\) for sufficiently large length scales, entailing the following asymptotic power laws: \(M(L) \sim L^{d_\text {f}}\) at \(p=p_{c}\) and for large probe sizes, \(L\rightarrow \infty\), i.e., fractal dimension \(d_\text {f}\) (e.g., Hausdorff dimension \(\dim _H\)) describes the scaling of the mass M of a critical cluster within a distance L (it characterizes how the mass M(L) changes with the linear size L of the system); see [21]. This follows from the following idea: If we consider a smaller part of a system of linear size \(bL ~(b < 1)\), then M(bL) is decreased by a factor of \(b^d\), i.e., \(M(bL) = b^d M(L)\). For example, for Sierpinski Gasket we have functional equation \(M(L/2)=M(L)/3\) yielding solution \(M(L)=AL^d_\text {f}\) with \(d_\text {f}=\ln (3)/\ln (2).\)

Depending on the method for obtaining the random network, usually one distinguishes between the site percolation threshold and the bond percolation threshold. More general systems may have several probabilities \(p_1\), \(p_2\), etc., and the transition is characterized by a critical surface or a manifold. In the classical systems, it is assumed that the occupation of a site or bond is completely random. This is the so-called Bernoulli percolation. Here, we want to emphasize that random SC does not fall under them. In 1974, Mandelbrot [14] introduced a process in \([0, 1]^2\) which he called “canonical curdling.” It is nothing else than the model from Remark "Appendix A.1" with \(\mathcal {B}_{p_{ij,n}}=\mathcal {B}_{p}.\) In paper [3], the authors study the connectivity or “percolation” properties of such sets. They showed there is a probability \(p_c\in (0, 1)\) so that if \(p <p_c\) then the set is “dustlike,” whereas if \(p\ge p_c\) opposing sides are connected with positive probability. To be precise, we introduce here the exact definition of \(p_c\). Let\(B_\infty =\bigcap _{n\in \mathbb {N}} B_n\) and let \(\Omega =\{B_\infty \ne \emptyset \}\). Notice that when \(\Omega\) occurs there is a up-to-down crossing of \([0, 1]^2\). Finally, \(p_c=\inf _p\{\mathbb {P}(\Omega )\}\). For example, from [3] we have that Mandelbrot percolation is \(p_c<0.9999\). Here, we have to emphasize that for approximation of \(A_\infty\) we cannot use this kind of definition. We just set \(p_c\) as the \(\frac{1}{2}\) threshold of probability \(\mathbb {P}(B_n=\emptyset )\).

Self-normalizing neural networks (SNNs) are expected to be robust to perturbations and not to have high variances in their training errors. SNNs push neuron activations to zero mean and unit variance, thereby leading to the same effect as batch normalization, which enables to learn many layers in a robust way. Here, we introduce our SNNs based on percolation function (5).

For a neural network with activation function Ac, we consider two consecutive layers connected by a weight matrix \(\mathbf{W}\). We assume that all activations \(x_i\) of the lower layer have the same mean \(\mathbb {E}[x_i]=\mu\) and variance \(\mathbb {V}[x_i]=\sigma ^2\) and are mutually independent. A single activation \(y=f(z)\) in the higher layer has network input \(z=\mathbf{w}^T\mathbf{x}\), mean \(\mathbb {E}[y]=\tilde{\mu }\) and variance \(\mathbb {V}[y]=\tilde{\sigma }^2\). From this, we can obtain \(\mathbb {E}[z]=\sum _{i=1}^n w_i\mathbb {E}[x_i]=\mu \,\sum _{i=1}^n w_i:=\mu \,\omega\) and \(\mathbb {V}[z]=\sum _{i=1}^n w_i^2\mathbb {V}[x_i]=\sigma ^2\,\sum _{i=1}^n w_i:=\sigma ^2\,\tau ^2.\) Central limit theorem implies under the regularity conditions that \(z\sim \mathcal {N}(\mu \,\omega ,\sigma ^2\tau ^2)\), i.e., the pdf of z has the form \({\displaystyle f(z)={\frac{1}{\sqrt{2\pi \sigma ^{2}\tau ^2}}}\mathrm {e}^{-{\frac{(z-\mu \,\omega )^{2}}{2\sigma ^{2}\tau ^2}}}}\). Consider now vector mapping \(\mathbf{g}\) that maps mean and variance of the activations from one layer to mean and variance of the activations in the next layer, i.e., \(g_1(\mu ,\sigma ^2)=\tilde{\mu }\) and \(g_2(\mu ,\sigma ^2)=\tilde{\sigma }^2\). The following definition recalls a self-normalizing neural network.

For arbitrary activation function Ac(z), the mapping \(\mathbf{g}\) is given by the relationsandObviously moments are given by integrals \(\mathbb {E}_f[Ac(z;\mathbf{a})]=\int _\mathbb {R}f(z)\,Ac(z;\mathbf{a})\,\mathrm {d}z\) and \(\mathbb {E}_f[Ac^2(z;\mathbf{a})]=\int _\mathbb {R}f(z)\,Ac^2(z;\mathbf{a})\,\mathrm {d}z\). [11] proposed \(\omega = 0\) and \(\tau ^2 = 1\) for all units in the higher layer for the weight initialization. If the Jacobian of \(\mathbf{g}\) has a norm smaller than 1 at the fixed point, then it is a contraction mapping and the fixed point is stable. The goal is to find parameters \(\mathbf{a}\in A\) such that fixed point [0, 1] does exist and that \(||\mathbf {J} _{\mathbf {f}}[0,1]||<1.\) This problem is formulated as to find \(\mathbf{a}\in A\) such that\(||\mathbf {J}||<1\), where (here we omit parameters for the sake of short notation)which can be simplified toThus, we can formulate the following problem. For a given activation function Ac, find \(\mathbf{a}\in A\) such that Eq. (10) and \(||\mathcal {J}||<1\) hold. The next theorem follows directly from the Hölder inequality for \(p=1\) and \(q=\infty .\) It gives sufficient condition for a stable and attracting fixed point [0, 1].

Here, we define “scaled polynomial constant unit” (SPOCU) as the novel, well-motivated activation function. We also study its properties. The SPOCU activation function is given bywhere \(\beta \in (0,1), ~\alpha , ~\gamma >0\) andwith \(r(x)=x^3(x^5-2x^4+2)\) and \(1\le c<\infty\). (We admit c goes to infinity with \(r(c)\rightarrow \infty\).) Clearly s is continuous \(s(0)=0\) and \(s'(x)=\frac{\alpha }{\gamma }\,h'\left( \frac{x}{\gamma }+\beta \right)\). Notice that for \(c=1\) one has \(h'(1^+)=h'(1^-)=0\) and \(h'(0^+)=h'(0^-)=0\) which implies that \(s'\) is continuous too. (This is not true for the second derivative.) For \(c=1\), the range of function s is \(H_s=[s(-\beta \,\gamma ),s((1-\beta )\,\gamma )]=[-\alpha \,r(\beta ),\alpha (1-r(\beta ))], ~r(\beta )\in [0,1]\).

Developing of the algorithmic methods which can assist in cancer risk assessment is an important topic nowadays. Discrimination between mammary cancer and mastopathy tissues plays a crucial role in a clinical practice; see [9]. Noninvasive techniques may produce generally an inverse problems, e.g., estimating a Hausdorff fractal dimension from boundary of examined tissue; see [10]. Main problem here can be formulated as follows: “How can be cancer tissue discriminated from healthy tissue?”

Here, we study benign prostatic hyperplasia (BPH) and normal prostate vs. prostate cancer (PC). We consider the standard coloring of images, by hematoxylin and eosin, and we consider two magnifications, namely \(100\times\) and \(200\times\) of images; see Figs. 13. Moreover, carcinoma of the breast and mastopathy are given in Fig. 14.

Here, we have used simple estimators, i.e., reduced estimators \(\mathcal {N}(k)=(k^3p)^n\), expressing retained elements in average, of \(k^{3n}\tilde{p}_n\) with \(k=2\) for standard SC and \(k=3\) for modified (9 elements instead of 8) from Sect. 2.2. This implies \(\hat{p}=N^\frac{1}{n}/3^k\), where N is the number of 1’s in measured matrix of data. Since we set the resolution of the figures to \(729\times 729\) and \(729=3^6\), we have \(n=6\). In our case, modified SC is suitable, since otherwise data implies estimation of p over 1. We had to use only two values—binarization, i.e., each dark pixel was converted to 1, and if it is clear, it was converted to 0. If the picture is not in black and white, it was converted according to the formula \((R+G+B)/3>0.5.\) We see the results in Table 4. One can see statistically significant difference between cancer and noncancer images, whether we take probability parameter p or fractal dimension \(\dim _H\). Moreover, we can also see that the package fractaldim cannot catch these differences. This makes this result very valuable.

Notice that we cannot use directly more parameters without obtain more information than resulting matrix. This is quite interesting since it seems to happen that one parameter is not enough, i.e., dimension of the problem is more than one. We have also estimated values for \([p\cdot q]\) model based on theory of [9] (notice that 0 and 1 has opposite meaning in their work). The obtained results confirmed the same. We illustrate this for both CA and MA; we obtained estimations \(\hat{p}=0.24447\) and \(\hat{q}=0.11515\) for CA and \(\hat{p}=0.1934\) and \(\hat{q}=0.09675\) for MA. An alternative from the perspective of inter-patient variability can be a multifractality (see [15]). Development of such kind of techniques for analysis of several slices from 3D tissue body will be of interest for complicated cases of the National Institutes of Health (NIH) databases, like [18]. This will be a valuable future research direction.

We introduced novel percolation-based activation function SPOCU which is flexible, and in several important setups, it overcame classical SELU or ReLU approaches. We successfully validated SPOCU on both large and small datasets, including Wisconsin Diagnostic Breast Cancer (WDBC) dataset and large dataset MNIST. We also provided careful theoretical comparisons of SPOCU to SELU or ReLU competitors.