Parametric RSigELU: a new trainable activation function for deep learning

This section presents proposed P+RSigELUS trainable activation function. Its formulation.

is given in Eq. (1).

The P+RSigELUS trainable function is active in positive, negative, and linear regions. In the Eq. (1), \({a}^{k}\) represents the positive and negative region slope coefficient, \({x}^{k}\) represents the input of the activation function, and \(f\left(x\right)\) represents the output of the activation function. In addition, \(k-th\) is number of channels. The alpha value in activation functions are determined the \(k-th\) channel by the input signal x and output y. The slope parameter of the P+RSigELUS activation function is trained together with the model parameters during the training process of deep learning models. The proposed P+RSigELUS activation function behaves as shown in Fig. 2a.

The derivative of the proposed P+RSigELUS trainable activation function is given in the Eq. (2).

This section presents proposed P+RSigELUD trainable activation function. Its formulation.

is given in Eq. (3).

The P+RSigELUD trainable function is active in positive, negative and linear regions. In the Eq. (3), \({a}^{k} \mathrm{represents t}\) he positive region slope coefficient, \({\beta }^{k}\) represent the negative region slope coefficient, \({x}^{k}\) represents the input of the activation function, and \(f\left(x\right)\) represents the output of the activation function. The the \({a}^{k}\) and \({\beta }^{k}\) slope parameters of the P+RSigELUD activation function are trained together with the model parameters during the training process of deep learning models. The proposed P+RSigELUD activation function behaves as shown in Fig. 2b.

The derivative of the proposed P+RSigELUD trainable activation function is given in the Eq. (4).

In deep learning architectures, backpropagation architecture is used to update parameters during learning [27]. In addition, it is an important feature that the proposed activation functions can be derivatived. It is seen that the proposed activation function is active on both positive and negative regions after derivative. The additional parameter \({a}^{k}\) and \({\beta }^{k}\) are learned jointly with the whole model using classical gradient-based methods with backpropagation without weight decay to avoid pushing α to zero during the training. \({a}^{k}\) and \({\beta }^{k}\) are parameters learned during the network training. Also, weights are initialized by randomly sampling from a normal distribution with appropriate variance across all layers. Therefore, the value generation range and steepness of the function are not constant like logistic sigmoid and tangent hyperbolic, but vary. While creating the network architecture, architectures are generally created in 2 different ways with this type of free parameter functions. The first is the structures in which all neurons in the network except the last layer have their own training parameters, and the second is the structures in which neurons in each hidden layer have common fixed parameters. In this study, the first method was preferred.

Advantages of the proposed P+RSigELU activation functions, it updates the slope parameters of the activation function during training to obtain the optimal values for each neuron in each iteration. They also overcomes the problems of vanishing gradient and negative region to improve the learning process. In addition, proposed P+RSigELU activation functions provide higher accuracy and faster convergence than fixed parameter ones. In addition, the proposed activation function include parametric, monotonic, and bounded features.

In this study, experimental evaluations were performed on MNIST, CIFAR-10, and CIFAR-100 benchmark datasets [25, 32, 33].

The MNIST dataset consists of 70,000 hand written digits in total [33]. It consists of 10 classes, and each image is 28 × 28 pixels in size. In this study, 60,000 of the images contained in the dataset were used for training and 10,000 of the images were used for testing.

The CIFAR-10 dataset consists of 60,000 32 × 32 colour images in 10 classes, with 6000 **images per class. There are 50,000 training images and 10,000 test images [32].

The CIFAR-100 dataset is a subset of the Tiny Images dataset and consists of 60,000 32 × 32 color images. The 100 classes in the CIFAR-100 are grouped into 20 superclasses [32].

Convolutional neural network (CNN) is widely preferred in many fields such as signal processing, object identification, disease detection, classification [7, 21, 27, 37]. Although it has been used successfully in many fields, it gives better results in image processing [1, 10]. CNN, developed by LeCun inspired by the visual center of animals [33]. CNN architectures consist of convolution, pooling, activation, dropuout, flatten, fully-connected, and softmax layers. The main purpose of CNN is to do more efficient learning by layer-by-layer applying different operations to the input data [25]. In this study, experimental evaluations of the proposed P-RSigELU activation functions were made in VGG-based CNN architecture. The VGG architecture used in the study is shown in Fig. 3 [25].

In Fig. 3, the VGG-based CNN architecture used in the experiments consists of four convolution blocks, a flatten layer, two fully-connected layers, and a softmax layer. To evaluate the proposed P-RSigELU activation functions, the MNIST, CIFAR-10 and CIFAR-100 datasets are first applied to the convolution operation in the VGG-based CNN architecture. The filters number, filter size, and steps number parameters in the convolution process are among the significant parameters chosen by the designers. The number of filters is represented by an odd number along with the filter size. The filter size gives information about how much the determined filter should move on the data in each step. In the convolution process, by circulating the 3 × 3 filter determined on the dataset are provided the discovery of features of data and obtaining the feature map matrix. After the convolution process, the activation functions used in the experimental evaluations are included in the obtained feature map matrix. In this study, PReLU, PELU, ALISA, GELU, P+FELU, PSigmoid, P+RSigELUS, and P+RSigELUD trainable activation functions have been used. In the VGG-based CNN architecture, after activation function block, the normalization method is utilized to adjust for the erratic distribution in the previously produced feature map matrix [11, 25, 26, 38]. Then, the maximum pooling method is applied to the feature map obtained with the values of pool = 2 and stride = 2. By halving the size of the feature map during the maximum pooling process, the model performs better and avoids delivering inaccurate rote results. In order to prevent over-learning after the pooling process, the dropout layer was applied as 0.25. The flatten layer is applied after the convolution blocks to obtain the feature vector suitable for the fully-connected layer. The fully-connected layer architecture is similar to the artificial neural network (ANN) architecture. Therefore, the obtained feature map matrix is performed classification by transforming into vector format. After these processes, a fully connected layer with 512 neurons is applied. Thanks to the fully connected layer, all units are connected to each other and the classification process is performed. Finally, the logistic regression-based softmax layer is used as the output function [3, 4, 12, 13].

The VGG-based CNN architecture used in the study consists of four CNN blocks in total. The first CNN block consists of two 32@3 × 3 filters, the second CNN block consists of two 48@3 × 3 filters, the third CNN block consists of three 64@3 × 3, and the last CNN block consists of 96@3 × 3 filters. Following the CNN blocks, experiments are carried out on the VGG architecture by connecting two 512-neuron fully-connected layers.

In the experiments, for comparison, PReLU, PELU, GELU, P+FELU, PSigmoid and ALISA activation functions were used. Experimental evaluations of the proposed and other activation functions on the MNIST dataset are presented in Tables 1, 2, and 3. Tables shows number of epochs, training loss (Train_Loss), training accuracy (Train_Acc.), validation loss (Val._Loss), and validation accuracy (Val._Acc.) for number of repeats or diffrent activation functions. In the experimental evaluations, datasets were separated as training and testing as stated in Sect. 4.1. Computationally, the training loss is calculated by taking the sum of errors for each example in the training set. The accuracy score in machine learning is a measurement statistic that compares the proportion of accurate predictions made by a model to all predictions made. We determine it by dividing the total number of forecasts by the number of correct guesses. In deep learning, metrics like accuracy, and loss score are frequently used to assess the performance of models. Accuracy: This is the ratio of total forecasts made to total predictions made correctly. The training loss is a metric used to assess how a deep learning model fits the training data. In the loss metric (in Eq. (6)), the \(m\) parameter returns the number of training samples, the \(i\) parameter returns the training example in a dataset, and the \({y}_{i}\) parameter returns the actual value for the ith training sample. It’s a straightforward statistic that provides a general sense of how well a model is doing. Each metric has a mathematical formula that is given in the following equations:

Tables 1, 2, and 3 present, the experimental results performed by repeating 5 times on the MNIST dataset. In the experimental results, it can be seen that the best performance value was obtained by the double parameter P+RSigELUD trainable activation function. In experimental evaluations, it was observed that an average of 0.9564 validation accuracy and 0.1458 validation loss values were obtained with P+RSigELUD. In addition to, for the single parameter P+RSigELUS trainable activation function, an average of 0.9471 validation accuracy and 0.1805 validation loss values were obtained. Also, it was observed that the lowest success was achieved with an average of 0.9335 validation accuracy and 0.2275 validation loss by ALISA activation function. In addition, the proposed activation function seems to work faster than other activation functions during the training process.

Figure 4 shows the average validation accuracy values (Fig. 4a) and validation loss values (Fig. 4b) of the activation function results for the MNIST dataset. The proposed P+RSigELUS and P+RSigELUD trainable activation functions gave the best average performance values with respect to the other activation functions. Also, as can be seen in Fig. 4a, the average val_accuracy value of the proposed P+RSigELUD activation function is bigger than that of PReLU, PELU, ALISA, GELU, P+FELU, PSigmoid and P+RSigELUS activation funcitons. In addition, the average accuracy performances are between 0.9380 and 0.9558 for P+RSigELUS and 0.9453 and 0.9674 for P+RSigELUD. The average loss performances are between 0.1475 and 0.2160 for P+RSigELUS and 0.1038 and 0.1923 for P+RSigELUD. In Fig. 4a, the lowest values of box plot of P+RSigELUS and P+RSigELUD are above that of the rest of the activation functions. Thus, it is observed that the P+RSigELUD activation function gives better results than the others. In Fig. 4, the loss values of the P+RSigELUD activation function are among the lowest values in the box plot compared to other box plots. In this case, when the box plots found in Fig. 4a, b are examined, it can be seen that the activation function of P+RSigELUD gives more consistent results than the other activation functions. Also, the lengths of the whiskers to the box plot are close and the median value is close to the middle of the box plot.

Experimental evaluations of the proposed and other activation functions on the CIFAR-10 dataset are presented in Tables 4, 5, and 6.

Tables 4, 5, and 6 present the experimental results performed on the CIFAR-10 dataset. In the experimental results, it can be seen that the best performance value was obtained by the double parameter P+RSigELUD trainable activation function. In experimental evaluations, it was observed that an average of 0.8212 validation accuracy and 0.6616 validation loss values were obtained with P+RSigELUD. In addition to, for the single parameter P+RSigELUS trainable activation function, an average of 0.8143 validation accuracy and 0.6839 validation loss values were obtained. Also, it was observed that the lowest success was achieved with an average of 0.7762 validation accuracy and 0.8135 validation loss by ALISA activation function. In addition, the proposed activation function seems to work faster than other activation functions during the training process.

Figure 5 shows the average validation accuracy values (Fig. 5a) and validation loss values (Fig. 5b) of the activation function results for the CIFAR-10 dataset. The proposed P+RSigELUS and P+RSigELUD trainable activation functions give the best validation accuracy and validation loss performances from other activation functions. Also, the validation accuracy values are are between 0.8056 and 0.8209 for P+RSigELUS and between 0.8136 and 0.8314 for P+RSigELUD. In addition, the validation loss values are between 0.6553 and 0.7062 for P+RSigELUS and 0.6314 and 0.6999 for P+RSigELUD. The highest values of boxes of P+RSigELUD are above that the of rest of the activation functions. In Fig. 5b, the lowest loss values of the activation functions other than ALISA and P+FELU activation functions are close to each other. In Fig. 5a, b, it can be seen that the ALISA and P+FELU activation functions have poor success compared to other activation functions and there is an inconsistency between the results. In addition, it can be seen that the proposed P+RSigELUS activation function gives more consistent results with a smaller box size than the others. Also, the lengths of the whiskers to the box plot are close and the median value is close to the middle of the box plot. When Figs. 5, 6 are examined, it is observed that the activation functions of PReLU and PELU behave like the proposed functions. However, when the results of ALISA and P+FELU activation functions are compared, too much fluctuation is observed. The reason for this is that there are too many fluctuations and inconsistencies because not most activation functions works in accordance with both the deep learning architecture and the dataset.Experimental evaluations of the proposed and other activation functions on the CIFAR-100 dataset are presented in Tables 7, 8, and 9.

Tables 7, 8, and 9 present the experimental results performed on the CIFAR-100 dataset. In the experimental results, it can be seen that the best performance value was obtained by the double parameter P+RSigELUD trainable activation function. The average performance results of the P+RSigELUD activation function were obtained of 0.5462 validation accuracy and 1.9888 validation loss. In addition, for the single parameter P+RSigELUS, an average of 0.5396 validation accuracy and 1.9952 validation loss values were obtained. Also, it was observed that the lowest success was achieved with an average of 0.4861 validation accuracy and 2.2368 validation loss by ALISA activation function. In addition, the proposed activation function seems to work faster than other activation functions during the training process.

Figure 6 shows the average validation accuracy values (Fig. 6a) and validation loss values (Fig. 6b) of the activation function results for the CIFAR-100 dataset. The proposed P+RSigELUS and P+RSigELUD trainable activation functions gave the best average validation accuracy and validation loss performances. The validation accuracy values are between 0.5329 and 0.5467 for P+RSigELUS and between 0.5404 and 0.5524 for P+RSigELUD. In addition, the validation loss values are between 1.9413 and 2.0330 for P+RSigELUS and between 1.9586 and 2.0193 for P+RSigELUD. In Fig. 6a, the lowest box plot values of P+RSigELUS and P+RSigELUD are above the that of ALISA and PELU activation functions and close to that of the rest of the activation functions. Also, as can be seen in Fig. 6a, b, the box sizes of the suggested activation functions are smaller than that of the others. Thus, it is observed that the proposed activation functions offer more successful and consistent results than the others. In addition, the lengths of the whiskers to the box plot are close and the median value is close to the middle of the box plot. When Table 6 and 9 are examined, it is observed validation accuracy of the proposed model is better than the other models and so during the evaluation process, interpretation was made by taking into account the validation values.

In this study, the P-RSigELU trainable activation functions are proposed to improve the learning process of deep learning models. Trainable activation functions update the slope parameters of the activation function to obtain the most appropriate values for each neuron during training. Thus, the weights suitable for each neuron are constantly updated by the back propagation algorithm, ensuring that the training process is continuous. The proposed P+RSigELUS and P+RSigELUD trainable activation functions gave the best results in experimental results for MNIST, CIFAR10, and CIFAR-100 datasets as shown in Table 3, 6, and 9, respectively, when compared to the existing trainable activation functions. In this study, the proposed P+RSigELUS and P+RSigELUD trainable activation functions are effective in the positive, negative, and linear activation regions. Also, they can overcome the problems of bias shift, negative region, and vanishing gradient successfully. The curve slope is consistent with the PSigELU, which ensures that the new activation function does not change the accuracy advantage of the RSigELU. In addition, the lengths of the whiskers to the box plot are close and the median value is close to the middle of the box plot. The lower accuracy value of box plots of the proposed P+RSigELU is close to the box than that of the other activation function and upper accuracy values of the proposed P+RSigELU are close to the highest. The proposed P+RSigELU trainable activation functions provide higher accuracy and faster convergence than fixed-parameter activation functions. In addition, when the proposed activation function is compared with the P+RSigELU activation functions, it is seen that it does not complete the training process in a shorter time in all three data sets. All deep learning architectures has different characteristics and therefore like the activation functions proposed in the literature, P+RSigELU may not have consistent result values for all architectures. In addition, the proposed activation function include parametric, monotonic, and bounded features. However, it does not have the smooth feature. In addition, it is observed that the proposed P+RSigELU activation functions gives consistent and stable results on the datasets in the experiments. In addition, it should be noted that the activation functions in the literature do not always give consistent results in all data sets.