Memetic algorithms for training feedforward neural networks: an approach based on gravitational search algorithm

Neural networks are one of the most popular and successful machine learning techniques. The aim of machine learning techniques is to provide algorithms with the goal of allowing the computer to learn automatically without human intervention, and to adjust actions accordingly. Specifically, FNNs are the best known neural network architecture (see [33]), with the layers interconnected in such a way that the output of ith layer is used as the input of the \(i + 1\)th layer. Most commercial applications which use neural networks implement this kind of architecture, since it has been shown to be suitable for addressing a large class of problems pertaining to pattern recognition or prediction (see [8, 41, 42, 67]). Moreover, an FNN with only one hidden layer is considered a universal approximator (see [31]) as it can approximate any continuous function.

In neural networks, the learning process takes place through the adjustment of the weights and biases which measure the strength of the connection between neurons from different layers. These weights are usually updated in order to minimize the error made by the network, which is formulated using a loss function, like mean square error (MSE), sum of square errors (SSE) or root mean square error (RMSE). The process of adjusting the weights and biases for a given network is known as training. Therefore, from the mathematical point of view, behind a neural network lies the solution of an unconstrained optimization problem.

In the past, gradient-based techniques were the most widely used for FNN optimization (see [32]). These algorithms use a specialized method for computing the gradient of the loss function with respect to the parameters of the network, known as the backpropagation technique. The most representative algorithm of this class is the steepest descent approach known as the BP algorithm (see [21]). This algorithm is based on the idea of propagating the errors made by the network to the back layers. This identifies how the error varies with respect to the network parameters and adjusts the weights and biases in order to minimize the loss function.

Training an FNN using a BP algorithm is difficult when the problems are non-differentiable or multimodal. This is because gradient-based algorithms are liable to becoming trapped local minima (see [26]), making their performance highly dependent on the initial values of weights, biases and the chosen hyperparameters of the optimization algorithm. Furthermore, the vanishing gradient problem becomes a major concern when many hidden layers are added to the network. These facts motivate the challenge of finding new approaches and algorithms which outperform the BP in these scenarios. In order to address these problems, metaheuristic algorithms have been successfully applied to training FNNs (see [12, 52, 53]). Metaheuristics improve the exploration capacity of gradient-based algorithms, but they exhibit slow convergence in comparison with gradient-based approaches, which achieve convergence at a (super-)linear rate (see [4]).

In the last few years, MAs have appeared as a new computation paradigm which tries to combine metaheuristic algorithms with one or more local search procedures in order to combine the advantages of both approaches (see [47, 50]). MAs have been successfully applied in different domains such as the train timetabling problems (see [14]) and segmentation of temporal series (see [40]).

Recently, [20] have proposed two memetic algorithms for unconstrained optimization based on the hybridization of the GSA (see [60]) or CGSA (see [44]) with quasi-Newton (qN) search directions. The resulting algorithms are named MGSA and MCGSA. In this previous study, the authors proved these memetic algorithms can be considered as part of the state of the art, since they outperformed metaheuristic algorithms from the state of the art in a wide set of synthetic and real-world global optimization problems. This paper studies the application of both memetic approaches to a challenging machine learning problem, specifically the training of an FNN. Both approaches are particularly suited to training a FNN as they consider a qN algorithm with a superlinear convergence rate and they include a promising evolutionary algorithm, so as not to become trapped in local minima. The main advantage of using a metaheuristic algorithm instead of BP is that metaheuristic algorithms are able to escape from local optima, while BP may be trapped in local minima. However, the exploitation capabilities of BP improve upon those of metaheuristic algorithms significantly. This situation motivates the use of the memetic approaches based on gradient information, which combine the advantages of BP and metaheuristic algorithms. Thus, the main advantage of using a memetic approach is a greater rate of convergence than with metaheuristic algorithms and the ability to find a global optimum of the problem, unlike BP. In order to do this, the implementation of these algorithms has been adapted to this problem and the results obtained are analyzed in order to compare the performance of memetic approaches to that of other metaheuristic algorithms from the state of the art.

Furthermore, [45] addressed the training problem of FNN using GSA, PSO and a hybrid of GSA with PSO. This study replicates their computational experiment in order to assess the performance of MGSA and MCGSA in training FNNs. Furthermore, a statistical comparison of both approaches with metaheuristic algorithms from the state of the art was carried out. To do this, PSO and GA were chosen as the traditional algorithms, while Rcr-JADE (see [24]) and COBIDE (see [69]) were selected as recent proposed algorithms, since these variants of the differential evolution (DE) algorithm have been widely used in recent years and give the best results in rankings (see [57]). The results obtained show MCGSA improves statistically, in terms of convergence and speed, on the results obtained by the metaheuristics from the state of the art when training a FNN.

The rest of the article is structured as follows: Sect. 2 gives a brief review of GSA applications in neural networks; Sect. 3 defines and formalizes the neural network training problem. Then, Sect. 4 describes GSA, CGSA and the memetic algorithms used in this paper. Section 5 shows the results of experiments carried out in this study, and finally, Sect.  6 summarizes the conclusions and further work derived from this article.

The success of metaheuristics and evolutionary algorithms in the field of optimization is well known. They are approximate and non-deterministic optimization methods which incorporate mechanisms to escape from local optima in order to reach the global optimum. The popularity of these methods is due, among other things, to their versatility, simple coding, the few hypotheses to be fulfilled and the fact they do not use derivatives. The applications of these algorithms cover many, if not all, fields of engineering.

Some recent applications are related to hydrology, such as [9] where an adaptive-network-based fuzzy inference system (ANFIS) model is designed for long-term prediction of discharges by the Manwan Hydropower Plant into the Lancangjiang River, in order to manage and schedule the hydroelectric reservoir. Subsequently, [68] proposed a neural network model coupled with the ensemble empirical mode decomposition (EEMD) designed for medium and long-term forecasting in hydrological time series. Hybrid algorithms have also been applied in this field, for example in [46], where a hybrid firefly algorithm with support vector regression is applied to predict evaporation with the purpose of managing water resources and [72], where an enhance extreme learning machine (EELM) is used to forecast river flow in the Kelantan River (Malaysia).

Regarding other fields of science and engineering, recent relevant applications can also be found. For example, in chemistry, [48] have recently proposed approaches based on neural networks, ANFIS and response surface methods to estimate and optimize the main parameters which affect the yield and cost of biodiesel production, and found that neural networks with radial basis functions give the maximum biodiesel yield production with the minimum production cost. In another field, [17] presented a survey of computational intelligence techniques applied to address severe flood management, identifying neural networks, soft computing techniques, decision trees, fuzzy logic and hybrid approaches. Finally, other recent applications in the field of power systems are [1] where a hybrid algorithm based on PSO and the bacterial foraging optimization algorithm (BFOA) is applied to power system stabilizer design, and [2] where the same authors used GSA to design a static synchronous series compensator for single and multi-machine power systems.

Metaheuristics and evolutionary algorithms in the area of neural networks have been applied to optimize the FNN components of: (i) connection weights, i.e., to find the optimal combination of weights and biases which provides the minimum error while keeping the other components fixed at their initial setting; (ii) architecture, i.e., the metaheuristic is used to search for optimum architecture from a compact space of FNN topology; and (iii) hyperparameters such as the learning rate in the BP (remember that the efficiency of these algorithms is highly conditioned by these values). The first two areas have attracted the attention of the research community, with many studies appearing related to these fields. The aim of this section is to review the studies and applications of GSA in the field of neural networks, paying special attention to groups (i) and (ii).

This section describes and formalizes the optimization problem which arises from the training of a FNN. In a FNN, also known as multilayer perceptron (MLP), the network topology is composed by the input layer, a set of hidden layers and, finally, the output layer. The connections between the neurons from different layers are always forward, and usually, all the neurons of one layer are connected to all neurons from the next layer. A general scheme of a FNN or MLP is shown in Fig. 1. In the figure, \(n_j\) is the number of neurons in the jth layer and the parameter c denotes the total number of layers in the architecture.

There are two main steps in the training process of a FNN. First, the inputs of the network are propagated forward in order to obtain an output. Next, the error between the output produced by the network and the desirable value is computed. Finally, in a second step, the errors are propagated backward, adjusting the weights and thresholds for each neuron in the topology in order to minimize an error or loss function (defined by the designer). This last step will be carried out by using an optimization method.

We consider a canonical FNN with fully connected layers. The first is the input layer, the following layers are hidden layers and the last one is the output layer. The sizes of the layers are \(n_j\) for \(j=1,\ldots ,c\). Let \(W_j \in {\mathbb {R}}^{n_j\times n_{j-1}}\) be the weight matrix associated with the connections from layer \(j-1\) to layer j, and let \(b_j\in {\mathbb {R}}^{n_j}\) be the threshold vector of the neurons from layer j for layers \(j =1,\ldots ,c\). The process of propagating the inputs of the network forward is the following: This process in which the neural network obtains the output vector \(o_i\) for an input vector \(x_i\), where the parametrization of the network \({\varvec{\omega }}=\{W_j,b_j\}_{j=1}^c\) is known, can be schematically represented by \(o_i(x_i, {\varvec{\omega }})\). Without loss of generality, it can be considered that \({{\varvec{\omega }}}\) is a parameter vector \({\varvec{\omega }} \in {\mathbb {R}}^n\).

The training problem consists of determining the parameter vector \({\varvec{\omega }}\) and involves a training dataset \(\{(x_1,y_1),\ldots ,(x_N,y_N)\}\) and the choice of a loss function \(\ell \), obtaining the resulting optimization problem:A popular choice for the loss function \(\ell \) is MSEwhere \(\Vert \cdot \Vert _2\) is the Euclidean norm. Hence, Eq.  (6) is the objective function of the problem, which can be formalized through Eq. (7).Therefore, and in accordance with the prior formalization, the training problem of a FNN (without specifying a loss function) can be stated asA small example is shown to illustrate the encoding strategy for the optimization algorithms. Given the fully connected FNN shown in Fig. 2 with three layers where \(n_0 = 3, n_1 = 4, n_2 = 2\), a parametrization would be encoded using Eq.  (9).

We use the vector \({\varvec{\omega }}\) to represent the parametrization of the networkIn population-based metaheuristics, each individual of the population is represented by a vector \({\varvec{\omega }}\) which includes the weights and biases of the different layers of the network. At the end of the optimization process, the best individual in the population will be used to set the parameters of the neural network.

This section describes the MGSA and MCGSA training algorithms FNNs. Firstly, the GSA, the CGSA and the qN algorithms are reviewed. Then, the two memetic proposals MGSA and MCGSA are presented.

This section will describe the experiments carried out in this paper in order to study the performance of memetic algorithms based on GSA for training FNN. This was done by replicating the experiments conducted in [45] under the same conditions. These experiments are intended to study if the use of MAs for training FNNs can be a suitable choice in comparison with the current state-of-the-art metaheuristic algorithms. In [45], a hybrid of PSO and GSA was used to train a FNN for three different benchmark problems, showing that the proposed hybrid algorithm outperforms individual GSA and individual PSO.

These authors chose a sample of three classical, well-known benchmark problems where neural networks are used: The first is a three-bit parity (XOR) problem, the second is a function approximation problem and the last one is a classification problem. These problems have been chosen because they are classic benchmarks widely used by the scientific community when a new method for training FNN is designed. Furthermore, this set of problems covers the spectrum of problems that FNN solves: function approximation, binary classification and multi-classification. Another advantage of choosing these problems is the possibility of replicating the experiments of [45] under the same conditions, in order to compare the results obtained with those obtained by the hybrid GSA of these authors. Currently, an alternative to classic benchmarks problems is training deep neural networks as test problems. In these problems, backpropagation has been proved to be inefficient for networks with two or more hidden layers, due to the vanishing gradient problem, poor generalization and the possibility of becoming trapped in a local optimum (see [23, 37, 38]). In this scenario, the challenge is to find alternative methods to deal with these problems. However, the main focus of this paper is on testing hybrid methods to train FNN.

To solve these problems, different algorithms, both classical and from the state of the art of metaheuristics, were chosen to train the FNNs. These are the following: the GSA proposed in [60], the CGSA with a sinusoidal map proposed in [44], and the MGSA and MCGSA described in Sect. 4. These memetic algorithms have been run using, in all trials, a basic configuration (\(\varepsilon =0, \gamma = 1, \texttt {tol} = 0.01\)) in order to ensure that the performance of these proposed algorithms is robust with respect to their parametrization, and the parameter tuning is not an inconvenience in their application. The choice of \(\varepsilon \) and \(\gamma \) means that the qN method is applied each time a mass of the population finds a better solution, especially when it manages to escape from a local minimum. Of course, an optimal tuning will guarantee better results. Also, the PSO algorithm implemented by the particleswarm MATLAB function was used as in [34, 45, 75]. The genetic algorithm implemented in MATLAB by the ga function has also been considered in this study in the same way as [34, 62, 65, 73]. Both PSO and GA have been chosen since they are classical metaheuristic algorithms used to train FNNs. Furthermore, in order to compare the memetic algorithms with the state of the art, two algorithms based on the differential evolution (DE) algorithm have been considered in this experiment (see [57]): The first is Rcr-JADE (see [24]). This algorithm modifies the adaptation rule used in the crossover. Also, the adaptation methods used in the control parameters of Rcr-JADE have also been applied in other versions of DE (see [25, 27, 64]). The second algorithm is COBIDE (see [69]). Their authors proposed a new DE variant based on covariance matrix learning and bimodal distribution parameter setting. Both variants of the DE algorithm were chosen since their codes could be publicly obtained and they were in the first positions of the ranking made in [57], where Rcr-JADE was the best algorithm over a set of thirty algorithms in twenty two real-world problems when 50,000 and 100,000 function evaluations were allowed, while the COBIDE algorithm was the best when 150,000 function evaluations were allowed. These experiments have been implemented using the MATLAB programming language, specifically MATLAB 2017b version, and run on a server equipped with a 4.00 gigahertz AMD FX-8370 Eight-Core-Processor.

In order to assess the performance of the algorithms involved in the computational experiments, the goodness of fit reached in the training process through MSE and the computational cost, given as the number of function evaluations, have been used (see [35]). Also, in [54], RMSE is used to assess the performance of the model in many classifiers applied to predict the travel modes of passengers. However, in the case of classification problems, alternative validation measures, such as accuracy, ROC curve or F1 score, are also widely used on training and test problems (see [6, 16, 28]). This paper focuses on the performance of the optimization algorithms employed, and this is the main reason why metrics based on the error measured for by the algorithm in the objective function and the number of function evaluations have been chosen as performance metrics (see [57, 58]).

These three problems are then discussed, with an emphasis on the performance of the memetic algorithms used in this paper, and a statistical comparison is made with the selected algorithms from the state of the art.

This paper applies two recently proposed MAs based on qN search directions to the FNN neural network training problem in order to study whether the memetic approach can improve the performance of metaheuristic algorithms when training such networks.

To do this, MGSA and MCGSA were applied to three classical benchmark problems, along with classical GSA and CGSA and other metaheuristics from the state of the art, such as PSO, GA, Rcr-JADE and COBIDE. The main conclusions arising from the computational experiments undertaken in this study are:This work has focused on the use of qN as a local search method. However, the framework of MA allows the inclusion of specialized algorithm for the FNN neural network training problem, such as BP. These alternatives should be explored.

Moreover, more effort must be made with regard to data quality, since the effectiveness of a FNN depends to a great extent on the data used to train it. Thus, problems like imbalanced data, insufficient or overabundant data and high-dimensional data must be addressed in order to improve the performance of FNNs (see [43]).

Currently, in the era of big data, stream data (see [76]) are a kind of high-dimensional data that are very common in many areas, such as natural language processing, speech processing and social network data. Although the processing of these data is analyzed through deep learning architectures, FNNs trained with MAs can be tested in order to study performance. Also, memetic FNNs could be useful in managing and reducing the data stream (see [30]).

Furthermore, the use of evolutionary algorithms for optimizing the topology of deep neural networks is currently a hot topic in the literature. Future work should look at specializing memetic algorithms for optimizing the topology of deep and convolutional neural networks.

Finally, regarding the data stream provided by internet of things (IoT) devices, such as human activity data, social activity data or smartphone data, these need to be processed using inexpensive complex models (see [59]). In this context, tools such as memetic FNNs can be a suitable choice for analyzing these data.