Approximation of two-variable functions using high-order Takagi–Sugeno fuzzy systems, sparse regressions, and metaheuristic optimization

The literature on using metaheuristic optimization methods to train fuzzy systems is extensive. Further are discussed the papers in which hybrid methods using metaheuristics and regressions were applied. In such methods, the antecedents are trained by metaheuristic methods, while the consequents are trained by regressions.

One of the most commonly used algorithms to train fuzzy systems is particle swarm optimization (PSO). An approach presented in Li and Wu (2011) combines particle swarm optimization and recursive least squares estimator to obtain a fuzzy approximation. The PSO is used to train the antecedent part of the first-order T–S system, whereas the consequent part is trained by the RLSE method. Building a type-2 neural-fuzzy system was discussed in Yeh et al. (2011). In the first step, a fuzzy clustering method is used to partition the dataset into clusters. Then, a type-2 fuzzy Takagi–Sugeno–Kang (TSK) rule is derived from each cluster. The parameters are refined using PSO and a divide-and-merge-based least squares. In Ying et al. (2011), an approach to function approximation using robust fuzzy regression and particle swarm optimization is proposed. A fuzzy regression is used to construct the first-order TSK fuzzy model, whereas particle swarm optimization is used to tune its parameters. A self-learning complex neuro-fuzzy system that uses Gaussian complex fuzzy sets was proposed in Li et al. (2012). The knowledge base consists of the T–S fuzzy rules with complex fuzzy sets in the antecedent part and linear models in the consequent part. The antecedent parameters and the consequent parameters are trained by the particle swarm optimization algorithm and recursive least squares, respectively. In the paper (Soltani et al. 2012), a method for fuzzy c-regression model clustering was proposed. The method combines the advantages of two algorithms: clustering and particle swarm optimization. The consequent parameters of the first-order T–S fuzzy rules are estimated by the orthogonal least squares method. A self-constructing radial basis function neural-fuzzy system was proposed in Yang et al. (2013). The proposed method uses particle swarm optimization for generating the antecedent parameters and the least-Wilcoxon norm for the consequent parameters, instead of the traditional least squares estimation. A two-step fuzzy model building algorithm based on particle swarm optimization and kernel ridge regression was presented in Boulkaibet et al. (2017). In the first step, the clustering based on particle swarm optimization separates the input data into clusters and obtains the antecedent parameters. In the second step, the consequent parameters are calculated using a kernel ridge regression. In Taieb et al. (2018), the adaptive chaos particle swarm optimization algorithm (ACPSO) using weighted recursive least squares was proposed. The ACPSO is used to optimize the parameters of the model, and then, the obtained parameters are used to initialize the fuzzy c-regression model. A fuzzy model identification method was proposed in Tsai and Chen (2018). Firstly, the fuzzy c-means algorithm is used to determine the rule number. Next, the initial fuzzy sets and the consequent parameters are obtained by particle swarm optimization. The final parameters are obtained using fuzzy c-regression and orthogonal least squares methods. In the paper (Tu and Li 2018), there is proposed a complex-fuzzy machine learning approach to function approximation. Particle swarm optimization is used to select the premise parameters of the first-order fuzzy model, while the recursive least squares estimator is used to find the consequent parameters.

Other commonly used methods are genetic algorithms (GA). A two-step approach to the construction of first-order fuzzy rules from data was proposed in Setnes and Roubos (2000). In the first step, fuzzy clustering and the weighted least squares are used to obtain an initial fuzzy model. In the second step, this model is optimized by a real-coded genetic algorithm that allows simultaneous tuning of the rule antecedents and consequents. In Wang et al. (2005), a scheme based on multi-objective hierarchical GA (MOHGA) is proposed. This scheme is used to extract interpretable rule-based knowledge from data. First, fuzzy clustering is applied to generate an initial rule-based model. Then, the MOHGA and the recursive least squares estimator (RLSE) are used to obtain the optimized fuzzy models. A fuzzy modeling approach for the identification of nonlinear control processes was discussed in Yusof et al. (2011). This approach is based on a combination of genetic algorithm and recursive least squares. The antecedent parameters of the first-order T–S model are tuned by a genetic algorithm, whereas the consequent part is identified by recursive least squares estimator.

Various other approaches to train fuzzy models using metaheuristic optimization can be found in Almaraashi et al. (2016), Cordón et al. (2000, (2001), Cheung et al. (2014), Juang and Lo (2008), Khayat et al. (2009), Khosla et al. (2005, (2007), Lin (2008), Lin et al. (2016), Martino et al. (2014), Niu et al. (2008), Prado et al. (2010), Rastegar et al. (2017), Shihabudheen et al. (2018), Yanar and Akyürek (2011), Zhao et al. (2010). Advantages and disadvantages of the reviewed methods and the proposed method are presented in Table 1.

From the literature review, it is seen that at most first-order polynomials are used in the consequent part to train fuzzy systems. In this paper, we propose to use high-order fuzzy systems for two-variable function approximation. In such systems, higher-order polynomials in the consequent part of rules are used, which can give greater flexibility in the selection of system parameters. Moreover, there is no use of sparse regressions for two-variable function approximation. Sparse regressions can generate sparse models (Sjöstrand et al. 2018), which are more compact, easier to interpret and implement. Summarizing, the main contributions of this paper can be stated as:In the proposed method, the premise parameters are determined manually or by metaheuristic optimization methods such as particle swarm optimization (PSO), genetic algorithm (GA), and simulated annealing (SA). The consequent parameters are calculated by ordinary least squares (OLS), ridge regression (RIDGE), and sparse regressions. The following sparse regressions have been used: forward selection (FS), least angle regression (LAR), least absolute shrinkage and selection operator (LASSO), and elastic net regression (ENET). The OLS regression was used as a reference model. This paper is a continuation of the work (Wiktorowicz and Krzeszowski 2020), where the approximation of one-variable functions was considered.

The structure of this paper is as follows. Section 2 describes the Takagi–Sugeno fuzzy system with two inputs and with high-order polynomials in the consequent parts of the fuzzy rules. Section 3 presents the training methods of the consequent parameters when using the OLS, RIDGE, and sparse regressions. Section 4 presents the training methods of the antecedent parameters when using the PSO, GA, and SA methods. The performance criterion is described in Sect. 5. Section 6 contains the design procedure for training fuzzy models. In Sect. 7, the experimental results are presented. Finally, the conclusions are given in Sect. 8.

The cost function to be minimized in the OLS is the sum of squared errorswhere \({\hat{y}}_i={\mathbf {h}}((x_1)_i,(x_2)_i){\mathbf {w}}\) is the estimated output of the system (see Eq. 15) for the ith observation. The optimal solution is given by Bishop (2006)where \({\mathbf {y}}=[y_1,\ldots ,y_n]^T\). Because the model parameters are computed directly from all the data contained in \({\mathbf {X}}\) and \({\mathbf {y}}\), this method is a batch least squares.

The cost function in the ridge regression (Hoerl and Kennard 1970) is the penalized sum of squared errorswhere \(\lambda \ge 0\) is a regularization parameter. The fuzzy model weights are given bywhere \({\mathbf {I}}\) is the identity matrix. The ridge regression is applied in this paper because it can be used for ill-conditioned problems, that is when the matrix \({\mathbf {X}}^T{\mathbf {X}}\) is close to singular. The ridge regression, similarly as the OLS, is a one-pass method, and therefore it is very fast.

The sparse regressions briefly described in this section allow the coefficients of a model to be exactly zero (Sjöstrand et al. 2018). These regressions lead to simplified models that are easier to interpret.

In the forward selection, that is an example of stepwise regression, the variables are added one by one to the model. In the beginning, all coefficients are equal to zero, and then a particular variable is chosen. The next variable to include can be chosen based on a number of criteria. For example, it can be the one that has the highest correlation with the current residual vector (Sjöstrand et al. 2018).

The least angle regression (Efron et al. 2004; Sjöstrand et al. 2018) works similarly to the FS procedure, but the algorithm does not move in the direction of one variable. In the LAR, the estimated parameters are calculated in a direction in which the angles with each of the variables currently in the model are equal. This algorithm is the basis for other sparse methods, such as the LASSO and elastic net regression.

The least absolute shrinkage and selection operator regression (Sjöstrand et al. 2018; Tibshirani 1996) has a mechanism that implements a coefficient shrinkage and variable selection. The cost function combines the sum of the squared errors and the penalty function based on the \(L_1\) norm:where \( \lambda \) is a nonnegative regularization parameter.

The elastic net regression (Sjöstrand et al. 2018; Zou and Hastie 2005) combines the features of the ridge regression and the LASSO. The cost function includes a penalty term related to both the \(L_1\) and the \(L_2\) norms:where \(\lambda \) and \(\delta \) are nonnegative regularization parameters. The solution is found by the LARS-EN algorithm, which is based on the LARS algorithm (Efron et al. 2004).

We can see that in the solution \(\varvec{\beta }_2\), the coefficient b is exactly zero, that results from using the sparse regressions. The selection of one of the solutions is based on a specific criterion, e.g., cross-validation, Akaike’s information criterion, or Bayesian information criterion (Sjöstrand et al. 2018).

Particle swarm optimization is a population-based algorithm developed by Kennedy and Eberhart Eberhart and Shi (2000), Kennedy and Eberhart (1995). It is based on the social behavior of living organisms that live in large groups like birds flock or fish school. In PSO, a group of particles (a population) forms a swarm, in which each particle represents a hypothetical solution. The particle remembers its best position \({\mathbf {pbest}}\) and has access to the best position \({\mathbf {gbest}}\) in the swarm. The best local and global positions are selected using an objective function (Sect. 5). The learning scheme is based on two components:The velocity \({\mathbf {v}}_k\) and the position \({\mathbf {x}}_k\) of the kth particle are calculated based on the following equations (Eberhart and Shi 2000; MathWorks 2019a):where \(\omega \) is the inertia weight, \({\mathbf {r}}_{1}\), \({\mathbf {r}}_{2}\) are vectors of random numbers uniformly distributed within [0,1], l is the current iteration number, and \(c_1\), \(c_2\) are the cognitive and social coefficients, respectively.

Genetic algorithm (Holland 1992; MathWorks 2019a; Whitley 1994) is a method for solving optimization problems inspired by the biological process of Darwinian evolution, where selection, crossover, and mutation play a major role. The GA repeatedly modifies a population to achieve new and possibly better solutions. In each generation of the GA, the individuals are randomly selected from the current population to be “parents” and used to obtain “children” for the next generation. In subsequent generations, the population “evolves” toward the optimal solution.

The GA uses three main types of rules to create the next generation from the current population:Crossover and mutation characterize the explorative and exploitative features of GA. Maintaining a balance between these two features is crucial to speed up the search process and to achieve high-quality solutions.

Simulated annealing (Kirkpatrick et al. 1983; MathWorks 2019a) is a method for solving unconstrained and bound-constrained optimization problems. This method was originally inspired by the process of annealing in metallurgy. The SA models the process of heating a material and then gradually lowering the temperature in order to reduce defects. The goal is to move the system from the initial state to the state with minimum energy. As the algorithm runs, a new state is randomly generated and accepted with a certain probability. The acceptance probability is a function that depends on the energies of the two states and the temperaturewhere \({\varDelta }E\) is the difference of energies of the present and previous solution (\({\varDelta }E = E_{k+1}-E_k\)) and T is the current temperature. The algorithm systematically decreases the temperature and stores the best state found so far. The energy determines how good the solution is, and it corresponds to the value of the objective function (Sect. 5).

This section gives examples of two-variable nonlinear function approximation. The following parameters were used in all experiments. The number of observations \(([(x_1)_i,(x_2)_i]^T,y_i)\) was \(n=81\), and they were evenly distributed in the space \({\mathbb {X}}_1\times {\mathbb {X}}_2\). The best method was selected using the Monte-Carlo cross-validation (MCCV) (Picard and Cook 1984), in which the data set was divided randomly into some fraction of data to form the training set and to assign the rest of the points to the validation set. This process was repeated 10 times, generating new training and validation partitions in the proportion of 70% test data and 30% validation data. Statistical analysis was carried out using Wilcoxon signed rank test for differences in \(\mathrm {RMSE}\) results between all methods and the reference method (OLS). For the inputs of the fuzzy system, three fuzzy sets were defined, which gave nine fuzzy inference rules. The widths of fuzzy sets were bounded in the intervals \([\sigma _{min},\sigma _{max}]=[\delta _{min},\delta _{max}]=[0.0849,2.123]\). The degree of the polynomials in the consequent part was set to two. For the ridge regression (23), \(\lambda =1\mathrm {e-}08\) and the ENET regression (25), \(\delta =1\mathrm {e-}8\). The number of objective function evaluations was 6000. The parameter in the quality criterion (36) was \(\alpha =0.5\). For metaheuristic algorithms, default parameter values were adopted in accordance with the implementation contained in the Matlab toolbox. The experiments were carried out on a mobile computer equipped with Intel(R) Core(TM) i5-7200U and 8GB RAM.

The function regress from the Matlab Statistics and Machine Learning Toolbox (MathWorks 2019b) has been used to apply the OLS regression. The ridge regression has been implemented in Matlab using a custom function.

The sparse regressions have been implemented in Matlab using the toolbox SpaSM (Sjöstrand et al. 2018). From this toolbox, the following functions have been used: forwardselection, lar, lasso, and elasticnet. These functions take the regression matrix \({{\mathbf {X}}}\) and the vector \({{\mathbf {y}}}\) as arguments. Moreover, the function elasticnet has the regularization parameter \(\delta \). As the output, the described functions return the solution path in the form of the coefficients \({{\mathbf {w}}}\), from which the best solution can be selected.

The metaheuristic methods have been implemented using the Global Optimization Toolbox in Matlab (MathWorks 2019a). From this toolbox, the following functions have been used: particleswarm, ga, and simulannealbnd. These functions allow the solution to be obtained subject to the bounds defined by the user. They operate on the vector that contains the parameters of Gaussian membership functions:where \(p_1, \ldots , p_{\rho }\), \(q_1, \ldots , q_{\rho }\) are the peaks of membership functions, \(\sigma _1, \ldots , \sigma _{\rho }\), \(\delta _1, \ldots ,\delta _{\rho }\) are the widths of membership functions.

We consider the nonlinear function (Yeh et al. 2011)

where \(x_1\in [-1,1]\) and \(x_2\in [0,1]\). The results of Experiment 1 are presented in Table 3. The statistical analysis of the \(\mathrm {RMSE}\) showed that most of the calculated models generate significantly different results (\(p < 0.05\)) compared to the OLS model. The exception is the LAR, LASSO and ENET models. The smallest value of the quality index \({\overline{Q}}\) is equal to 0.1450 and was obtained for the PSO-ENET method. For this method, the validation error \(\overline{\mathrm {RMSE}}\) is \(1.864\mathrm {e-}03\), and it is smaller than for the reference model for which this error is equal to \(4.800\mathrm {e-}02\). The sparsity \({\overline{S}}\) is 0.7489 that means that the PSO-ENET method zeroed out 75% of 45 coefficients. Thanks to this, the model is easier to interpret and implement. Table 4 contains the parameters of the fuzzy systems obtained by the OLS and the PSO-ENET methods. Based on this table, the fuzzy rules for the PSO-ENET model can be written as

It is worth noting that in the PSO-ENET model, three rules (\(R_4\), \(R_5\), \(R_6\)) have the zero polynomial in the consequent part. Figures 3, 4, 5 show the goal function y, the estimator \({\hat{y}}\), and the approximation error \(y-{\hat{y}}\) for the best model. The average time of training high-order T–S fuzzy systems using PSO-ENET method for one MCCV data subset was about 40.28 s (Table 5). The methods with manually chosen fuzzy sets (OLS, RIDGE, FS, LAR, LASSO, ENET) have the shortest calculation times, while the longest times have been obtained by algorithms using SA.

This experiment applies the nonlinear function (Yeh et al. 2011)where \(x_1\in [-1,1]\) and \(x_2\in [0,1]\). The results are presented in Table 6. The statistical analysis of the \(\mathrm {RMSE}\) showed that all calculated models generate significantly different results (\(p < 0.05\)) compared to the OLS model. The smallest value of the quality index \({\overline{Q}}\) is equal to 0.1515 and was obtained for the PSO-FS method. For this method, the validation error \(\overline{\mathrm {RMSE}}\) is \(3.403\mathrm {e-}02\), and the sparsity \({\overline{S}}\) is 0.7956. The PSO-FS method zeroed out 80% of 45 coefficients. The OLS method achieved the validation error \(\overline{\mathrm {RMSE}}\) equal to \(3.457\mathrm {e-}01\). Table 7 contains the parameters of the fuzzy systems obtained by the OLS and the PSO-FS methods. The fuzzy rules for the PSO-FS model can be written asIt is seen that two rules (\(R_1\) and \(R_7\)) have the zero polynomial in the consequent part. Figures 6, 7 and 8 show the goal function y, the estimator \({\hat{y}}\), and the approximation error \(y-{\hat{y}}\) for the best model. The average time of training high-order T–S fuzzy systems using the PSO-FS method for one MCCV data subset was about 48.48 s (Table 5). As in Experiment 1, the methods with manually chosen fuzzy sets (OLS, RIDGE, FS, LAR, LASSO, ENET) have the shortest calculation times, while the longest times have been obtained by algorithms using SA.