Weighted Lagrange ε-twin support vector regression

doi:10.1016/j.neucom.2016.01.038

Neurocomputing

Volume 197, 12 July 2016, Pages 53-68

https://doi.org/10.1016/j.neucom.2016.01.038 Get rights and content

Highlights

•
Weighted Lagrangian ε-twin support vector regression (WL-ε-TSVR) is proposed.
•
Weight matrix D is introduced to reduce the impact of outliers.
•
WL-ε-TSVR just needs to solve the simple unconstrained minimization problems (UMPs).
•
A linearly convergent Lagrangian algorithm is used to obtain the solutions of UMPs.
•
Experimental results indicate that WL-ε-TSVR has remarkably improved generalization performance.

Abstract

In this paper, an efficient weighted Lagrangian ε-twin support vector regression with quadratic loss functions (WL-ε-TSVR) has been proposed. In our WL-ε-TSVR, to reduce the impact of outliers, the weight matrix is introduced to give different penalties for the samples located in different places. Further, by using the quadratic loss functions, we just need to solve the unconstrained minimization problems (UMPs) with differentiable convex objective functions in a space of dimensionality that equals to the number of training samples. In addition, the UMPs in WL-ε-TSVR could be solved by an extremely simple linearly convergent Lagrangian algorithm. Experimental results on both three artificial data sets and nine benchmark data sets show that compared with TSVR, ε-TSVR, ULTSVR, WSVR, and WTSVR, our WL-ε-TSVR achieves better generalization performance with comparable training time, and therefore confirm the superiority of our method.

Introduction

Support vector machines (SVMs) [1], [2], [3], computationally powerful tools for pattern classification and regression, have been successfully applied to various real-world problems [4], [5], [6], [7]. For support vector classification (SVC), many classical methods, such as C-support vector classification (C-SVC) [8], v-support vector classification (v-SVC) [9], and least square support vector classification (LS-SVC) [10] have been proposed. For support vector regression (SVR), there also exist some classical methods, such as ε-support vector regression (ε-SVR) [3], v-support vector regression (v-SVR) [9], least square support vector regression (LS-SVR) [11] and a variety of extended researches [12], [13], [14], [15], [16], [17]. ε-SVR, an important regression tool among these classical methods, aims to find a regression function f(x) such that, on the one hand, more training samples locate in the ε-intensive tube between $f (x) - ε$ and $f (x) + ε$ , and on the other hand, the regression function f(x) is as flat as possible by introducing the regularization term. Thus, the structural risk minimization principle is implemented. However, one of the main challenges for ε-SVR is the high computational complexity.

In order to improve the computational speed of ε-SVR, Peng [18] proposed twin support vector regression (TSVR) in the spirit of twin support vector machine (TWSVM) [19], [20], [21], [22], [23]. Different from ε-SVR, TSVR generates the regressor by seeking two nonparallel up- and down-bound functions by solving a pair of small sized quadratic programming problems (QPPs). However, only the empirical risk minimization principle is considered in TSVR. Later, Shao et al. [24] proposed an ε-insensitive twin support vector regression (ε-TSVR), which implements the structural risk minimization principle similar to ε-SVR and speeds up the training procedure by using the successive overrelaxation (SOR) technique. Preliminary experimental results in [18], [24] showed the effectiveness of TSVR and ε-TSVR over ε-SVR in terms of both generalization performance and training time. Consequently, twin-type SVR has been studied extensively [25], [26], [27], [28], [29], [30].

Recently, motivated by the work of TSVR and the Newton approach for the dual SVM, Balasundaram et al. [31] proposed a new unconstrained Lagrangian TSVR (ULTSVR) to further improve the computational speed by solving a pair unconstrained minimization problems. However, in TSVR, ε-TSVR, and ULTSVR, all samples are given the same penalties which may reduce the regression performance due to the impact of outliers. In fact, reducing the impact of outliers is one of the important issues for twin-type SVR. Considering the presence of outliers in practical regression, it is more reasonable to give the data samples different penalties to reduce the impact of outliers on the regressor.

Motivated by this, we propose an efficient weighted Lagrangian ε-twin support vector regression with quadratic loss functions (WL-ε-TSVR) in this paper. In our WL-ε-TSVR, the samples are given different penalties by introducing a weight matrix D to reduce the impact of the outliers on the regressor to a certain extent. In order to obtain more suitable weights for different samples, a fast retraining procedure is used. Meanwhile, in our WL-ε-TSVR, only an unconstrained differentiable convex function in a space of dimensionality equal to the number of training samples is minimized. In order to improve the computational speed, an extremely simple linearly convergent Lagrangian algorithm is used. The effectiveness of our WL-ε-TSVR is demonstrated by numerical experiments on three artificial data sets and nine benchmark data sets. Experimental results show that compared with the TSVR, ε-TSVR, ULTSVR, WSVR, and WTSVR, our WL-ε-TSVR achieves significant better generalization performance.

This study is organized as follows. Section 2 briefly dwells on ε-SVR and ε-TSVR. Section 3 proposes our WL-ε-TSVR. Experimental results are described in Section 4, and concluding remarks are given in Section 5.

Section snippets

Brief introduction to ε-SVR and ε-TSVR

Consider the following regression problem. Suppose that the training set is denoted by (X,Y), where X is a $l \times n$ matrix and the ith row of $X_{i} \in R^{n}$ represents the i-th training sample, $i = 1, 2, \dots, l$ . $Y = (y_{1}; y_{2}; \dots; y_{l})$ denotes the response vector of the training samples, where $y_{i} \in R$ , $i = 1, \dots, l$ . Here, we briefly introduce some methods which are related to our method, including ε-SVR and ε-TSVR. For simplicity, we only consider their linear cases.

Weighted Lagrangian ε-twin support vector regressor

All samples in ε-TSVR are given the least square loss plus the ε-intensive loss penalties. It implies that the samples may suffer the same penalties and maybe reduce the regression performance due to the impact of outliers. In fact, they have different effects on the regressor. Considering the presence of outliers, it is more reasonable to give the data samples different penalties to reduce the impact of outliers on the regressor. Motivated by weighted SVM, we propose a weighted Lagrange ε-twin

Experimental results

In this section, some experiments are conducted to demonstrate the regression performance of our WL-ε-TSVR compared with TSVR, ε-TSVR, ULTSVR, WSVR, and WTSVR on several data sets, including three types of artificial data sets and nine benchmark data sets [42]. All of these methods are implemented in a MATLAB 7.0 [43] environment on a PC with an Intel P4 processor (2.9 GHz) with 1 GB RAM. In our experiments, the best value of parameters is chosen by using a 10-fold cross validation procedure. We

Conclusions

In this paper, an efficient weighted Lagrangian ε-twin support vector regressor with quadratic loss functions (WL-ε-TSVR) has been proposed. Our WL-ε-TSVR provides different penalties for different samples depending on their different effect on the regressor. Further, a fast retraining procedure is used to obtain the proper weight for each sample, and a Lagrangian algorithm is used to solve the optimal problems. Computational comparisons between WL-ε-TSVR and ε-TSVR, TSVR, ULTSVR, WSVR, WTSVR,

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Nos. 11201426, 11161045, 11426202, and 11371365), the Zhejiang Provincial Natural Science Foundation of China (Nos. LQ12A01020, LQ13F030010, LQ14G010004, and LY15F030013), the Ministry of Education, Humanities and Social Sciences Research Project of China (No. 13YJC910011), and the Zhejiang Provincial University Students Science and Technology Innovation Activities Program (Xinmiao Talents Program) (No. 2014R403063),

Ya-Fen Ye received her master׳s degree in quantitative economics, and Ph.D. degree in statistics in College of statistics and mathematics from the Zhejiang Gongshang University, China, in 2008 and 2011, respectively. Currently, she is an Associate Professor at the Zhijiang College, Zhejiang University of Technology. Her research interests include quantitative economics, machine learning and data mining.

References (52)

J. Yang et al.
Brain activation detection by neighborhood one-class SVM
Cogn. Syst. Res.
(2010)
H.Q. Yang et al.
Localized support vector regression for time series prediction
Neurocomputing
(2009)
X.J. Peng
TSVRan efficient twin support vector machine for regression
Neural Netw.
(2010)
S. Mehrkanoon et al.
Non-parallel support vector classifiers with different loss functions
Neurocomputing
(2014)
Z.Q. Qi et al.
Robust twin support vector machine for pattern classification
Pattern Recognit.
(2013)
Z.Q. Qi et al.
Structural twin support vector machine for classification
Knowl.-Based Syst.
(2013)
X.J. Peng
Primal twin support vector regression and its sparse approximation
Neurocomputing
(2010)
Y.F. Ye et al.
Exploring determinants of inflation in China based on L₁-ε twin support vector regression
Proc. Comput. Sci.
(2013)
X.J. Peng et al.
A twin projection support vector machine for data regression
Neurocomputing
(2014)
Y.T. Xu et al.
A weighted twin support vector regression
Knowl.-Based Syst.
(2012)

S. Balasundaram et al.

Training Lagrangian twin support vector regression via unconstrained convex minimization

Knowl.-Based Syst.

(2014)

J. Bi et al.

A geometric approach to support vector regression

Neurocomputing

(2003)

J.A.K. Suykens et al.

Weighted least squares support vector machines: robustness and sparse approximation

Neurocomputing

(2002)

H.S. Tang et al.

Online weighted LS-SVM for hysteretic structural system identification

Eng. Struct.

(2006)

Y.H. Shao et al.

Weighted linear loss twin support vector machine for large-scale classification

Knowl.-Based Syst.

(2015)

R.J. Hyndman et al.

Another look at measures of forecast accuracy

Int. J. Forecast.

(2006)

S. Garcia et al.

Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data miningexperimental analysis of power

Inf. Sci.

(2010)

V. Vapnik et al.

Support vector networks

Mach. Learn.

(1995)

V. Vapnik

Statistical Learning Theory

(1998)

C. Burges

A tutorial on support vector machines for pattern recognition

Data Min. Knowl. Discov.

(1998)

Y.X. Li et al.

Improved prediction of palmitoylation sites using PWMs and SVM

Protein Pept. Lett.

(2011)

H.Q. Yang et al.

Support vector machine regression for volatile stock market prediction

Intell. Data Eng. Autom. Learn.

(2002)

Y.X. Li et al.

An efficient support vector machine approach for identifying protein s-nitrosylation sites

Protein Pept. Lett.

(2011)

N.Y. Deng et al.

Support Vector Machines: Optimization Based Theory, Algorithms, and Extensions

(2012)

B. Scholkopf et al.

New support vector algorithms

Neural Comput.

(2000)

J.A.K. Suykens, L. Lukas, P. Vandooren, B. Demoor, J. Vandewalle, Least squares support vector machine classifiers: a...

Cited by (41)

An overview on twin support vector regression
2022, Neurocomputing
Twin support vector regression (TSVR) is a useful extension of traditional support vector regression (SVR). As a new regression model, the basic idea of TSVR is generating a pair of nonparallel functions on both sides of the training data points, such that the $ε$ -insensitive upper and lower bounds of the regression function can be determined. Owing to its excellent learning ability, TSVR has become a research hotspot in the field of machine learning. With the deepening of such research, scholars have found that TSVR also has certain limitations, and thus various improved models have been proposed. This review aims to report the recent developments in twin support vector regression. First, the basic concepts and basic models of TSVR are introduced. Second, the improved algorithms and applications of TSVR in recent years are summarized, and the advantages and disadvantages of its representative algorithms are analyzed and compared with the experiments. Finally, we discuss the research conducted on TSVR.
Shape optimization of GFRP elastic gridshells by the weighted Lagrange ε-twin support vector machine and multi-objective particle swarm optimization algorithm considering structural weight
2021, Structures
Elastic gridshell is a type of free-form structure usually with double-curved shape and it is constructed by deforming an initially flat grid to achieve the final structural form. Determining a reasonable structural shape considering structural performance is an essential task in the design of such structures. This paper presents a shape optimization method for elastic gridshells considering structural weight, aiming to minimize the structural stress and deformation. Support vector machine is utilized to predict the structural performance in the optimization process in order to avoid the time-consuming structural analysis. The least square support vector machine (LSSVM), weighted least square support vector machine (WLSSVM), a combination of particle swarm optimization and least square support vector machine (PSO-LSSVM), and weighted Lagrange ε-twin support vector machine (WL-ε-TSVM) are first compared for predicting the structural analysis results. The WL-ε-TSVM algorithm shows superior performance and is further adopted in the optimization method. The k-fold cross validation is implemented during the validation process of this algorithm to improve the predictive performance. Based on predicted structural performance, the optimal shape of gridshell is provided by using the multi-objective particle swarm optimization (MOPSO) algorithm. The Taguchi technique is applied to tune the parameters of the MOPSO algorithm. Afterwards, the technique for order preference by similarity to ideal solution (TOPSIS) is implemented to determine the most desirable solution from the Pareto optimal set. The presented method is validated through an example and the structural behavior of the optimized structure is further assessed by finite element analysis. Results demonstrate that the presented method is applicable for finding the optimum shape of gridshells with high structural performance.
A new grey prediction model and its application in landslide displacement prediction
2021, Chaos, Solitons and Fractals
Citation Excerpt :
Twin support vector regression (TSVR) is an important variant of support vector machine (SVM) [26]. Published studies [27–29] proved that the performance of TSVR is superior to that of SVM and LSSVM. At present, no prediction model based on TSVR and grey prediction theory has been developed.
Developing a grey prediction model with high nonlinear prediction accuracy is an important issue in grey system theory. A new grey prediction model was developed that was the first to combine the idea of twin support vector regression with Hausdorff derivative operator. The new model is a non-linear data-driven model. An improved salp swarm algorithm is used to determine parameters of the model. Two numerical examples show that the error of the new model is smaller than the existing grey prediction models and least square support vector machine model. Moreover, with the displacement, precipitation, reservoir level elevation, variation velocity of reservoir level elevation, and displacement velocity of the previous month as the input variables, the new model was successfully used to predict the displacement of a landslide in the real-world. The new model is a powerful tool for solving nonlinear prediction problems.
On robust asymmetric Lagrangian ν-twin support vector regression using pinball loss function
2021, Applied Soft Computing
The main objective of twin support vector regression (TSVR) is to find the optimum regression function based on the $ε$ -insensitive up- and down-bound with equal influences on the regression function where all the data points have a different location above the up-bound points and below the down-bound points. However, the effects of all data points must be distinct based on their distribution in the regression function. Recently, asymmetric $ν$ -twin support vector regression (Asy- $ν$ -TSVR) is encouraged on the same subject but still, the present matrices in the mathematical formulation have faced the problem of semi-definite. In order to handle this problem effectively, a new regressor model named as robust asymmetric Lagrangian $ν$ -twin support vector regression using pinball loss function (URALTSVR) proposes as a pair of the unconstrained minimization problem to handle not only the noise sensitivity and instability of re-sampling but also consist positive definite matrices. Here, we suggest the proposed model URALTSVR in such a way where the pinball loss function is playing a vital role to control the fitting error inside the asymmetric tube. One of the advantages is that unlike TSVR and Asy- $ν$ -TSVR, it considers the concept of structural risk minimization principle through the inclusion of regularization term as well as change the one-norm of the vector of the slack variable by two-norm, which yields the dual problem to be strongly convex, stable and well-posed. Aforementioned, the proposed formulation has a continuous and piecewise quadratic problem that is solved by their gradients based iterative approaches. Specifically, we analyze the three implementations of URALTSVR with the baselines approaches support vector regression (SVR), TSVR and Asy- $ν$ -TSVR, which discard the dependencies to solve a pair of quadratic programming problem (QPP) for obtaining the unique global solution. Overall, SRALTSVR1 based on smooth approximation function performs outstanding for artificial and real-world datasets.
Robust twin support vector regression based on rescaled Hinge loss
2020, Pattern Recognition
Citation Excerpt :
So, the Algorithm 1 converges to a point (w1, b1, t) which is an optimum solution to (18). In this subsection, we demonstrate the performance of the proposed Res-TSVR on three synthetic data sets that have been used in [29] and [31]. These synthetic data sets are defined in Table 2.
In this work, with the help of the rescaled Hinge loss, we propose a twin support vector regression (TSVR) model that is robust to noise. The corresponding optimization problem turns out to be non-convex with smooth l₂ regularizer. To solve the problem efficiently, we convert it to its dual form, thereby transforming it into a convex optimization problem. An algorithm, named Res-TSVR, is provided to solve the formulated dual problem. The proof of the convergence of the algorithm is given. It is shown that the maximum number of iterations to achieve an ε-precision solution to the dual problem is $O (\log (\frac{1}{ε}))$ . We conduct a set of numerical experiments to compare the proposed method with the recently proposed robust approaches of TSVR and the standard SVR. Experimental results reveal that the proposed approach outperforms other robust methods of TSVR in terms of generalization performance and robustness to noise with comparable training time. This claim is based on the experiments performed using seven real-world data sets and three synthetic data sets.
Robust support vector regression with generic quadratic nonconvex ε-insensitive loss
2020, Applied Mathematical Modelling
In this paper, we propose a robust support vector regression with a novel generic nonconvex quadratic ε-insensitive loss function. The proposed method is robust to outliers or noise since it can adaptively control the loss value and decrease the negative influence of outliers or noise on the decision function by adjusting the elastic interval parameter and adaptive robustification parameter. Given the nature of the nonconvexity of the optimization problem, a concave-convex programming procedure is employed to solve the proposed problem. Experimental results on two artificial data sets and three real-world data sets indicate that the proposed method outperforms support vector regression, L₁-norm support vector regression, least squares support vector regression, robust least squares support vector regression, and support vector regression with the Huber loss function on both robustness and generalization ability.

View all citing articles on Scopus

Lan Bai received her Doctor׳s degree in College of Mathematics from the Jilin University, China, in 2014. Currently, she is a lecturer in School of Mathematical Sciences from the Inner Mongolia University. Her research interests include pattern recognition, feature selection, and data mining.

Xiang-Yu Hua received his B.S. degree in College of Mathematics and Computer Science from the Yunnan University of Nationalities, Kunming, China, in 2011. Currently, he is Ph.D. student in School of Economics and Management from the Zhejiang University of Technology, Hangzhou, China. His research interests include machine learning, data mining, economics and management.

Yuan-Hai Shao received his B.S. degree in information and computing science in College of Mathematics from the Jilin University, the master׳s degree in applied mathematics, and Ph.D. degree in operations research and management in College of Science from China Agricultural University, China, in 2006, 2008 and 2011, respectively. Currently, he is an associate professor at the Zhijiang College, Zhejiang University of Technology. His research interests include optimization methods, machine learning and data mining. He has published over 40 refereed papers.

Zhen Wang received his Doctor׳s degree in College of Mathematics from the Jilin University, China, in 2014. Currently, he is lecturer in School of Mathematical Sciences from Inner Mongolia University. His research interests include pattern recognition, text categorization, and data mining.

Nai-Yang Deng received the M.Sc. degrees in Department of Mathematics from the Peking University, China, in 1967. Now, he is a professor in College of Science, China Agricultural University, he is an honorary director of China Operations Research Society, Managing Editor Journal of Operational Research, International Operations Research Abstracts Editor. His research interests mainly including operational research, optimization, machine learning and data mining. He has published over 100 refereed papers.

View full text

Weighted Lagrange ε-twin support vector regression

Highlights

Abstract

Introduction

Section snippets

Brief introduction to ε-SVR and ε-TSVR

Weighted Lagrangian ε-twin support vector regressor

Experimental results

Conclusions

Acknowledgments

Cogn. Syst. Res.

Neurocomputing

Neural Netw.

Neurocomputing

Pattern Recognit.

Knowl.-Based Syst.

Neurocomputing

Proc. Comput. Sci.

Neurocomputing

Knowl.-Based Syst.

Knowl.-Based Syst.

Neurocomputing

Neurocomputing

Eng. Struct.

Knowl.-Based Syst.

Int. J. Forecast.

Inf. Sci.

Support vector networks

Mach. Learn.

Statistical Learning Theory

A tutorial on support vector machines for pattern recognition

Data Min. Knowl. Discov.

Improved prediction of palmitoylation sites using PWMs and SVM

Protein Pept. Lett.

Support vector machine regression for volatile stock market prediction

Intell. Data Eng. Autom. Learn.

An efficient support vector machine approach for identifying protein s-nitrosylation sites

Protein Pept. Lett.

Support Vector Machines: Optimization Based Theory, Algorithms, and Extensions

New support vector algorithms

Neural Comput.