Hybrid SVMR-GPR for modeling of chaotic time series systems with noise and outliers

Abstract

In this paper, the hybrid support vector machines for regression (SVMR) and Gaussian processes for regression (GPR) are proposed to deal with training data set with noise and outliers for the chaotic time series systems. In the proposed approach, there are two-stage strategies and can be a sparse approximation. In stage I, the SVMR approach is used to filter out some large noise and outliers in the training data set. Because the large noises and outliers in the training data set are almost removed, the affection of large noises and outliers is also reduced. That is, the proposed approach can be against the large noise and outliers. Hence, the proposed approach is also a robust approach. After stage I, the rest of the training data set is directly used to train the GPR in stage II. From the simulation results, the performance of the proposed approach is superior to least squares support vector machines regression (LS-SVMR), GPR, weighted LS-SVM and robust support vector regression networks when there are noise and outliers on the chaotic time-series systems.

Introduction

Subspace learning approaches have been developed in many data-analysis tasks to find a suitable representation of the data. That is, subspace learning is also an effective dimensionality reduction approach. In general, subspace learning approaches can be categorized as reconstructive approaches with principal components analysis or independent component analysis and discriminative approach with linear discriminant analysis [1] or canonical correlation analysis. Recently, there are many kernel-based machine approaches [2] such as support vector machines (SVM), principal components analysis, Gaussian processes (GPs) for subspace learning. In this study, the hybrid support vector machines for regression and Gaussian process for regression (SVMR-GPR) kernel machines are proposed to deal with training data set with noise and outliers for data-analysis tasks on chaotic time-series systems.

The SVM was proposed in Ref. [3]. An SVM is a machine learning method that is based on the statistical learning theory (SLT); the concept of the SLT is to infer the rule of the event from observed natural phenomenon or simulation results. The main capacities of the SVM are statistical classifier and regression analysis. The statistical classifier uses training data sets for learning, then accordingly predicts data sets to simulate, and obtains the results of prediction. The regression analysis is a numerical analysis method. It is expected to obtain the special relation between two data sets, but the most important thing is to find the function that can present the observation data. This function can express the relationship between variables; therefore, we can establish a math model to observe some variable to predict other variables. In an SVM, support vectors (SVs) is a very important concept. That is, SVs are feature subsets that are chosen from the training data sets. Besides, SVs can present the whole training data sets.

In general, the SVM is very suitable to approximate a high dimensionality space [3]. The support vector algorithm consists of a quadratic programming (QP) problem that can be guaranteed to find a global extremum solution under the selected hyperparameters. Hence, an SVM algorithm has more applications in many research fields, such as regression evaluation, image processing, probability function evaluation and so on. However, an SVM has to solve complicated quadratic programming. So Johan Suykens modified the SVM to least squares support vector machines (LS-SVM). The concept is to replace quadratic programming by linear function and it also investigated for regression [4]. The least squares method is an optimization numerical technology in math, which finds the optimal function from certain data by using squares to minimize error in the LS-SVM. Hence, an LS-SVM is computationally more effective than an SVM [5]. However, in reality applications, if the data have noises, it will affect the system performance. The traditional SVM have robustness that can avoid this problem; the robustness presents the capacity of tolerable error, but LS-SVM does not have this ability, and so we developed a new method: the so-called weighted LS-SVM [6]. Weighted LS-SVM can solve the condition that training data sets with noise and outliers. On the other hand, robust support vector regression networks [7], which are defined as RSVMRNs in this paper, are proposed to remove the large noise and outliers. That is, the weighted LS-SVM and the RSVMRNs have the robust property for the data with noise and outlier. Besides, the selection of the hyperparameters for the SVM can be found in [8]. GPs that are based on the statistical method are also a machine learning method. Besides, GPs are flexible, simple to implement, and is a full probabilistic method suitable for a wide range of problems in regression and classification [9]. Besides, the advantages of the GPs framework are to choose hyperparameters and covariance directly from training data.

In this study, the hybrid SVMR-GPR are proposed to overcome the above problem and deal with training data set with noise and outliers for the chaotic time-series systems. The time series have been widely used in many research areas, such as weather forecast [10], and noise cancellation [11], economic, business planning, production control, etc. And if a time series has a chaotic state, there are some rules to let the time series to be presented as a dynamic system. But the deterministic equations are not easy to understand. In general, predictions solve the empirical regularities via the experimental observations of the actual systems. In the proposed approach, there are two-stage strategies. In stage I, the support vector machine regression (SVMR) approach is used to filter out the large noise and outliers in the training data set. Because the large noises point and outliers in the training data set are almost removed, the affect of large noises and outliers is also reduced. After stage I, the rest of the training data set is directly used to train the Gaussian processes for regression (GPR) in stage II. Hence, the proposed approach has robustness for noise and outliers in stage I and is simple to implement and is a full probabilistic method in stage II. From the simulation results, the performance of the proposed approach is superior to the LS-SVMR, the GPR, the RSVMRNs and the weighted LS-SVM when the noise and outliers exist in the training data set.