Noise model based $\nu$-support vector regression with its application to short-term wind speed forecasting

Abstract

Support vector regression (SVR) techniques are aimed at discovering a linear or nonlinear structure hidden in sample data. Most existing regression techniques take the assumption that the error distribution is Gaussian. However, it was observed that the noise in some real-world applications, such as wind power forecasting and direction of the arrival estimation problem, does not satisfy Gaussian distribution, but a beta distribution, Laplacian distribution, or other models. In these cases the current regression techniques are not optimal. According to the Bayesian approach, we derive a general loss function and develop a technique of the uniform model of $\nu$-support vector regression for the general noise model (N-SVR). The Augmented Lagrange Multiplier method is introduced to solve N-SVR. Numerical experiments on artificial data sets, UCI data and short-term wind speed prediction are conducted. The results show the effectiveness of the proposed technique.

Introduction

Regression is an old topic in the domain of learning functions from a set of samples (Hastie, Tibshirani, & Friedman, 2009). It provides researchers and engineers with a powerful tool to extract hidden rules of data. The trained model is used to predict future events with the information of past or present events. Regression analysis is now successfully applied in nearly all fields of science and technology, including the social sciences, economics, finance, wind power prediction for grid operation. However this domain is still attracting much attention from research and application domains.

Generally speaking, there are three important issues in designing a regression algorithm: model structures, objective functions and optimization strategies. The model structures include linear or nonlinear functions (Park & Lee, 2005), neural networks (Spech, 1990), decision trees (Esposito, Malerba, & Semeraro, 1997), and so on; optimization objectives include $ϵ$-insensitive loss (Cortes and Vapnik, 1995, Vapnik, 1995, Vapnik et al., 1996), squared loss (Suykens et al., 2000, Wu, 2010, Wu and Law, 2011), robust Huber loss (Olvi & David, 2000) etc. According to the formulation of optimization functions, a collection of optimization algorithms (Ma, 2010) have been developed. In this work, we focus on the problem which optimal formulation should be considered with respect to different error models.

Suppose we are given a set of training data

$$D_l=\{(x_1,y_1),(x_2,y_2),\dots ,(x_l,y_l)\}\text{,}$$

where $x_i\in R^L,y_i\in R,i=1,2,\dots ,l$. Take a multivariate linear regression task $f$ as an example. The form is

$$f\left(x\right)={\omega }^T\cdot x+b\text{,}$$

where $\omega \in R^L,b\in R,i=1,2,\dots ,l$. The task is to learn the parameter vectors $\omega$ and parameter $b$, by minimizing the objective function

$$g_{LR}=\sum _{i=1}^l{(y_i-{\omega }^T\cdot x_i-b)}^2\text{.}$$

The objective function of the sum-of-squares error is usually used in regression. The trained model is optimal, if the samples have been corrupted by independent and identical probability distributions (i.i.d.). Noise satisfying Gaussian distribution with zeros mean and variance ${\sigma }^2$, i.e., $y_i=f\left(x_i\right)+{\xi }_i,i=1,\dots ,l,{\xi }_i\sim N(0,{\sigma }^2)$.

In the recent years, the support vector regressor (SVR) is growing up as a popular technique (Cortes and Vapnik, 1995, Cristianini and Shawe, 2000, Smola and Schölkopf, 2004, Vapnik, 1995, Vapnik, 1998, Vapnik, 1999, Vapnik et al., 1996, Wu, 2010, Wu and Law, 2011). It is a universal regression machine based on the V–C dimension theory. This technique is developed with the Structural Risk Minimization (SRM) principle, which has shown its effectiveness in applications. The classical SVR is optimized by minimizing Vapnik’s $ϵ$-insensitive loss function of residuals and has achieved good performance in a variety of practical applications (Bayro-Corrochano and Arana-Daniel, 2010, Duan et al., 2012, Huang et al., 2012, Kwok and Tsang, 2003, Lopez and Dorronsoro, 2012, Yang and Ong, 2011).

In 1995, $ϵ\text{-}\mathrm{SV\; R}$ was proposed by Vapnik and his research team (Cortes and Vapnik, 1995, Vapnik, 1995, Vapnik et al., 1996). In 2000, $\nu \text{-}\mathrm{SV\; R}$ was introduced by Schölkopf, Smola, Williamson, and Bartlett (2000), which automatically computes $ϵ$. Suykens et al. (2000) constructed least squares support vector regression with Gaussian noise (LS-SVR). Wu (Wu, 2010, Wu and Law, 2011) and Pontil, Mukherjee, and Girosi (1998) constructed $\nu$-support vector regression with Gaussian noise (GN-SVR). If the noise obeys the Gaussian distribution, the outputs of the models are optimal. However, it was found that the noise in some real-world applications, just like wind power forecast and direction-of-arrival estimation problem, does not satisfy Gaussian distribution, but a beta distribution or Laplace distribution, respectively. In these cases these regression techniques are not optimal.

The principle of $\nu$-support vector regression ($\nu \text{-}\mathrm{SV\; R}$) can be written as (Chalimourda et al., 2004, Chih-Chung and Chih-Jen, 2002, Schölkopf et al., 2000):

$$min\{g_{P_{\nu \text{-}\mathrm{SVR}}}=\frac{1}{2}{\Vert \omega \Vert }^2+C\cdot (\nu ϵ+\frac{1}{l}\sum _{i=1}^l({\xi }_i+\underset{i}{\overset{\ast }{\xi }}))\}$$$$\text{Subject to :}$$$${\omega }^T\cdot x_i+b-y_i\le ϵ+{\xi }_i$$$$y_i-{\omega }^T\cdot x_i-b\le ϵ+{\xi }_i^{\ast }$$$${\xi }_i,{\xi }_i^{\ast }\ge 0\text{,}i=1,2,\dots ,l,ϵ\ge 0\text{,}$$

where ${\xi }_i,{\xi }_i^{\ast }$ are two slack variables. The constant $C>0$ determines the trade-off between the flatness of $f$ and the amount up to which deviations larger than $ϵ$ are tolerated. $\nu \in (0,1]$ is a constant which controls the number of support vectors. In the $\nu \text{-}\mathrm{SV\; R}$ the size of $ϵ$ is not given a priori but a variable. Its value is traded off against the model complexity and slack variables via a constant $\nu$ (Chalimourda et al., 2004). This corresponds to dealing with a so-called $ϵ$-insensitive loss function (Cortes and Vapnik, 1995, Vapnik, 1995) described by

$$c_{ϵ}\left(\xi \right)={\left|\xi \right|}_{ϵ}=\{\begin{array}{cc}0\text{,}\hfill & \text{if }\left|\xi \right|\le ϵ\text{,}\hfill \\ \left|\xi \right|-ϵ\text{,}\hfill & \text{otherwise .}\hfill \end{array}$$

In 2002, $ϵ\text{-}\mathrm{SV\; R}$ for a general noise model was proposed in Schölkopf and Smola (2002):

$$min\{g_{ϵ\text{-}\mathrm{SV\; R}}=\frac{1}{2}{\Vert \omega \Vert }^2+C\cdot (\sum _{i=1}^l\tilde{c}\left({\xi }_i\right)+\tilde{c}\left(\underset{i}{\overset{\ast }{\xi }}\right))\}$$$$\text{Subject to :}$$$${\omega }^T\cdot x_i+b-y_i\le ϵ+{\xi }_i$$$$y_i-{\omega }^T\cdot x_i-b\le ϵ+{\xi }_i^{\ast }$$$${\xi }_i,{\xi }_i^{\ast }\ge 0\text{,}i=1,2,\dots ,l\text{,}$$

where $c(x,y,f\left(x\right))=\tilde{c}\left({|y-f\left(x\right)|}_{ϵ}\right)$ is a general convex loss function in the sample point $(x_i,y_i)$ of $D_l$. ${|y-f\left(x\right)|}_{ϵ}$ in (5) is Vapnik’s $ϵ$-insensitive loss function.

Using Lagrange multiplier techniques (Cortes and Vapnik, 1995, Vapnik, 1995), Problem (4) can be transformed to a convex optimization problem with a global minimum. At the optimum, the regression estimate takes the form $f\left(x\right)={\sum }_{i=1}^l({\alpha }_i^{\ast }-{\alpha }_i)(x_i\cdot x)+b$, where $(x_i\cdot x)$ is the inner product.

In 2002, Bofinger, Luig, and Beyer (2002) found that the output of wind turbine systems is limited between zero and the maximum power and the error statistics do not follow a normal distribution. In 2005, Fabbri, Román, Abbad, and Quezada (2005) believed that the normalized produced power $p$ must be within the interval $[0,1]$ and the beta function is more appropriate to fit the error than the standard normal distribution function. Bludszuweit, Antonio, and Llombart (2008) showed the advantages of using the beta probability distribution function (PDF), instead of the Gaussian PDF, for approximating the forecast error distribution. The error $ϵ$ between the predicted values $x_p$ and the measured values $x_m$ obeys the beta distribution in the forecast of wind power, and the PDF of $ϵ$ is $f\left(ϵ\right)={ϵ}^{m-1}\cdot {(1-ϵ)}^{n-1}\cdot h,ϵ\in (0,1)$, the parameters $m$ and $n$ are often called hyperparameters because they control the distribution of the variable $ϵ$ ($m>1,n>1$), $h$ is the normalization factor and parameters $m$ and $n$ are determined by the values of the mean (which is the predicted power) and the standard deviation (Bishop, 2006, Canavos, 1984). Fig. 1 shows plots of Gaussian distribution and the beta distribution for different values of hyperparameters. In 2007, Zhang, Wan, Zhao, and Yang (2007) and Randazzo, Abou-Khousa, Pastorino, and Zoughi (2007) presented the estimation results under a Laplacian noise environment in the direction-of-arrival of coherent electromagnetic waves impinging estimation problem. Laplace distribution is frequently encountered in various machine learning areas, e.g., the over-complete wavelet transform coefficients of images, processing in Natural images, etc. (Eltoft et al., 2006, Park and Lee, 2005).

Based on the above analysis, we know that the error distributions do not satisfy Gaussian distribution in some real-world applications. We try to study the optimal loss functions for different error models.

It is not suitable to apply the GN-SVR to fit functions from data with non-Gaussian noise. In order to solve the above problems, we derive a general loss function and construct $\nu$-support vector regression machines for a general noise model.

Finally, we design a technique to find the optimal solution to the corresponding regression tasks. While there are a large number of implementations of SVR algorithms in the past years, we introduce the Augmented Lagrange Multiplier (ALM) method, presented in Section  4. If the task is non-differentiable or discontinuous, the sub-gradient descent method can be used (Ma, 2010), and if there are very large scale of samples, SMO can also be used (Shevade, Keerthi, Bhattacharyya, & Murthy, 2000).

The main contributions of our work are listed as follows: (1) we derive the optimal loss functions for different error models by the use of Bayesian approach and optimization theory; (2) we develop the uniform $\nu$-support vector regression model for the general noise with inequality constraints ($N\text{-}\mathrm{SV\; R}$); (3) the Augmented Lagrange Multiplier method is applied to solve $N\text{-}\mathrm{SV\; R}$, which guarantees the stability and validity of the solution in $N\text{-}\mathrm{SV\; R}$; (4) we utilize $N\text{-}\mathrm{SV\; R}$ to short-term wind speed prediction and show the effectiveness of the proposed model in practical applications.

This paper is organized as follows: in Section  2 we derive the optimal loss function corresponding to a noise model by using the Bayesian approach; in Section  3 we describe the proposed $\nu$-support vector regression technique for general noise model ($N\text{-}\mathrm{SV\; R}$); in Section  4 we give the solution and algorithm design of $N\text{-}\mathrm{SV\; R}$; numerical experiments are conducted on artificial data sets, UCI data and short-term wind speed prediction in Sections  5 Experimental analysis, 6 Short-term wind speed prediction with the proposed algorithm; finally, we conclude the work in Section  7.