Software reliability forecasting by support vector machines with simulated annealing algorithms

https://doi.org/10.1016/j.jss.2005.02.025Get rights and content

Abstract

Support vector machines (SVMs) have been successfully employed to solve non-linear regression and time series problems. However, SVMs have rarely been applied to forecasting software reliability. This investigation elucidates the feasibility of the use of SVMs to forecast software reliability. Simulated annealing algorithms (SA) are used to select the parameters of an SVM model. Numerical examples taken from the existing literature are used to demonstrate the performance of software reliability forecasting. The experimental results reveal that the SVM model with simulated annealing algorithms (SVMSA) results in better predictions than the other methods. Hence, the proposed model is a valid and promising alternative for forecasting software reliability.

Introduction

In general, four phases are contained in a software development process (Myers, 1976): specification, design, coding and testing. Software reliability deals with behavior of a software system and is defined as the probability of working without failure for a specified interval of time. For software reliability forecasting, the testing phase has received increasing attention due to high cost for simulating user environment. Therefore, an appropriate forecasting model is important in software reliability forecasting. In the last few decades, many methods, such as times-between-failures model, non-homogeneous Poisson process model (NHPP), Markov processes, operational-profile model, and Kalman filter model, had been developed in forecasting software reliability (Cai, 1998, Kimura et al., 1995, Lyu, 1996, Musa et al., 1987, Singpurwalla and Soyer, 1985, Xie, 1991). Recently, along with the development of statistical theory, Bayesian approach has been applied in forecasting software reliability. El-Aroui and Soler (1996) first employed Bayesian statistical model to predict software reliability. They assumed the successive times between software failures follow the exponential distribution. The numerical examples illustrate the proposed model is useful for simulated software failure data. Pham and Pham, 2000, Pham and Pham, 2001 applied Bayesian approach to predict software reliability. They presumed the times between software failures follow the Weibull distribution, then used “hazard function” and “pseudo-failure” to predict the distribution of times interval. The empirical results indicate that proposed models outperform other times-between-failures models in terms of the sum of square errors (SSE). In addition, Pham and Zhang (2003) proposed a software reliability forecasting model based on NHPP approaches. The experimental results indicate that proposed models have better goodness-of-fit than other exist NHPP models. Two forecasting models, namely the super model and the ARIMA model, were developed by Burtschy et al. (1997) to forecast software reliability. The numerical results show that proposed models provide more accurate forecasting results than the other forecasting models. Brocklehurst et al. (1990) presented a recalibration method to predict software reliability. They recalculated the parameters during trial and error phases to improve forecasting accuracy. However, those forecasting approaches are insufficient while the data patterns of software reliability appear to be non-linear.

Due to the general non-linear mapping capabilities, artificial neural networks have been applied in time series forecasting. However, the literature on the application of artificial neural networks to forecast reliability is very limited. Liu et al. (1995) applied counter-propagation and back-propagation neural network models to estimate parameters of reliability distribution with a small data set. The experimental results indicate that the proposed models improve the accuracy of reliability forecasting. Sitte (1999) employed neural network technology to predict software-reliability-growth, two neural networks, namely the fee-forward- network and the Elman-recurrent-network are used in this investigation. The input is the normalized failure-occurrence time; the output is the accumulated failure number. Empirical results show that these two neural networks outperform the parametric-recalibration models in terms of forecasting accuracy. Cai et al. (2001) proposed a multi-layer perception neural network to forecast software reliability. The authors evaluated the performance of neural network models by different network architectures. They claimed that neural network models are better suited for a smooth reliability data set than a fluctuating one. Su et al. (1997) proposed an ICBPNN forecasting model (input-combined back-propagation neural networks), which combines time series with neural network methods to predict engine reliability. The ICBPNN has a greater forecasting accuracy than ARIMA models and BPNN models. Amjady and Ehsan (1999) developed a neural-network-based expert system to evaluate the reliability of power systems. The proposed systems overcome certain difficulties, including low accuracy, complex modeling and heavy computations. Ho et al. (2002) undertook comparative analysis of neural networks and ARIMA techniques to forecast reparable systems. Their experimental results demonstrate that both recurrent neural networks and the ARIMA approach are superior to multi-layer feed-forward neural networks in terms of forecasting accuracy. Xu et al. (2003) applied feed-forward multi-layer perceptron neural networks and radial basis function (RBF) neural networks to forecast the reliability of engine systems. The experimental results show that the neural network approaches provide more accurate forecasting results than the ARIMA model.

Support vector machines (SVMs) have been widely employed to deal with non-linear forecasting problems. Originally, SVMs are developed to solve pattern recognition problems. Recently, the introduction of Vapnik’s ε-insensitive loss function has enabled SVMs to be extended to solve non-linear regression estimation problems. SVMs have been successfully applied to solve forecasting problems in many fields. Tay and Cao (2001) employed SVMs to forecast financial time series. Their numerical results reveal that SVMs outperform a multi-layer back-propagation neural network in forecasting financial time series. Wang et al. (2003) applied SVMs to predict air quality. Their experimental results show that the SVMs outperform conventional Radial Basis Function networks. Cao and Gu (2002) proposed a dynamic SVM model to solve non-stationary time series problems. Their experimental results demonstrate that DSVMs outperform standard SVMs in forecasting non-stationary time series. In the same year, Tay and Cao (2002) presented a C-ascending SVM to model non-stationary financial time series. Their experimental results indicate that C-ascending SVMs with actually ordered sample data consistently outperform standard SVMs. Cao (2003) addressed expert SVMs to forecast time series. Generalized expert SVMs contain a two-stage neural network architecture. The numerical results show that the expert SVMs provide better generalization than single SVM models. Pai and Lin (in press a) employed SVMs to forecast the production value of the machinery industry in Taiwan. They report that SVMs outperform the seasonal ARIMA model and the general regression neural networks model. Pai and Lin (in press b) proposed a hybrid model with the strength of an ARIMA model and the SVMs model in forecasting stock prices problems. By using ten stocks to examine the performance, their numerical results indicated that the proposed hybrid model outperforms other forecasting methods. Mohandes et al. (2004) applied SVMs to predict wind speed. Their experimental results show that the SVM model yields lower root mean square errors than multi-layer perceptron neural networks.

The selection of three positive parameters, σ, ε and C, of a SVM model is important to the accuracy of forecasting. However, structural methods for confirming efficiently the selection of parameters efficiently are lacking. Therefore, simulated annealing algorithm (SA) is used in the proposed SVM model to optimize parameter selection. The rest of this article is organized as follows. Section 2 introduces the basic SVM model. The proposed SVMSA model is presented in Section 3. Section 4 addresses two numerical examples taken from the literature to demonstrate the forecasting capacity of the proposed model. Finally, conclusions are made in Section 5.

Section snippets

Support vector machine model

Vapnik proposed support vector machines based on the structured risk minimization (SRM) principle (Vapnik, 1995). Rather than finding empirical errors, SVMs aim to minimize an upper bound of the generalization error. They also generate a regression function by applying a set of high dimensional linear functions. Hence, given a set of data G={(xi,di)}i=1N (where xi is the input vector; di is the actual value, and N is the total number of data patterns), the regression function is formulated as

Simulated annealing algorithms for selecting parameters in the SVM model

The simulated annealing algorithm is an optimization technique, analogous to the annealing process of material physics. Boltzmann pointed out, in 1877, if the system is in thermal equilibrium at a temperature T, then the probability PT(s) of the system being in a given state s is given by the Boltzmann distribution:PT(s)=exp(-E(s)/kT)wSexp(-E(w)/kT)where E(s) denotes the energy of state s; k represents the Boltzmann constant and S is the set of all possible states. However, Eq. (9) does not

Numerical examples and simulation results

In this study, two numerical examples are used to demonstrate the forecasting performance of the presented model. A rolling-based forecasting procedure and a one-step-ahead forecasting policy are applied in both examples. The software inter-failure times taken from a telemetry network system by AT&T Bell Laboratories (Pham and Pham, 2000, Pham and Pham, 2001), are used as the first numerical example to evaluate the software reliability forecasting performance. Appendix A lists the original

Conclusions

In the past decades, many software reliability forecasting models have been proposed. However, no single model has been suitable in all circumstance. In this investigation, a SVMSA model is proposed as another viable alternative to forecast software reliability. A comparative study of the forecasting performance of various models is evaluated. Results from two case studies indicate that SVMSA models provide more accurate forecasting results than the other forecasting models. The superior

Acknowledgment

This research was conducted with the support of National Science Council (NSC 92-2213-E-212-001 and NSC 93-2213-E-212-001).

Ping-Feng Pai is currently an associate professor in the Department of Information Management, National Chi Nan University, Taiwan, R.O.C. He received his Ph.D. in Industrial Engineering from Kansas State University in 1998. His research interests are forecasting techniques, neural networks, and evolutionary algorithms.

References (37)

  • N. Amjady et al.

    Evaluation of power systems reliability by artificial neural network

    IEEE Transaction on Power Systems

    (1999)
  • S. Brocklehurst et al.

    Recalibrating software reliability models

    IEEE Transactions on Software Engineering

    (1990)
  • K.Y. Cai

    Software defect and operational profile modeling

    (1998)
  • L. Cao et al.

    Dynamic support vector machines for non-stationary time series forecasting

    Intelligent Data Analysis

    (2002)
  • A. Dekkers et al.

    Global optimization and simulated annealing

    Mathematical Programming

    (1991)
  • M.A. El-Aroui et al.

    A Bayes nonparametric framework for software-reliability analysis

    IEEE Transactions on Reliability

    (1996)
  • S. Kirkpatrick et al.

    Optimization by simulated annealing

    Science

    (1983)
  • M.C. Liu et al.

    An exploratory study of a neural network approach for reliability data analysis

    Quality and Reliability Engineering International

    (1995)
  • Cited by (0)

    Ping-Feng Pai is currently an associate professor in the Department of Information Management, National Chi Nan University, Taiwan, R.O.C. He received his Ph.D. in Industrial Engineering from Kansas State University in 1998. His research interests are forecasting techniques, neural networks, and evolutionary algorithms.

    Wei-Chiang Hong is currently a Ph.D. student in the School of Management, Da-Yeh University, Taiwan, R.O.C. His research interesting is the application of forecasting techniques.

    View full text