Multiobjective analysis of chaotic dynamic systems with sparse learning machines

https://doi.org/10.1016/j.advwatres.2005.05.011Get rights and content

Abstract

Sparse learning machines provide a viable framework for modeling chaotic time-series systems. A powerful state-space reconstruction methodology using both support vector machines (SVM) and relevance vector machines (RVM) within a multiobjective optimization framework is presented in this paper. The utility and practicality of the proposed approaches have been demonstrated on the time series of the Great Salt Lake (GSL) biweekly volumes from 1848 to 2004. A comparison of the two methods is made based on their predictive power and robustness. The reconstruction of the dynamics of the Great Salt Lake volume time series is attained using the most relevant feature subset of the training data. In this paper, efforts are also made to assess the uncertainty and robustness of the machines in learning and forecasting as a function of model structure, model parameters, and bootstrapping samples. The resulting model will normally have a structure, including parameterization, that suits the information content of the available data, and can be used to develop time series forecasts for multiple lead times ranging from two weeks to several months.

Introduction

Chaotic systems are nonlinear, dynamic, fully deterministic, highly sensitive to initial conditions, and can be modeled using state-space reconstruction via a time delay embedding theorem [40]. Though Poincare described chaotic behavior in nonlinear systems in the late 1880s, a resurgence in chaos theory arose with Lorenz [50] in his work with weather prediction models wherein he discovered that nonlinear models could be exponentially divergent (i.e., sensitive to small differences in initial conditions) [71], [70], [68], [24]. Thus, unlike many other processes the erratic data produced from chaos are due to complex outcome of a nonlinear system and initial conditions that are identified from uncertain data rather than only intrinsic randomness. The behaviors of many water resources systems have been observed to be chaotic and thus chaos has received significant attention in hydrology.

Chaos theory states that the time series itself carries enough information about the behavior of the system to carry out forecasting [50]. Therefore, a deterministic chaotic system behaves in the future in a similar manner as in the past. The embedding theorem emerged in the light of chaos theory, which states that given a recognized state-space representation of a chaotic time series, through estimation of the time delay and the embedding dimension (i.e., state-space reconstruction) a full knowledge of the system behavior is guaranteed [72]. Nonetheless, the time series must: be sampled at sufficient resolution, not be corrupted by noise, and measured over a long period of time [59] in order to avoid the biases of many state-space reconstruction techniques. In addition, the state evolution of a chaotic system is dynamic and constitutes an inverse problem for which there is no unique solution, and for which there might be no stable solution, either.

Capturing the behavior of a chaotic time series becomes more complicated in the presence of noise (i.e., background noise, or an inaccuracy of the measurements of system behavior) [67]. The process of measuring system states using physical sensors, in addition to the lack or neglect of exogenous stresses, introduces some amount of noise [25]. This noise causes some uncertainties in both the model structure and, accordingly, in predictions about the future performance of the system. Contamination with noise is almost inherent in any hydrological time series. This, in essence, runs counter to many widely used methodologies that are based on theories developed on assumptions of infinite and noise-free time series [67]. Moreover, the structure of the hydrological processes exhibits temporal, spatial, and scale variability. A failure to account for the underlying system structure limits the ability of the modeling approaches to identify a unique mathematical representation of the hydrological processes [67]. This translated to an impediment to both traditional state-space forecasting methodologies and learning machines to predict future system behaviors with confidence. In light of these modeling issues, a principal objective of this paper is to quantify the amount of uncertainty introduced in the analysis of complex hydrological processes by the specification of model structure.

From a pragmatic engineering point of view, state-space reconstruction techniques have shortcomings that can be attributed to the fact that their prediction accuracy is often inadequate since the state-space parameters are not derived with the intention of minimizing the prediction error, but instead are developed to characterize the nonlinear dynamic process in question [12], [85], [59].

In this sense, the other objective of this paper is to link the powerful state-space reconstruction methodology via exploiting the appealing regularization concepts of both support vector machines (SVM) and relevance vector machines (RVM) within a multiobjective optimization framework. The parameters of chaos theory and the unintuitive parameters of learning machines will be optimized with the assistance of a multiobjective shuffle complex evolution Metropolis algorithm (MOSCEM). The chosen objective functions will be optimized both independently and simultaneously. This will yield multiple feasible solutions accounting for the trade-off (e.g., bias-variance trade-off; trade-offs between seepage, precipitation, and evaporation induced signals) and moreover capture the uncertainty in the model structure.

In this manuscript, efforts will be made to assess the uncertainty and robustness of the machines in learning and forecasting as a function of model structure and bootstrapping samples. The proposed framework, using sparse learning techniques, allows for compact representations of system dynamics. In other words, models that are developed from the learning machines used here normally have a structure, including parameterization that suits the information content of the available data, and can be used to develop time series forecasts for multiple lead times. The goal of this paper is to introduce new learning machines in a multiobjective framework that identify a suite of model parameters and that consequently enable an ensemble forecast of time series. A theoretical background is first described. Then the framework utility is demonstrated by applying it to a Great Salt Lake (GSL) biweekly volume dataset.

Section snippets

Chaotic and nonlinear time series

Chaos occurs as a feature of orbits x(t) arising from systems of differential equations of dx(t)/dt = F(x(t)) with three or more degrees of freedom or invertible maps of x(t + 1) = F(x(t)). As a class of observable signals, x(t), chaos lies logically between the well-studied domain of predictable, regular, or quasi-periodic signal and the totally irregular stochastic signals [5]. In many systems the interaction between the underlying physical processes that are responsible for the evolution of system

Description of the study area

The Great Salt Lake (GSL) of Utah is the fourth largest terminal (i.e., has no outlet) lake in the world (http://ut.water.usgs.gov/greatsaltlake/). The GSL basin encompasses a drainage area of 89,000 km2 including much of Utah, parts of southeastern Idaho, and southwestern Wyoming (Fig. 2) [32]. The three rivers that drain into the GSL are the Bear, the Weber, and the Jordan which, in total, comprise about 66% of the average annual water inflow to the GSL; precipitation contributes about 31%,

Results and discussion

Experience with the forecasting of complex dynamical processes has shown that the resulting predictions always suffer from different sources of error. It is reasonable to speculate that any hydrological model can fall victim to errors resulting from missing processes and parameters, limited knowledge of the governing equations and laws underlying the processes (i.e., heuristic assumptions), errors in the measured data, approximations in the computations (e.g., numerical discretization),

Conclusions

The ability of both SVMs and RVMs to capture the behavior of a chaotic dynamical system from a single observable time series has been demonstrated. Both the SVM and RVM models provide an accurate forecast methodology that could be exploited for the planning and management of the GSL. The sparse machines are theoretically elegant and well-regularized. While SVMs rely on structural risk minimization to reach a sparse structure, RVMs integrate over the uncertainty of state estimates and capture

Acknowledgments

Portions of this work were supported by the Utah Water Research Laboratory, College of Engineering, Utah State University, and Utah Center for Water Resources Research. The authors would like to thank Dr. Wallace Gwynn of the Utah Geological Survey, Utah Department of Natural Resources, for providing the biweekly GSL stage and volume data used in the study. Thanks are also due to anonymous reviewers for their insightful comments.

References (87)

  • Arnow T, Stephens D. Hydrologic characteristics of the Great Salt Lake, Utah: 1847–1986. US Geological Survey...
  • ASCE Task Committee on Application of the Artificial Neural Networks in Hydrology, Artificial neural networks in...
  • Asefa T. Statistical learning machine concepts and application in water resources management. PhD dissertation....
  • T. Asefa et al.

    Support vectors-based groundwater head observation networks design

    Water Resour Res

    (2004)
  • Baskin RL, Waddell KM, Thiros SA, Giddings EM, Hadley HK, Stephens DW, et al. Water-quality assessment of the Great...
  • J.O. Berger

    Statistical decision theory and Bayesian analysis

    (1985)
  • K. Beven

    Prophecy, reality, and uncertainty in distributed hydrologic modeling

    Adv Water Resour

    (1993)
  • B.P. Bezruchko et al.

    Reconstruction of time-delay systems from chaotic time series

    J Phys Rev E

    (2001)
  • B. Chun-Hua et al.

    Determining the minimum embedding dimension of nonlinear time series based on prediction method

    J Chin Phys

    (2004)
  • C. Cortes et al.

    Support vector networks

    J Mach Learning

    (1995)
  • R.L. David et al.

    Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation

    Water Resour Res

    (1999)
  • Q.A. Duan et al.

    Effective and efficient global optimization for conceptual rainfall-runoff models

    Water Resour Res

    (1992)
  • Q.A. Duan et al.

    Shuffle complex evolution approach for effective and efficient global minimization

    J Optim Theory Applic

    (1993)
  • J.D. Farmer et al.

    Predicting chaotic time series

    Phys Rev Lett

    (1987)
  • A.M. Frazer et al.

    Independent coordinates for strange attractors from mutual information

    Phys Rev A

    (1996)
  • J. Friedman

    Multivariate adaptive regression splines

    Ann Stat

    (1991)
  • T.Y. Gan et al.

    Automatic calibration of conceptual rainfall-runoff models: optimization algorithms, catchments conditions, and model structure

    Water Resour Res

    (1996)
  • S. Geman et al.

    Neural networks and the bias/variance dilemma

    Neural Comput

    (1992)
  • Gleick J. Chaos: making a new science. Viking Harmondsworth Middlesex, England,...
  • M.S. Grewal et al.

    Kalman filtering: theory and practice using MATLAB

    (2001)
  • H.V. Gupta et al.

    Parameter estimation of land surface scheme using multi-criteria methods

    J Geophys Res

    (1999)
  • H.V. Gupta et al.

    Multiple criteria global optimization for watershed model calibration

  • H.V. Gupta et al.

    Toward improved calibration of hydrologic models: multiple and noncommensurable measures of information

    Water Resour Res

    (1998)
  • Y. Ibrahim et al.

    Calibration strategy for urban catchment parameters

    J Hydr Eng

    (1992)
  • Johnson B, Tarboton DG. Great Salt Lake Basin hydrologic observatory prospectus. Submitted to CUAHSI for consideration...
  • M.I. Jordan et al.

    Neural networks

  • V. Kecman

    Learning and soft computing: support vector machines, neural networks, and fuzzy logic models

    (2001)
  • M.B. Kennel et al.

    Determining embedding dimension for phase-space reconstruction using a geometrical construction

    Phys Rev Lett A

    (1992)
  • I.M. Khadam et al.

    Use of soft information to describe the relative uncertainty of calibration data in hydrologic models

    Water Resour Res

    (2004)
  • A.F. Khalil et al.

    Applicability of statistical learning algorithms in ground water quality modeling

    Water Resour Res

    (2005)
  • Khalil AF. Computational learning theory and data-driven modeling for ware resources management and hydrology. PhD...
  • A.F. Khalil et al.

    Basin-Scale water management and forecasting using multisensor data and neutral networks

    J Am Water Resour Assoc

    (2005)
  • D. Koutsoyiannis et al.

    Deterministic chaos versus stochasticity in analysis and modeling of point rainfall series

    J Geophys Res

    (1996)
  • Cited by (67)

    • Concepts and methodologies for agricultural water management

      2020, Agricultural Water Management: Theories and Practices
    • Projection of climate change impacts on precipitation using soft-computing techniques: A case study in Zayandeh-rud Basin, Iran

      2016, Global and Planetary Change
      Citation Excerpt :

      This method incorporates the effect of serial correlation in detection of the trend statistic. In this method, the original projected precipitation is resampled in predetermined blocks for a large number of times to estimate the significance of the Mann-Kendall (MK) test statistic S. Briefly, the method involves the following steps (Khalil et al., 2006): (i) estimate the test statistic of the selected trend identification test (e.g. rSRC) from the original projected precipitation time series, (ii) estimate the number of significant serial correlations k at αL level, (iii) resample precipitation time series in blocks of k + η for a large number of times and estimate the trend identification test static for each simulated time series to produce a simulated distribution of the test statistic, (iv) estimate the significance of the observed test statistic in step 1 from the simulated distribution in step 3. More details about the BBS method is found in Khalil et al. (Khalil et al., 2006) and (Önöz and Bayazit, 2012).

    View all citing articles on Scopus
    View full text