Skip to main content

2012 | Buch

Learning in Non-Stationary Environments

Methods and Applications

herausgegeben von: Moamar Sayed-Mouchaweh, Edwin Lughofer

Verlag: Springer New York

insite
SUCHEN

Über dieses Buch

Recent decades have seen rapid advances in automatization processes, supported by modern machines and computers. The result is significant increases in system complexity and state changes, information sources, the need for faster data handling and the integration of environmental influences. Intelligent systems, equipped with a taxonomy of data-driven system identification and machine learning algorithms, can handle these problems partially. Conventional learning algorithms in a batch off-line setting fail whenever dynamic changes of the process appear due to non-stationary environments and external influences.

Learning in Non-Stationary Environments: Methods and Applications offers a wide-ranging, comprehensive review of recent developments and important methodologies in the field. The coverage focuses on dynamic learning in unsupervised problems, dynamic learning in supervised classification and dynamic learning in supervised regression problems. A later section is dedicated to applications in which dynamic learning methods serve as keystones for achieving models with high accuracy.

Rather than rely on a mathematical theorem/proof style, the editors highlight numerous figures, tables, examples and applications, together with their explanations.

This approach offers a useful basis for further investigation and fresh ideas and motivates and inspires newcomers to explore this promising and still emerging field of research.

Inhaltsverzeichnis

Frontmatter
Chapter 1. Prologue
Abstract
This introductory chapter intends to provide a general overview about the most essential requirements, demands and challenges with respect to dynamic learning of data-driven models in non-stationary environments and applications. It outlines the main lines of research investigated during the last decade in order to cope with the requirements, inter alia to handle high system dynamics, online data streams recorded with a high frequency, drifting system states and very large data bases within fast sample-wise and single-pass model updates conducted on-the-fly and in incremental manner. The last part of this chapter outlines a compact summary of the contents of the book by providing a paragraph about each of the single contributions.
Moamar Sayed-Mouchaweh, Edwin Lughofer

Dynamic Methods for Unsupervised Learning Problems

Frontmatter
Chapter 2. Incremental Statistical Measures
Abstract
Statistical measures provide essential and valuable information about data and are needed for any kind of data analysis. Statistical measures can be used in a purely exploratory context to describe properties of the data, but also as estimators for model parameters or in the context of hypothesis testing. For example, the mean value is a measure for location, but also an estimator for the expected value of a probability distribution from which the data are sampled. Statistical moments of higher order than the mean provide information about the variance, the skewness, and the kurtosis of a probability distribution. The Pearson correlation coefficient is a measure for linear dependency between two variables. In robust statistics, quantiles play an important role, since they are less sensitive to outliers. The median is an alternative measure of location, the interquartile range an alternative measure of dispersion. The application of statistical measures to data streams requires online calculation. Since data come in step by step, incremental calculations are needed to avoid to start the computation process each time new data arrive and to save memory so that not the whole data set needs to be kept in the memory. Statistical measures like the mean, the variance, moments in general, and the Pearson correlation coefficient render themselves easily to incremental computations, whereas recursive or incremental algorithms for quantiles are not as simple or obvious. Nonstationarity is another important aspect of data streams that needs to be taken into account. This means that the parameters of the underlying sampling distribution might change over time. Change detection and online adaptation of statistical estimators is required for nonstationary data streams. Hypothesis tests like the χ2- or the t-test can be a basis for change detection, since they can also be calculated in an incremental fashion. Based on change detection strategies, one can derive information on the sampling strategy, for instance the optimal size of a time window for parameter estimations of nonstationary data streams.
Katharina Tschumitschew, Frank Klawonn
Chapter 3. A Granular Description of Data: A Study in Evolvable Systems
Abstract
A human-centric way of data analysis, especially when dealing with data distributed in space and time, is concerned with data representation in an interpretable way where a perspective from which the data are analyzed is actively established by the user. Being motivated by this essential feature of data analysis, in the study we present a granular way of data analysis where the data and relationships therein are described through a collection of information granules defined in the spatial and temporal domain. We show that the data, expressed in a relational fashion, can be effectively described through a collection of Cartesian products of information granules forming a collection of semantically meaningful data descriptors. The design of the codebooks (vocabularies) of such information granules used to describe the data is guided through a process of information granulation and degranulation. This scheme comes with a certain performance index whose minimization becomes instrumental in the optimization of the codebooks. A description of logical relationships between elements of the codebooks used in the granular description of spatiotemporal data present in consecutive time frames is elaborated on as well.
Witold Pedrycz, John Berezowski, Iqbal Jamal
Chapter 4. Incremental Spectral Clustering
Abstract
In the present contribution, a novel algorithm for off-line spectral clustering algorithm is introduced and an online extension is derived in order to deal with sequential data. The proposed algorithm aims at dealing with nonconvex clusters having different forms. It relies on the notion of communicability that allows to handle the contiguity of data distribution. In the second part of the paper, an incremental extension of the fuzzy c-varieties is proposed to serve as a building block of the incremental spectral clustering algorithm (ISC). Initial simulations are presented towards the end of the contribution to show the performance of the ISC algorithm.
Abdelhamid Bouchachia, Markus Prossegger

Dynamic Methods for Supervised Classi?cation Problems

Chapter 5. Semisupervised Dynamic Fuzzy K-Nearest Neighbors
Abstract
This chapter presents a semi-supervised dynamic classification method to deal with the problem of diagnosis of industrial evolving systems. Indeed, when a functioning mode evolves, the system characteristics change and the observations, i.e. the patterns representing observations in the feature space, obtained on the system change too. Thus, each class membership function must be adapted to take into account these temporal changes and to keep representative patterns only. This requires an adaptive method with a mechanism for adjusting its parameters over time. The developed approach is named Semi-Supervised Dynamic Fuzzy K-Nearest Neighbors (SS-DFKNN) and comprises three phases: a detection phase to detect and confirm classes evolutions, an adaptation phase realized incrementally to update the evolved classes parameters and to create new classes if necessary and a validation phase to keep useful classes only. To illustrate this approach, the diagnosis of a welding system is realized to detect the weldings quality (good or bad), based on acoustic noises issued of weldings operations.
Laurent Hartert, Moamar Sayed-Mouchaweh
Chapter 6. Making Early Predictions of the Accuracy of Machine Learning Classifiers
Abstract
The accuracy of machine learning systems is a widely studied research topic. Established techniques such as cross validation predict the accuracy on unseen data of the classifier produced by applying a given learning method to a given training data set. However, they do not predict whether incurring the cost of obtaining more data and undergoing further training will lead to higher accuracy. In this chapter, we investigate techniques for making such early predictions. We note that when a machine learning algorithm is presented with a training set the classifier produced, and hence its error, will depend on the characteristics of the algorithm, on training set’s size, and also on its specific composition. In particular we hypothesize that if a number of classifiers are produced, and their observed error is decomposed into bias and variance terms, then although these components may behave differently, their behavior may be predictable. Experimental results confirm this hypothesis, and show that our predictions are very highly correlated with the values observed after undertaking the extra training. This has particular relevance to learning in nonstationary environments, since we can use our characterization of bias and variance to detect whether perceived changes in the data stream arise from sampling variability or because the underlying data distributions have changed, which can be perceived as changes in bias.
James Edward Smith, Muhammad Atif Tahir, Davy Sannen, Hendrik Van Brussel
Chapter 7. Incremental Classifier Fusion and Its Applications in Industrial Monitoring and Diagnostics
Abstract
Pattern recognition techniques have shown their usefulness for monitoring and diagnosing many industrial applications. The increasing production rates and the growing databases generated by these applications require learning techniques that can adapt their models incrementally, without revisiting previously used data. Ensembles of classifiers have been shown to improve the predictive accuracy as well as the robustness of classification systems. In this work, several well-known classifier fusion methods (Fuzzy Integral, Decision Templates, Dempster–Shafer Combination, and Discounted Dempster–Shafer Combination) are extended to allow incremental adaptation. Additionally, an incremental classifier fusion method using an evolving clustering approach is introduced—named Incremental Direct Cluster-based ensemble. A framework for strict incremental learning is proposed in which the ensemble and its member classifiers are adapted concurrently. The proposed incremental classifier fusion methods are evaluated within this framework for two industrial applications: online visual quality inspection of CD imprints and prediction of maintenance actions for copiers from a large historical database.
Davy Sannen, Jean-Michel Papy, Steve Vandenplas, Edwin Lughofer, Hendrik Van Brussel
Chapter 8. Instance-Based Classification and Regression on Data Streams
Abstract
In order to be useful and effectively applicable in dynamically evolving environments, machine learning methods have to meet several requirements, including the ability to analyze incoming data in an online, incremental manner, to observe tight time and memory constraints, and to appropriately respond to changes of the data characteristics and underlying distributions. This paper advocates an instance-based learning algorithm for that purpose, both for classification and regression problems. This algorithm has a number of desirable properties that are not, at least not as a whole, shared by currently existing alternatives. Notably, our method is very flexible and thus able to adapt to an evolving environment quickly, a point of utmost importance in the data stream context. At the same time, the algorithm is relatively robust and thus applicable to streams with different characteristics.
Ammar Shaker, Eyke Hüllermeier

Dynamic Methods for Supervised Regression Problems

Frontmatter
Chapter 9. Flexible Evolving Fuzzy Inference Systems from Data Streams (FLEXFIS++)
Abstract
Data streams are usually characterized by an ordered sequence of samples recorded and loaded on-line with a certain frequency arriving continuously over time. Extracting models from such type of data within a reasonable on-line computational performance can be only achieved by a training procedure which is able to incrementally build up the models, ideally in a single-pass fashion (not using any prior samples). This chapter deals with data-driven design of fuzzy systems which are able to handle sample-wise loaded data within a streaming context. These are called flexible evolving fuzzy inference systems (FLEXFIS) as they may permanently change their structures and parameters with newly recorded data, achieving maximal flexibility according to new operating conditions, dynamic system behaviors, or exceptional occurrences. We are explaining how to deal with parameter adaptation and structure evolution on demand for regression as well as classification problems. In the second part of the chapter, several key extensions of the FLEXFIS family will be described (leading to the FLEXFIS + + and FLEXFIS-Class + + variants), including concepts for on-line rule merging, dealing with drifts, dynamically reducing the curse of dimensionality, as well as interpretability considerations and reliability in model predictions. Successful applications of the FLEXFIS family are summarized in a separate section. An extensive evaluation of the proposed methods and techniques will be demonstrated in a separate chapter (Chap. 14), when dealing with the application of flexible fuzzy systems in on-line quality-control systems.
Edwin Lughofer
Chapter 10. Sequential Adaptive Fuzzy Inference System for Function Approximation Problems
Abstract
In the classic approaches to design a fuzzy inference system, the fuzzy rules are determined by a domain expert a priori and then they are maintained unchanged during the learning. These fixed fuzzy rules may not be appropriate in real-time applications where the environment or model often meets unpredicted disturbances or damages. Hence, poor performance may be observed. In comparison to the conventional methods, fuzzy inference systems based on neural networks, called fuzzy-neural systems, have begun to exhibit great potential for adapting to the changes by utilizing the learning ability and adaptive capability of neural networks. Thus, a fuzzy inference system can be built using the standard structure of neural networks. Nevertheless, the determination of the number of fuzzy rules and the adjustment of the parameters in the if-then fuzzy rules are still open issues. A sequential adaptive fuzzy inference system (SAFIS) is developed to determine the number of fuzzy rules during learning and modify the parameters in fuzzy rules simultaneously. SAFIS uses the concept of influence of a fuzzy rule for adding and removing rules during learning. The influence of a fuzzy rule is defined as its contribution to the system output in a statistical sense when the input data is uniformly distributed. When there is no addition of fuzzy rules, only the parameters of the “closest” (in a Euclidean sense) rule are updated using an extended Kalman filter (EKF) scheme. The performance of SAFIS is evaluated based on some function approximation problems, via, nonlinear system identification problems and a chaotic time-series prediction problem. Results indicate that SAFIS produces similar or better accuracies with lesser number of rules compared to other algorithms.
Hai-Jun Rong
Chapter 11. Interval Approach for Evolving Granular System Modeling
Abstract
Physical systems change over time and usually produce considerable amount of nonstationary data. Evolving modeling of time-varying systems requires adaptive and flexible procedures to deal with heterogeneous data. Granular computing provides a rich framework for modeling time-varying systems using nonstationary granular data streams. This work considers interval granular objects to accommodate essential information from data streams and simplify complex real-world problems. We briefly discuss a new class of problems emerging in data stream mining where data may be either singular or granular. Particularly, we emphasize interval data and interval modeling framework. Interval-based evolving modeling (IBeM) approach recursively adapts both parameters and structure of rule-based models. IBeM uses ∪-closure granular structures to approximate functions. In general, approximand functions can be time series, decision boundaries between classes, control, or regression functions. Essentially, IBeM accesses data sequentially and discards previous examples; incoming data may trigger structural adaptation of models. The IBeM learning algorithm evolves and updates rules quickly to track system and environment changes. Experiments using heterogeneous streams of meteorological and financial data are performed to show the usefulness of the IBeM approach in actual scenarios.
Daniel Leite, Pyramo Costa, Fernando Gomide

Applications of Learning in Non-stationary Environments

Chapter 12. Dynamic Learning of Multiple Time Series in a Nonstationary Environment
Abstract
This chapter introduces two distinct solutions to the problem of capturing the dynamics of multiple time series and the extraction of useful knowledge over time. As these dynamics would change in a nonstationary environment, the key characteristic of the methods is the ability to evolve their structure continuously over time. In addition, reviews of existing methods of dynamic single time series analysis and modeling such as the dynamic neuro-fuzzy inference system and the neuro-fuzzy inference method for transductive reasoning, which inspired the proposed methods, are presented. This chapter also presents a comprehensive evaluation of the performance of the proposed methods on a real-world problem, which consists of predicting movement of global stock market indexes over time.
Harya Widiputra, Russel Pears, Nikola Kasabov
Chapter 13. Optimizing Feature Calculation in Adaptive Machine Vision Systems
Abstract
A classifier’s accuracy substantially depends on the features that are utilized to characterize an input sample. The selection of a representative and—ideally—small set of features that yields high discriminative power is an important step in setting up a classification system. The features are a set of functions that transform the raw input data (an image in the case of machine vision systems) into a vector of real numbers. This transformation may be a quite complex algorithm, with lots of parameters to tune and consequently with much room for optimization. In order to efficiently use this additional room for optimizing the features, we propose an integrated optimization step that adapts the feature parameters in such a way that the separation of the classes in feature space is improved, thus reducing the number of misclassifications. Furthermore, these optimization techniques may be used to “shape” the decision boundary in such a way that it can be easily modeled by a classifier. After covering the relevant elements of the theory behind this automatic feature optimization process, we will demonstrate and assess the performance on two typical machine vision applications. The first one is a quality control task, where different types of defects need to be distinguished, and the second example is a texture classification problem as it appears in image segmentation tasks. We will show how the optimization process can be successfully applied in morphological and textural features that both offer a number of parameters to tune and select.
Christian Eitzinger, Stefan Thumfart
Chapter 14. Online Quality Control with Flexible Evolving Fuzzy Systems
Abstract
This chapter is dealing with the application of flexible evolving fuzzy systems (described in Chap.​ 9) in online quality-control systems and therefore also provides a complete evaluation of these on (noisy) real-world data sets. Hereby, we are tackling with two different types of quality-control applications:
  • The first one is based on visual inspection of production items and therefore can be seen as a postsupervision step whether items or parts of items are ok or not, laying the basis for sorting out of bad products and decreasing customers’ claims.
  • The second one is conducted directly during the production process as dealing with a plausibility analysis of process measurements (such as temperatures, pressures, etc.) and therefore opens the possibility of an early intervention for product improvement (internal correction or external reaction).
In both scenarios, permanent update of nonlinear fuzzy models/classifiers during online operation based on data streams is an essential issue in order to cope with changing system dynamics, range extensions of measurements and features, and the inclusion of new operating conditions (e.g., fault classes) on demand without requiring time-intensive retraining phases. In the result section of this chapter, we will explicitly highlight the performance gains achieved when using flexible evolving fuzzy systems (EFS) in both quality-control paths.
Edwin Lughofer, Christian Eitzinger, Carlos Guardiola
Chapter 15. Identification of a Class of Hybrid Dynamic Systems
Abstract
The behavior of hybrid dynamic systems (HDS) switches between several modes with different dynamics involving both discrete and continuous variables in the course of time. Their identification aims at finding an accurate model of the system dynamics based on its past inputs and outputs. The identification can be achieved by two steps: the clustering and the regression. The clustering step aims at the estimation of the mode (discrete state) of each input–output data point as well as the switching sequence among these modes. The regression step determines the sub-models controlling the dynamic (continuous states) in each mode. In Pattern Recognition (PR) methods, each mode is represented by a set of similar patterns forming restricted regions in the feature space, called classes. A pattern is a vector built from past inputs and outputs. In this chapter, we propose to use an unsupervised PR method to realize the clustering step of the identification of switched linear HDS. The determination of the number of modes as well as the switching sequence does not require any information in advance about the modes, for example, their distribution, their shape, …, or their number.
Moamar Sayed-Mouchaweh, Nadhir Messai, Omar Ayad, Sofiane Mazeghrane
Backmatter
Metadaten
Titel
Learning in Non-Stationary Environments
herausgegeben von
Moamar Sayed-Mouchaweh
Edwin Lughofer
Copyright-Jahr
2012
Verlag
Springer New York
Electronic ISBN
978-1-4419-8020-5
Print ISBN
978-1-4419-8019-9
DOI
https://doi.org/10.1007/978-1-4419-8020-5