Neural network design and model reduction approach for black box nonlinear system identification with reduced number of parameters

Abstract

In this paper a dedicated recurrent neural network design and a model reduction approach are proposed in order to improve the balance between complexity and quality of black box nonlinear system identification models. The proposed neural network design, based on a three-layers architecture, helps to reduce the number of parameters of the model after the training phase without any significant loss of estimation accuracy. Nevertheless, the proposed architecture remains sufficiently general to provide a wide range of models among the most encountered in the literature. This reduction, achieved by a convenient choice of the activation functions and the initial conditions of the synaptic weights, is developed in two steps. The first step is to train the proposed architecture under two reasonable assumptions. Then the recurrent three-layers neural network is transformed into a representation of two-layer with less number of neurons, that is, a significant reduced number of parameters. The constructed architecture provided models with reasonable reduced number of parameters with a convenient estimation accuracy. To validate the proposed approach, we identify the Wiener–Hammerstein benchmark nonlinear system proposed in SYSID2009 [1].

Introduction

Models are used in engineering fields for analysis, supervision, fault detection, prediction, estimation of unmeasurable variables, optimization and model-based control process [2], [3], [4], [5], [6], [7], [8], [9], [10]. Therefore, accurate process representations with a reasonable low complexity are naturally required [4]. In the sequel, we shall denote for greatest convenience, the skillful balance between accuracy and complexity by model quality [11].

Basically, a model can be constructed along two routes or a combination of them [12]. One route is called physical modeling: this approach is based on the physical mechanisms that govern the system's behavior. The models achieved by this way are adequate approximations of the real process [11]. But, in many cases involving complex nonlinear systems, it is very difficult or impossible to derive dynamic models based on all the physical processes involved [4], [13], [14]. Numerous systems in the industrial fields show nonlinearities and uncertainties, which can profitably be considered as partially or totally black box [15].

Black-box models use general mathematical approximation functions in order to describe the systems input/output relations. One of the most important advantages of black box system identification techniques is the limited physical insights required to develop the model [16], but as a trade-off, these techniques imply the use of model structures that are as flexible as possible. Often, this variability leads to a high number of parameters [17]. Since the model quality is defined in terms of model complexity and its capabilities to reproduce the system behavior [11], the aim of this paper is to tackle this constraint with the design of a recurrent neural network adapted to system identification purposes and a model reduction approach. We shall see that using this original procedure yields to ready-to-use models with a small number of parameters in a black box approach.

Artificial neural networks consist of a large number of interconnected processing elements known as neurons that operate as microprocessors [18]. They are characterized by the following features [19], [20], [21], [22], [23], [24], [25]: approximation capability of nonlinear functions via activation functions, ability to process many inputs and outputs and the synaptic weights are automatically adjusted by the learning algorithm during the training. Recent research results show that neural networks are very effective for modeling the complex nonlinear systems when we consider the plant as a black box [15], [5], especially those which are hard to describe mathematically [18], [26], [27]. However, the main problem of neural networks is that they require a large number of neurons to deal with complex systems [25]. More neurons presume a better approximation but lead to a more complex model too [18], [28].

Even if, nowadays we have at our disposal modern CPU, FPGA or graphics processors architectures, in applications like adaptive control, in particular inverse control, the adaptive controllers have the same complexity as the reference models [6], [7], [8]. If low order reference models are introduced, the number of parameters to be computed will be reduced [29], therefore, models with small number of parameters are preferred. By reducing the number of neurons we reduce directly the number of parameters (synaptic weights) of the neural network model.

Different works have been achieved in order to solve the problem of large number of neurons and approximation accuracy. Some techniques consist in determining the “optimal” number of neurons. Trial and error method is one of them [30], but this approach is laborious and may not arrive to an optimum structure [31]. Network pruning techniques have also been successfully used for structural optimization [32], [6], [33]. These techniques consist in selecting a fairly good number of neurons, to be gradually reduced in the course of a series of trainings in order to find the “optimal” number of neurons. However, these methods still suffer from slow convergence [31]. More recently, evolutionary techniques as genetic algorithms (GAs) and particle swarm optimization (PSO) have been employed to optimize weights, layers, number of input–output nodes, neurons and derive optimal neural network structures [31], [34]. Ge et al. [35] proposed a learning algorithm centered on a dissimilation particle swarm optimization, serving to compute the optimal synaptic weights, the learning rate, and the architecture evolution (optimal number of neurons in the hidden layer). Xie et al. [27] developed an algorithm to optimize the neural network by means of a connection matrix and genetic algorithms. The matrix represents the connection between neurons in adjacent layers (direct matrix mapping encoding, DMME) and the genetic algorithms helps to find the optimal number of neurons in the hidden layer. Coelho and Wicthoff [4] use genetic programming (GP) to estimate the evolution of its architecture in order to generate more appropriate or accurate models. In [36] an hybrid multiobjective evolutionary artificial neural network, based on a combination of GAs and singular value architectural recombination (SVAR), is proposed to prune the neural network. A disadvantage of the previously outlined techniques is the excessive time requirements to find the most convenient number of neurons, since the neural network must be trained every time the model is modified or restructured [26]. Moreover, to solve the problem of finding the best trade-off between model complexity and model accuracy, rather subjective criterion is always used to decide whether the evolution of the neural network is appropriate and sufficient, thus, we can consider that this problem still remains unsolved.

Other techniques trying to solve the same problem are based on the design of the neural network. In [24] a novel time-delay recurrent neural network (TDRNN) is proposed to generate a simple structure. In [28] a neural network using a competitive scheme is proposed in order to provide an effective method with less network complexity. In [37] the selection of an appropriate FLANN (functional link artificial neural network) structure as the backbone of the model offers low complexity by means of a single layer ANN structure.

In the same vein, to overcome the disadvantages of the “architecture evolution techniques” and with the conviction that the improvement of the model quality is linked to a suitable neural network design, we decided to tackle the problem by proposing a dedicated neural network design and a model reduction approach. In this sense, the main contribution of the paper is to develop a particular architecture depending on two factors: (1) the activation functions in each layer and (2) the initial conditions of the synaptic weights. The reader shall notice that the proposed architecture nevertheless remains sufficiently general to provide a wide range of useful model types. The model reduction approach is developed in two steps: the first step consists in training a recurrent three-layers neural network chosen to tolerate an initial large number of neurons. In a second step, the three-layers architecture is transformed into a two-layers representation with a significant reduced number of weights retaining the approximation accuracy of the previous three-layers model. The preceding neural network provide models, with reasonable reduced number of parameters and a convenient estimation accuracy. These model types are currently used for model-based control techniques [5], [6], [7], [8]. The learning algorithm used to optimize the synaptic weights is the classical steepest descent algorithm with a back propagation configuration, since in this paper the main purpose is not to optimize the learning algorithm.

In the sequel, the paper is organized as follows: First we shall introduce the new neural network structure which allows us the investigation of the balance complexity/accuracy (see Section 2). In Section 3 the theorem on which our model reduction approach is based is given, as well as the related description of the method. In Section 4, the paper discusses the results of the identification of a benchmark system. Subsequently conclusions and perspectives are given in Section 5. Finally, the neural network training procedure is presented in Appendix A.