Restricted Boltzmann machine and softmax regression for fault detection and classification

Restricted Boltzmann machine is a stochastic neural network with a visible and hidden layer. Each unit of the visible layer is having a undirected connection with each unit of the hidden layer, with weights associated with them. Each unit of the visible and hidden layer is also connected with their respective bias units. The structure of RBM is shown in Fig. 3. The RBMs do not have connections among the visible units and similarly in hidden units also. This restriction on connection makes it restricted Boltzmann machine. The state of a neuron unit in a hidden layer is stochastically updated based on the state of the visible layer and vice versa for the visible unit.

The energy function of the RBM model for visible and hidden units can be computed aswhere a and b are bias of the visible units and hidden units, respectively. The parameter W is weights of the connection between visible and hidden layer units. The joint probability distribution of visible units v and hidden units h of the RBM is defined bywhere Z is partition function and defined as sum of energy functions of over all possible configurations:The conditional probability of activation of hidden layer given the visible state v is computed aswhere function \(\sigma \) is logistic function. Similarly, the conditional probability of activation of visible layer given the hidden state h is computed asOn training, the RBM updates its weights W and bias a and b, to maximize the probabilities assigned to training set \(x_{l}^{(i)}\). For training of the RBM, the contrastive divergence (CD) training method [17, 18] is used. The contrastive divergence training is performed with the stochastic steepest ascent. The change of the parameter W by the CD training is given bywhere parameter \(\epsilon \) is learning rate and \(v_i\) is state of visible layer unit given by Eq. (5) and \(h_j\) is state of hidden layer unit given by Eq. (4). The weight matrix W is initialized by some random values and then updated by the value \(\Delta W\) for each training data set. Similarly, the increments in bias are computed and the bias vectors a and b are updated. The term \(\langle v_ih_j \rangle \) represents the average of the state values products. The subscript “data” is for the value of hidden state computed by Eq. (4), and subscript “recon” is for the value of visible state computed by Eq. (5). The more detail of RBM implementation and training with CD is given in the guide by Hilton [19].

The number of neurons in the visible layers is always equal to input training vector of size m, but the number of neurons in hidden layer n is selected based on the factor by which dimension of training data needs to be reduced. The training data matrix of size \(m\times v\) is reduced to feature matrix W of size \(m\times n\), where \(n \ll v\). The weight matrix W has n linearly independent basis vectors and each represents a unique feature learned from the data.

In a typical case of RBM2 with 50 hidden neurons, there are 50 feature vectors in the features matrix W. A typical pattern of fault data is shown in Fig. 4 and plots of some of the typical learned feature patterns from the features matrix W are shown in Fig. 5.

In the proposed technique, the structure of both RBM1 and RBM2 is exactly same. Initially, the RBM1 is trained with the unlabeled data set \(x_{\mathrm{ul}}^{(i)}\) and the learned weight matrix \(W_1\) and bias vectors \(a_1,b_1\) are used as initial values of the \(W_2\), \(a_2\), and \(b_2\) of the RBM2. RBM2 is then trained with the training data set \(x_{l}^{(i)}\) and the learned weight matrix \(W_2\) is further used for transformation. The learning rate, \(\epsilon = 0.01\), was used in training of both RBMs.

The feature matrix \({W_2}\) is used to linearly transform the input training and testing data vectors into lower dimensional feature vectors. The training data vector \(x_l^{(i)}\in {R^m}\) and testing data vector \(x_t^{(i)}\in {R^m}\) are transformed into \(\hat{x_l}^{(i)}\in {R^n}\) and \(\hat{x_t}^{(i)}\in {R^n}\), respectively, as follows: These transformed vectors \(\hat{x_l}^{(i)}\) and \(\hat{x_t}^{(i)} \) are now represented as the weighted linear combination of the feature vectors from \({W_2}\). In other words, the features of \(x_l^{(i)}\) and \(x_t^{(i)}\) are compressed and represented in terms of these learned features. The new training data set \(\hat{x_l}^{(i)}\) with v number of labeled training data vectors is used to train the softmax regression classifier.

The size of the transformed training and testing data vectors is n, which is very less than original size m. This size reduction is due to the number of hidden layer neurons that are less than the number of input layer neurons or \(n \ll m\). This way the proposed technique improves the classification performance by enhancing the feature representation and reducing the size of the training and testing data vectors. In typical case, an input training and testing spectrum vector of size 6000 is reduced to hidden layer size of 50 after transformation. The small size of training vector requires small set of weight in a classifier and the cost function is easy to optimize for these small set of weights. This improves the classification performance with reducing training time.

The softmax regression is a generalization of the logistic regression [9, 11], where the output class labels are multi-class \(y_i\in (1,2,\ldots k)\), instead of binary output classes. The input training set for softmax regression with v numbers of data vectors \(\{(x_1,y_1),(x_2,y_2),\ldots (x_v,y_v)\}\), where \(x_i\in R^n\). In the softmax regression-based classifier, the probability \(P(Y=j/X)\) of X belonging to each class from set of k classes is given aswhere the parameters \(j=1,\ldots ,k\) and \(Y=[y_1,y_2,\ldots ,y_k]\) are output class. The input variables to this probability function are feature vector \(X=[x_1,x_2,\ldots ,x_v]\), and the weight or model parameter \(\theta =[\theta _0,\theta _1,\ldots ,\theta _k \in R^n]\) of softmax regression model. The generalized softmax regression cost function is defined asThis softmax regression cost function has no closed form way to minimize the cost value, so the iterative algorithm, gradient descent is used. To make the softmax Regression cost function strictly convex, so that it can converge to a global minimum, a weight decay term is added. The modified cost function with its gradient is given as follows: where the weight decay parameter \(\lambda \) shall always be positive. The weight parameters are updated by \({\theta {}}_j={\theta {}}_j-\alpha {}{\nabla {}}_{{\theta {}}_j}J\left( \theta {}\right) \) for \(j=1,\ldots ,k\). The weights \(\theta \) of softmax regression are initialized with random values, and these weights are updated with each training vector \(\hat{x_l}^{(i)}\), to minimize the value of the cost function. The number of weight vectors \([\theta _0,\theta _1,\ldots ,\theta _k]\) in the softmax regression is equal to the number of output classes and size of each weight vector \(\theta _k \in R^n\) is equal to the size of input data vector. In this case, the size of input data vector is equal to the number of hidden layer neurons in RBM. In the implementation of softmax regression, \(\lambda \) = 0.001 was used.

For both the labeled and unlabeled data sets, fault signal was recorded from all four positions of the sensor. For the labeled data set, the acoustic fault data are recorded for three different types of seeded faults and one normal operation, as shown in Table 1. For the unlabeled data set, three different types of faults are seeded randomly. These seeded faults for the unlabeled data set are different from the faults seeded for the labeled data set. Each position of the sensor on the setup represents a data distribution \(D_i\). The unlabeled data from distribution \(D_1\) to \(D_4\) are merged to form the unlabeled data set \(x_{\mathrm{ul}}\). This common unlabeled data set \(x_{\mathrm{ul}}\) is used by RBM1 to compute unlabeled weights \(W_1\) and bias \(a_1\) and \(b_1\). These unlabeled weights and bias are then used as the initial weight and bias of the RBM2. This technique is tested separately for each sensor position, with their respective labeled data sets and with this common unlabeled data set.

Table 1 shows the seeded faults and number of data sets recorded for each fault type. The details of each fault are described in [4, 20, 21]. There are total 248 labeled data sets that are recorded for each sensor position. In testing of this technique, the last three faults from Table 1 were used as part of the unlabeled data set and the rest were part of labeled data set. For testing, the labeled data set for each position of the sensor is divided into different ratios of training and testing data set, as shown in Table 2.

In this work, the majority voting (MV) is the majority of classification type among all four sensor positions. If the classification type has more than two votes for a class, then the classification belongs to that particular class. In addition, if there is a tie between votes, then the classification is assumed from incorrect class only. The classification performance is depicted in %, total correct classification * 100/total test cases, in all the tables.

To test this technique, the labeled data set is divided into different ratios of training and testing data sets, as shown in Table 2. In each division ratio, the training and testing data are randomly selected and classification performance is computed for 100 iterations. Table 2 shows the average classification performance of these 100 iterations. The classification performance of each position with different division ratios of the labeled data set in training and testing data is shown in Table 2, along with majority voting (MV) among all four positions with its standard deviation (SD). The results in Table 2 are with hidden layer size of 50 neurons in both RBM1 and RBM2.

In typical ratio of (15–85%), where only 38 training data sets for all four faults types were used for training of the RBM2. The classification performance is more than 90% for each position, as shown in Table 2. The performance after majority voting, among all positions, is 99.35%. With the increase in the size of the training data, there is a small improvement in the MV classification performance and its SD. The small value of SD shows the consistency in MV classification accuracy or small variations in the MV classification accuracy.

Table 3 shows a typical case of 15–85% division ratio (38 training sets and 210 testing sets). The individual classification performance for each fault type is more than 90%, except one case TAPPET fault in position 3. In this case, the overall MV classification performance is 209 correct classifications out of 210 test cases. In all 210 test cases, only one case was wrongly classified by two classifiers on majority voting.

From the above analysis, it can be concluded that the proposed technique works very well in the industrial environment, with performance more than 99%. This performance was achieved with the 40 Hz rotation speed of the engine with variations in the range of \({\pm }2\%\). The performance in case of 5–95% is only 81.50%; this is due to insufficient training data sets for the training of the RBM2 and softmax regression classifier. In this case, total numbers of training data sets are only 12 for all four fault classes. This amount is too small to train any machine learning algorithm. As compared to unsupervised feature extraction based on the sparse autoencoder proposed in [10], the proposed technique reduces the requirement of training data significantly with large improvement in the performance. Due to use of the unlabeled features for as initial weights of RBM2, the cost function of the RBM2 does not get trapped in local minima and reaches global minima very fast with small number of training examples.

For comparison of the proposed RBM-based feature extraction with PCA, the 50 most significant eigen vectors of the training data set after PCA were used as the features training and testing data. From Table 6, it can be seen that the proposed RBM-based feature extraction has outperformed the PCA-based feature extraction with consistency in results for the same size of the training data set size. At the small size of training data set (15–85%), the performance of the proposed technique is far better than PCA. PCA is a linear operation and does not extracts the complex features in the data, but the RBM has a nonlinear sigmoid function in its basic unit neuron. This helps the RBM to learn complex nonlinear relations in the data more efficiently with small size of the training data. This makes the classification more consistent with small size of training data. As the number of training data increases, the consistency in the results by PCA improves but still not at the level of RBM (Table 6).

Yadav et al. [20] have proposed an FFT and correlation-based technique using acoustic data from the same type of IC Engine Test Rig. In this work, the final classification accuracy for four different types of fault classes was less than 93%. The classification accuracy for CHN fault was 80%, and for MRN fault, it was 93%. In this technique, the faulty engine was compared with a prototype engine, so no classifier was used. Nidadavolu et al. [21] also proposed a fault detection technique based on empirical mode decomposition (EMD) and Morlet wavelet for the same type of IC engine. The overall classification accuracy for the proposed technique was less than 90% for each fault type and each position. In this work, they have used five different types of fault classes with total 540 data sets. Out of these 540 data sets, 70% were used for training of an artificial neural network (ANN) classifier and rest 30% were used for testing. In a similar type of fault detection by Wu et al. [3], the feature extraction was done using WPT and ’Shannon entropy’ from acoustic data of the GDI (gasoline direct-injection) engine. They have recorded 150 experimental data for each operating condition of the engine for six different types of faults. Out of these 150 data sets, 30 were used for training and the remaining were used for testing. The average classification accuracy for an ANN classifier for the different operating conditions of the engine was around 95%. Yadav et al. [4] has proposed a spectrogram based statical feature extraction technique for the same type of test rig. The majority voting accuracy of their technique was less than 93% for all fault classes. In this work, they have used 400 training data sets to train ANN classifier with seven different types of fault classes and 200 data sets for testing. For the same data set, Chopra et al. [10] have proposed a unsupervised feature extraction technique by a sparse autoencoder. This technique has 98% classification accuracy for four different types of fault classes without any unlabeled data for 62 training data sets. The proposed techniques overcome the problem of random initialization [10] of the sparse-autoencoder weights with more accuracy.

As compared to the above-discussed techniques, the proposed technique is at par with the other techniques in terms of classification performance without any hand-engineered feature extraction. This technique requires less training data as compared to other techniques available in the literature.

In the industrial environment, where a lot of noise is there in recordings of sensor data, the RBM-based feature extraction is very much successful. This way the proposed technique proves its robustness for the industrial environment. The use of the unlabeled data reduces the requirement of labeled training data significantly along with significant enhancement in the performance and consistency in the results.

The implementation and analysis of this technique were done on Matlab-2013, on an Intel i5 CPU with 8GB RAM.