Learning Informative Features from Restricted Boltzmann Machines

Learning successful classifier or other predictor greatly depends on appropriately prepared data representation. The representation captures latent factors (features) explaining variations in data. Obviously, formulating the data representation enforces application of feature engineering, i.e., formulating a set of features using human knowledge and prior information about the considered problem. However, it would be highly desirable to learn features automatically from low-level sensory data, and with possibly no contribution of human hand-tuning [4]. Among many manners of learning representations, deep learning becomes its leading paradigm [3] with applications to various domains, e.g., object recognition [31], speech recognition [16], natural language processing [9], neuroimaging [19].

Explanation of the success of deep learning is still unclear, nonetheless, it is hypothesized that deep architectures help in disentangling factors in data [3]. However, there is no common perspective on how the learning of deep models should be performed. Some insight into the matter has been outlined in [5] and further elaborated in [4] where several general guidelines have been formulated. These guidelines, which were named general (broad) priors, point out possible directions in developing new models and learning algorithms. One of such priors is sparsity of features [7, 12], i.e., only a small fraction of hidden factors are relevant. Sparsity is motivated by biological brain where only 1–4 % of neurons are active at any time [3]. Possible strength of the sparse representation is easier propagation of gradient and better coding of information. However, it is not obvious whether sparsity in shallow and deep models gives the same results or how it influences information coding. Nonetheless, it is believed that sparsity is one the main causes of successful data representations.

Other prior aims at formulating learning objective which enforces robustness of the representation [28]. Recently, an empirical evidence was presented that indeed the deep hierarchy can be helpful in better disentangling the hidden factors [6]. A different question is what type of learning objective results in successful representation. This issue is especially problematic since the representation learning can be performed in the unsupervised manner. For example, in the case of classification the learning objective is rather straightforward, i.e., we want to minimize the number of misclassifications. In the context of unsupervised learning the objective is less obvious, because it is unclear how to formulate the learning objective for providing a representation that contains possibly all information about data and allows simple disentangling underlying factor of variation. In [4] the authors hypothesized that the typical learning objective, i.e., the log-likelihood function, may be expanded by an additional data-dependent regularization term that helps to learn the set of features capable of disentangling factors. Such regularization techniques were proposed and empirically proved to increase quality of the hidden representation, e.g., Dropout [35] which is an adaptive (data-dependent) regularizer [36], or its extension Dropconnect [37]. In this paper, we will follow this line of thinking and propose new data-dependent regularizers.

The class of deep architectures is wide, however, one of the most popular models that is used as a building block for unsupervised deep representation learning is the Restricted Boltzmann Machine (RBM). The RBM is a Markov random field with a bipartite structure consisting of one layer of observable variables and one layer of hidden units (features). Typically, the RBM is stacked in a hierarchy to build a deep network or it is used as a feature extractor [3]. However, in order to obtain better generalization ability of the RBM, and thus better data representation, learning could be improved by incorporating some prior about the world or the representation itself, as mentioned earlier. The prior can be introduced in form of constraints or as a penalty (regularization) term in the learning objective. In the context of deep learning, the widely-used regularization is weight decay, i.e., the \(\ell _2\) norm on parameters [18] that prevents the model from overfitting and helps to stabilize learning process. Other kind of regularization aims at keeping hidden units activation at a constant but small level, which eventually results in sparse representation [24].

Realizing the challenging issue of learning data representation, in this paper we propose a new approach for learning a hidden representation on the example of the RBM. The learning objective for the RBM consists of two main components: the likelihood function and the regularization term that enforces features to maximize entropy and to be pairwise uncorrelated. Particularly, we formulate unconstrained learning problem and further provide information-theoretic regularization, i.e., the sum of entropy of each hidden unit for given observables, and soft orthonormality constraint to obtain uncorrelated features. We call the features trained using these two regularizers informative features to highlight the connection of our approach to information theory and binary codes learning [38]. Basing on this formulation of learning informative features, we calculate gradients with respect to weights that can be later used in the stochastic gradient descent learning algorithm. Additionally, we give a close relation of the proposed entropy-based regularization to the variance-based regularization used in learning binary hashing codes [38]. Eventually, we empirically show that the proposed framework for learning informative features results in more discriminative data representation that maintains its generative capabilities measured by the test log-likelihood function.

The remainder of the paper is organized as follows. We first outline the RBM and formulate unconstrained learning problem in Sect. 2. We further present the idea of learning informative features in Sect. 3. In the following subsections we propose the core of our approach, i.e., the information-theoretic regularization (in Sect. 3.1) and the orthonormality regularization and the reconstruction cost (in Sects. 3.2.1 and 3.2.2). In Sect. 4 we relate our proposition to existing approaches. Section 5 gives the experimental results for three datasets, namely, MNIST, CalTech 101 Silhouettes, and 20-newsgroups. At the end, conclusions are drawn in Sect. 7.

The Restricted Boltzmann Machine (RBM) is a Markov random field that defines the joint distribution over binary visible and hidden units [34]. A distinctive trait of the RBM is its bipartite structure in which visible units (visibles or observables, for short) and hidden units form two layers and connections within the same layer are prohibited. The relationships among units are specified through the energy function:where \(\mathbf {x} \in \{0,1\}^{D}\) are the visible units, \(\mathbf {h} \in \{0,1\}^{M}\) are the hidden units, and \(\varvec{\theta } = \{\mathbf {W}, \mathbf {b}, \mathbf {c}\}\) is a set of parameters, \(\mathbf {W}\in \mathbb {R}^{D\times M}\), \(\mathbf {b}\in \mathbb {R}^{D}\), and \(\mathbf {c}\in \mathbb {R}^{M}\) are, respectively, weights, visible biases, and hidden biases. For the energy function (1) the RBM is defined by the Gibbs distribution:where \(Z(\varvec{\theta })\) is the partition function, \(Z(\varvec{\theta }) = \sum _{\mathbf {x}} \sum _{\mathbf {h}} \mathrm {exp} \big ( - E(\mathbf {x},\mathbf {h}|\varvec{\theta }) \big )\).

Since there are no connections within the same layer, the RBM possesses very useful property that the conditional distribution over the hidden units factorizes given the visible units and the probability of a hidden unit is as follows:¹ \(^{,}\) ² Likewise, the conditional distribution over the visible units factorizes given the hidden units and the probability of a visible unit is the following:

Learning data representation is conceptually closely related to learning binary codes where a single bit can be associated with a feature. In the learning binary codes, ideally one should propose a code-book that encodes the reality such that the information provided by each bit is maximized. In other words, the entropy of each bit for given data is maximized. Additionally, one would like to avoid redundancy in bits as much as possible that is equivalent to bits that are pairwise uncorrelated. We will use these two principles for learning data representation of the RBM by introducing constraints enforcing entropy of hidden units to be maximized and probabilities of hidden units to be pairwise uncorrelated. We call features trained in this manner informative features to highlight that our motivation is anchored in the theory of learning binary codes [14, 33, 38, 39]. The learning binary codes problem is formulated as a constrained optimization problem with two hard constraints, namely, balancedness of hidden units (i.e., maximization of entropy of hidden representation) and orthonormality of columns in the matrix of weights (a relaxed version of the pairwise decorrelation of hidden units).

In the context of the RBM we believe that the entropy constraint could improve mixing of Markov chain during training because the Markov chain started from a training example can easier change the corresponding hidden states. Additionally, we hope that the balancedness of hidden units can make it harder for the classifier using the features from the RBM to overfit and thus can improve the discriminative capabilities of the representation. Furthermore, we expect that application of the constraint enforcing decorrelation of the probabilities of hidden units results in features for which better separability of classes can be achieved.

The idea of exploiting entropy as a regularization term is well known in machine learning. However, in the considered case we introduce entropy-based regularization in order to obtain specific property of the hidden representation while typically the entropy-based regularization is used in other context, e.g.., in the semi-supervised learning [10, 15, 27] or supervised regularization of deep models [13].

The idea of our approach is closely related to the one utilized in learning binary hashing codes [14, 38]. However, there are two major differences. First, sometimes the entropy of the binary code serves as the objective [14] while in our case it is a constraint (similarly as it is done in [38, 39]). Second, in learning binary hashing codes each bit is described by a signum function and thus direct calculation of derivatives w.r.t. to weights is prohibited and some kind of relaxation of the regularization term is needed. The authors of [38] proposed a lower bound of the sum of variances (since maximum of bit entropy and variance coincide and both quantities are convex). However, we deal with sigmoids and thus the derivative of the regularizer can be calculated exactly (see Eq. 11). Nonetheless, it could be tempting to follow the same line of thinking as in [38] to re-express the balancedness constraint in terms of variance instead of entropy that leads to the following regularizer: Further, calculating the derivative w.r.t. to \(W_{dm}\) gives:Taking a closer look at (11) and (21) one immediately notices that the difference in the derivatives is how the variances are modified. Applying the entropy-based regularization results in non-linear modification while the variance-based regularization outcomes linear modification.

The hard orthonormality constraint is usually used in the context of learning binary codes [14, 38, 39]. However, as pointed out by the authors of [22], this constraint is also utilized in the problem statement of the Independent Component Analysis (ICA). Moreover, in [22] the reconstruction cost is proposed as a soft orthonormality constraint that results in a new formulation of the ICA. However, in our work we apply the reconstruction cost to obtain specific property of hidden representation of a deep model (RBM) instead of learning data representation directly as in the ICA.

The results of the experiments are presented in Tables 1, 2, and 3 for the test classification error, test log-likelihood, and average number of active hidden units, respectively. In Fig. 1 the exemplary weights of the RBM trained without any regularization term and with the entropy-based regularizer and the reconstruction cost are presented. In Fig. 2 the number of epochs during learning process for different learning schema is depicted. For clarity of visual comparison, we have omitted results of the orthonormality regularization (RBM+ortho), the reconstruction cost (RBM+rc), the weight decay with the orthonormality regularization (RBM+wd+ortho), and the weight decay with the reconstruction cost (RBM+wd+rc), the sparsity and the max-norm regularization, Dropout and Dropconnect from Fig. 2. It is worth noting here that until now Dropout and Dropconnect were mainly analyzed in the context of the feedforward neural nets and the convolutional nets. For the RBM only some qualitative conclusions were given for the Dropout that it leads to sparse representation [35], however, due to our knowledge no quantitative results for the RBM are reported.

We notice that application of the information theoretic regularization alone increases the discriminative performance of the data representation comparing to the RBM trained without any regularization. However, better improvement can be seen in the case of the orthonormality regularization and the reconstruction cost. Though, the biggest decrease in classification error was achieved by the combination of both the information theoretic regularization and the reconstruction cost. This result reveals that indeed both constraints on the data representation are significant in learning informative features. Slightly worst but still very good performance was obtained by the weight decay with the reconstruction cost. This result is not surprising because the weight decay maximizes entropy implicitly, as pointed out in Sect. 3.1. Additionally, we observe that application of the reconstruction cost may be superior to the orthonormality regularization (see eight and ninth rows of Table 1). Quite good results were obtained by the sparse regularization (second best on MNIST and slightly better than the RBM on CalTech), however, still worst than the RBM+itr+rc. The Dropout worked similarly to the ordinary RBM with almost no positive effect (the only except was CalTech dataset). The max-norm regularization and the Dropconnect gave better results than the RBM only on 20Newsgroups but they completely failed on MNIST and CalTech. Surprisingly, ClassRBM performed very badly in comparison to the logistic regression trained with the features extracted from the RBM. However, this effect has been already noticed, e.g., ClassRBM requires 6000 of hidden units to reach the level of error equal 3.39 on MNIST [21].

In the context of generative evaluation, application of any of the considered regularizers alone did not result in any improvement. However, the combination of the information theoretic regularization and the reconstruction cost, similarly to the discriminative performance, led to the best outcome measured in terms of the test log-probabilities (on MNIST and CalTech datasets, see Table 2). This effect was not encountered on the 20Newsgroups dataset where the RBM without any regularization performed the best but it can be explained in the following manner. It may happen that the test log-likelihoods become smaller because smoothing a contribution from each hidden unit could potentially increase the peaks around training samples which further results in lower probability being assigned to nearby test examples. However, the difference between the RBM and the RBM+itr+rc on the 20Newsgroups is slight and hence it is difficult to draw any decisive conclusion. Interestingly, the sparse regularization helps to obtain better generative performance than the ordinary RBM but is is still worst than the RBM+itr+rc. The max-norm regularization fails completely, similarly to the Dropout and the Dropconnect. This may be a reason why these regularizers are useful in the feedforward neural networks but are rarely used in the context of the RBM.

In order to get a better understanding of how the considered regularizers affect the data representation, we also examined the average number of active hidden units. As stated earlier, we expected that the application of any of the regularizers increases the activity of hidden units and indeed the experiments confirmed our presumptions partially (see Table 3). It seems that application of the information theoretic regularization does not influence the activation of the hidden units. In the case of the orthonormality regularization it is hard to conclude (slight increase on MNIST only, see Table 3). However, for RBM+rc, RBM+itr+ortho, RBM+itr+rc, RBM+wd+ortho and RBM+wd+rc the effect of increase of hidden units activation is apparent. The reason why the value of active hidden units is increased follows directly from the applied regularizers. The proposed regularization terms imitate how the binary codes are trained, i.e., each bit (a feature) tries to maximize information in data while a code (the whole representation) avoids redundancy in bits. As a consequence, the more two observables differ (in the sense of Hamming distance), the more distinct features (hidden units) switch on. Therefore, we notice increased value of active hidden units for the proposed regularizers because possibly the trained representation activates a number of common features but also a number of specific hidden units for given data. We leave inspecting this issue for future research.

We also analyzed a potential influence of the considered regularizers on the number of epochs in the learning process. It turned out that the application of the soft orthonormality constraint reduced the number of learning iterations (the only exception was RBM+itr+rc on CalTech, see Fig. 2b). This effect is especially evident on 20Newsgroups dataset where the number of epochs was dramatically smaller for RBM+itr+rc and RBM+itr+ortho in comparison to other learning schema (see Fig. 2c). The possible explanation of this result may arise from the fact that the informative features allow better mixing of Gibbs sampler in the contrastive divergence procedure. Since entropy and orthonormality constraints enforce the probabilities of hidden units to be balanced and uncorrelated, different set of features should be activated for given different visibles. Therefore, the blocked Gibbs sampling procedure can easier mix between modes and thus the learning process needs less iterations. However, this claim needs further investigation.

Eventually, as a qualitative assessment of the trained RBM models, we show trained weights and visible biases for the ordinary RBM and the one trained with the information-theoretic regularization and the reconstruction cost on the MNIST benchmark dataset. The weights with highest value of \(\ell _2\) norm are shown in Fig. 1 where the top left cell in each figure is the visible bias vector. We can easily verify that the application of the proposed regularization leads to different filters, i.e., spot filters, and strokes filters.

For future research, we would like to investigate the use of the proposed regularization term in pre-training of deep networks. The results for single RBM are very promising, hence we believe that the effect of learning informative features can be more evident in deeper architectures. Especially, since we aim at learning uncorrelated hidden units, a deep model can easier disentangle factors at consecutive levels. Moreover, the application of orthonormality regularization or the reconstruction cost increases hidden units activation. This results is in contrary to the common opinion that deep models should be sparse [12]. We find that combining sparsity with learning informative features is potentially an interesting research direction. Moreover, in the experiment we found that application of the soft entropy and orthonormality constraints lead to reduction of the number of learning iterations. Our hypothesis is that the informative features allow better mixing of the Gibbs sampler in the contrastive divergence procedure. This statement requires more thorough investigation. In the experiments we assumed a fixed number of hidden units. However, there is plenty of works on how to choose proper structure of a model using the model selection with different complexity penalty terms [2], naming only a few, Akaike’s information theoretic criterion [1, 32] or Rissanen’s minimum description length [29], Last but not least, in this work we considered a probabilistic deep model. However, there is a vast of deterministic deep models based on auto-encoders. Since exact application of the proposed approach is impossible in the context of auto-encoders, it is challenging to suggest corresponding fashion of learning informative features.