A Hierarchical Predictive Coding Model of Object Recognition in Natural Images

The training procedure for the first processing-stage was as follows.

Following the training of both stages, described above, the hierarchical PC/BC-DIM model can be used to recognise objects in novel, test, images. The test image is pre-processed into ON and OFF channels as described in the “Training” section. These are input to the first processing stage, and the PC/BC-DIM algorithm (as described in the “The PC/BC-DIM Algorithm” section) is executed. The first-stage prediction neuron responses are then provided as inputs to the second processing stage and the PC/BC-DIM algorithm (as described in the “The PC/BC-DIM Algorithm” section) is executed for the second stage. The second-stage prediction neuron responses are then used to identify the to-be-recognised objects. For those tasks in which the location and scale of the object was fixed and for which each image contained exactly one object (digit and face recognition), the maximum response was taken to indicate the class of the image. For those tasks in which the location of the object could vary and in which the number of objects in each image could vary (car recognition), the presence of an object was indicated by prediction neurons responses that were peaks in the spatial neighbourhood and which exceeded a global threshold.

The main mathematical operation required to implement the PC/BC-DIM algorithm is the calculation of sums of products. The algorithm can therefore be equally simply implemented using matrix multiplication or convolution.

The matrix-multiplication version of PC/BC-DIM is illustrated in Fig. 1b and was implemented using the following equations: Where x is a (m by 1) vector of input activations; e is a (m by 1) vector of error neuron activations; r is a (m by 1) vector of reconstruction neuron activations; y is a (n by 1) vector of prediction neuron activations; W is a (n by m) matrix of feedforward synaptic weight values, defined by the training process described in the “Training” section; V is a (m by n) matrix of feedback synaptic weight values; [v] _𝜖 = max(𝜖, v); 𝜖 ₁ and 𝜖 ₂ are parameters; ⊘ and ⊙ indicate element-wise division and multiplication, respectively; and ← means that the left-hand side of the equation is assigned the value of the right-hand side. The matrix V is equal to the transpose of the W but each column of V is normalized to have a maximum value of one. Hence, the feedforward and feedback weights are simply rescaled versions of each other.

The convolutional version of PC/BC-DIM was implemented using the following equations: Where X _i is a two-dimensional array representing channel i of the input; R _i is a two-dimensional array representing the network’s reconstruction of X _i ; E _i is a two-dimensional array representing the error between X _i and R _i ; Y _j is a two-dimensional array that represent the prediction neuron responses for a particular class, j, of prediction neuron; w _ji is a two-dimensional kernel representing the feedforward synaptic weights from a particular channel, i, of the input to a particular class, j, of prediction neuron, defined by the training process described in the “Training” section; v _ji is a two-dimensional kernel representing the feedback synaptic weights from a particular class, j, of prediction neuron to a particular channel, i of the input; and ⋆ represents cross-correlation. The weights v _ij are equal to the weights w _ij but are rotated by 180 ^∘ and are normalised so that for each j the maximum weight value, across all i, is equal to one. Hence, the feedforward weights, between a pair of error-detecting and prediction neurons, and the feedback weights, between the corresponding pair of reconstruction and prediction neurons, are simply re-scaled versions of each other.

The matrix-multiplication and convolutional version of PC/BC-DIM are interchangeable, and which particular method was used depended on which was most convenient for the particular task. For example, the convolutional version was used when prediction neurons with identical RFs were required to be replicated at every pixel location in an image. To simplify the description of the proposed method, the rest of the text will refer only to the matrix-multiplication version of PC/BC-DIM.

For all the experiments described in this paper, 𝜖 ₁ and 𝜖 ₂ were given the values \(\epsilon _{1}=\frac {\epsilon _{2}}{max\left (\tilde {V}\right )}\) (where \(\tilde {V}\) is a vector containing the sum of each row of V, i.e., the sums of feedback weights targeting each reconstruction neuron) and 𝜖 ₂ = 1×10⁻². Parameter 𝜖 ₁ prevents prediction neurons becoming permanently non-responsive. It also sets each prediction neuron’s baseline activity rate and controls the rate at which its activity increases when a new stimulus appears at the input to the network. Parameter 𝜖 ₂ prevents division-by zero errors and determines the minimum strength that an input is required to have in order to effect prediction neuron response. As in all previous work with PC/BC-DIM, these parameters have been given small values compared to typical values of y and x, and hence, have negligible effects on the steady-state activity of the network. To determine this steady-state activity, the values of y were all set to zero, and Eqs. 1 to 3 were then iteratively updated with the new values of y calculated by Eq. 3 substituted into Eqs. 1 and 3 to recursively calculate the neural activations. This process was terminated after 50 iterations. After 50 iterations, values of y less than 0.001 were set to zero. To perform simulations with a hierarchical model, the steady-state responses for the first processing-stage were determined. The first-stage prediction neuron responses were then provided as input to the second processing-stage, and Eqs. 1 to 3 applied to the second processing-stage to determine its response.³

The values of y represent predictions of the causes underlying the inputs to the network. The values of r represent the expected inputs given the predicted causes. The values of e represent the discrepancy (or residual error) between the reconstruction, r, and the actual input, x. The full range of possible causes that the network can represent are defined by the weights, W (and V). Each row of W (which correspond to the weights targeting an individual prediction neuron, i.e., its RF) can be thought of as a “dictionary element,” or “basis vector” or “elementary component” or “preferred stimulus,” and W as a whole can be thought of as a “dictionary” or “codebook” of possible representations, or a model of the external environment. The activation dynamics, described by Eqs. 1, 2, and 3, perform gradient descent on the reconstruction error in order to find prediction neuron activations that accurately reconstruct the input [14, 18, 62]. Specifically, the equations operate to find values for y that minimise the Kullback-Leibler (KL) divergence between the input (x) and the reconstruction of the input (r) [14, 63]. The activation dynamics thus result in the PC/BC-DIM algorithm selecting a subset of active prediction neurons whose RFs (which correspond to dictionary elements) best explain the underlying causes of the sensory input. The strength of activation reflects the strength with which each dictionary element is required to be present in order to accurately reconstruct the input. This strength of response also reflects the probability with which that dictionary element (the preferred stimulus of the active prediction neuron) is believed to be present, taking into account the evidence provided by the input signal and the full range of alternative explanations encoded in the RFs of the whole population of prediction neurons.

Compared to some earlier implementations of the PC/BC-DIM model, the algorithm described here differs in the following respects: These changes help PC/BC-DIM to scale-up to very large networks of neurons. Specifically, for a very large population of prediction neurons, adding 𝜖 ₁ to each prediction neuron response (even when 𝜖 ₁ is very small) will cause the responses of the reconstruction neurons to be elevated, and the error neurons responses to be suppressed, which will in turn effect the prediction neuron responses. The second change above reduces this effect of 𝜖 ₁ on the neural responses. The first and third changes allow 𝜖 ₁ to be given the largest value possible (which speeds-up convergence to the steady-state) while preventing 𝜖 ₁ from effecting the responses.

In addition, in some earlier implementations of the PC/BC-DIM model, the reconstruction has been used purely as a means to calculate the errors, and hence, Eqs. 1 and 2 have been combined into a single equation. Here, the underlying mathematical model is identical to that used in previous work, but the interpretation has changed in order to consider the reconstruction to be represented by a separate neural population. This change, therefore, has no effect on the current results. However, other recent results have shown that a separate neural population encoding the reconstruction can perform a useful computational role [42, 64, 65].

Open-source software, written in MATLAB, which performs all the experiments described in this article is available for download from: http://​www.​corinet.​org/​mike/​Code/​pcbc_​image_​recognition.​zip.

To test the ability of the proposed method to categorize images with tolerance to within-class variation, it was applied to the MNIST hand-written digits dataset.⁴ This datset consists of 28-by-28 pixel grayscale images of isolated digits. The training set contains 60,000 images and the test set contains 10,000 images. For this task, the following parameters were used: the similarity threshold for the clustering performed on the image patches was set equal to κ = 0.85; the threshold on the number of patches in each cluster was set equal to λ = 0; and the standard deviation of the Gaussian used to pre-process both the images and RFs of the first processing-stage was set equal to σ = 4 pixels. After pre-processing, each individual input image was rescaled to fill the range [0,1]. The training procedure for the first processing stage (see the “Training” section) produced a dictionary containing 35,956 elements. Examples of these dictionary elements are shown in Fig. 2a.

This dictionary was used to define the weights for 35,956 prediction neurons in the first processing stage (see the “Training” section). As there were ten classes, the second processing stage contained ten prediction neurons. The responses of the first- and second-stage prediction neurons to two test images are shown in Fig. 2c, d. When tested on all images from the test set, it was found that 2.19 % of these images were misclassified. Examples of incorrectly classified test images are shown in Fig. 2b. The classification error of the proposed method is compared to those of a variety of other algorithms in Table 1. It can be seen that while the results of the proposed method are good, they fall far short of the current state-of-the-art.

Most of these state-of-the-art algorithms are deep hierarchical neural networks. Deep architectures can be sub-divided into two main types: (1) stacked generative models, such as deep belief networks [54, 55], and stacked autoencoders [56‐58]; and (2) discriminative models with alternating layers of feature detection and pooling, such as convolutional neural networks CNN;[36‐41], HMAX [20, 33‐35, 61], and Neocognitron [30‐32].

In common with architectures of the first type, the proposed algorithm also employs a hierarchy of generative models. However, the generative models are implemented using a different algorithm: PC/BC-DIM. Furthermore, PC/BC-DIM employs the generative model during inference: the generative model is used to make predictions of the expected sensory inputs, and through the iterative activation dynamics described by Eqs. 1 to 3, determine the prediction neuron activations that minimise the discrepancy between the predicted and actual inputs. In contrast, autoencoders and restricted Boltzmann machines RBM;[84, 85] which are the building blocks of previous architectures of the first type, only employ the generative model during learning. Once the weights have been set to allow these models to reconstruct the input, new inputs are processed using the feedforward weights only.

In common with architectures of the second type, the proposed algorithm has alternate processing stages that specialize in creating more discriminate representations in one layer, and more invariant representations in the next layer. This is achieved by defining the weights differently, but by applying the same algorithm to determine the neural activations during inference. In contrast, existing architectures of the second type use completely different mathematical operations to perform these two functions. For example, more specialized representations are often created by applying a linear filtering operation, while more tolerant representations are usually formed by finding the maximum response within a sub-population of pre-synaptic neurons. The proposed model is thus simpler, in that it only requires one type of processing stage.

Another difference between the proposed architecture and deep architectures of both type 1 and 2 is that in the proposed model, classification is performed by the last processing stage of the PC/BC-DIM hierarchy. In contrast, most existing deep architectures are used only as a method of feature extraction [57] to provide input to a distinct classification algorithm, such as a support vector machine (SVM) or a logistic regression classifier. The proposed model is thus simpler, in that it integrates feature extraction and classification within a single homogeneous framework, rather than using different methods for each.

However, as illustrated by the results in Table 1, deep architectures have an advantage in terms of classification accuracy. There are many reasons for this. Firstly, it is known that the deeper the architecture, the better the performance [86]. The proposed architecture is very shallow compared to most deep architectures. Creating deeper PC/BC-DIM hierarchies by stacking more processing-stages, might thus allow better performance, and potentially create a better model of the ventral pathway. However, doing so will require more sophisticated methods of defining the weights in those processing stages. The current model uses an unsupervized learning method. In contrast, much of the success deep architectures derives from using supervised learning. Using more training data is also known to generally improve performance. One way to generate additional training data is to generate images that are affine deformations of the original training images. This can result in a significant improvement in performance. For example, [83] report an error rate of 0.35 % on MNIST with deformation, and 1.47 % without.⁵ Expanding the dataset in this way could also be used to potentially improve the performance of the proposed PC/BC-DIM architecture. State-of-the-art performance on many classification tasks has been generated using an ensemble of deep architectures [36]: where multiple, different, deep networks are used to independently classify the input, and the final classification is a combination of these individual classifications. If classification accuracy, rather than biological-plausibility, were the main motivation then using the current architecture as the building block for an ensemble might also be considered.

To test the ability of the proposed method to perform sub-ordinate level categorization (i.e., identification) with tolerance to illumination, it was applied to the cropped and aligned version of the Extended Yale Face Database B⁶ [87, 88]. This dataset consists of 168-by-192 pixel grayscale images of faces taken from a fixed viewpoint in front of the face under varying lighting conditions. There are approximately 64 images for each of 38 individuals. Following the method used in previous work with this dataset [89‐93], half the images for each class were used for training and the other half for testing.

In previous work, classification has been performed using images down-sampled to 21-by-24 pixels (or fewer). This has been necessary as previous methods have used pre-processing steps (such as the calculation of Eigenfaces and Laplacian-faces) that are too memory intensive to be performed on larger images [89]. To allow a direct comparison with this previous work results are presented for the proposed method using images that have also been resized by a scale factor \(\delta =\frac {1}{8}\) to 21-by-24. However, as the proposed method can work successfully with larger images, results are also presented for images at the original size (i.e., for δ = 1).

For this task, the following parameters were used: the similarity threshold for the clustering performed on the image patches was set equal to κ = 0.9; the threshold on the number of patches in each cluster was set equal to λ = 0; and the standard deviation of the Gaussian used to pre-process both the images and the RFs of the first processing-stage was set equal to \(\sigma =2.5\sqrt {\delta }\) pixels. After pre-processing, each individual input image was rescaled to fill the range [0,1]. For the 21-by-24 pixel images, the training procedure for the first processing stage (see “Training” section) produced a dictionary containing 806 elements. Examples of these dictionary elements are shown in Fig. 3a. This dictionary was used to define the weights for 806 prediction neurons in the first processing stage (see “Training” section). As there were 38 individuals, the second processing stage contained 38 prediction neurons. The responses of the first- and second-stage prediction neurons to two test images are shown in Fig. 3c, d. The incorrectly identified test images, for the 21-by-24 pixel version of this task, are shown in Fig. 3b. It can be seen that all the misclassified images were taken under very poor lighting conditions.

The classification error of the proposed method is compared to those of a variety of other algorithms in Table 2. It can be seen that the performance of the proposed method is competitive with the current state-of-the-art for this task. The current state-of-the-art algorithms are based on sparse coding. These algorithms represent the image using a sparse set of elements selected from an overcomplete dictionary. They then perform classification by analysing the reconstruction errors produced by dictionary elements associated with different classes [71, 75, 89, 93]. In common with these algorithms, PC/BC-DIM also represents the input images using a sparse code (examples can be seen in the lower histograms in Fig. 3c, d, where it can be seen that only a very small subset of the first stage prediction neurons are active). However, in contrast to most existing sparse dictionary-based classifiers, the proposed method makes the classification using the sparse code (the prediction neuron responses) rather than the reconstruction error (the error neuron responses). This latter method is more biologically-plausible, but less accurate [71]. It has been found that the performance of sparse dictionary-based classifiers is improved by the supervised learning of more discriminative dictionaries [75, 82, 92, 94‐96]. Such learning might potentially also improve the performance of the proposed algorithm.

To test the ability of the proposed method to localize and recognize objects in natural images with tolerance to position, illumination, size, partial occlusion, and within-category shape variation, it was applied to the UIUC cars dataset [97, 98].⁷ This dataset consists of greyscale images of outdoor scenes. The training set consists of 550 car images and 500 images that do not contain cars. There are two sub-tasks: recognising side views of cars at a single scale (the location and number of cars varies between test images), and recognizing side views of cars across multiple scales (the size, location, and number of cars varies between test images). For the single-scale task, the test set contains 170 images containing 200 side views of cars. The multi-scale task has a test set of 108 images containing 139 cars.

The same training set, and the same parameter values, were used for both sub-tasks. Specifically, the similarity threshold for the clustering performed on the image patches was set equal to κ = 0.4, the threshold on the number of patches in each cluster was set equal to λ = 12, and the standard deviation of the Gaussian used to pre-process both the images and the RFs of the first processing stage was set equal to σ = 3.5 pixels. Training of the dictionary used to define the weights of the first processing stage was performed on 15-by-15 pixel patches extracted from the training images around keypoints located using the Harris corner detector. For the single-scale task, the patches taken from the car images were clustered into 273 dictionary elements. The non-car image patches were clustered into 140 dictionary elements. Examples of these first-stage dictionary elements are shown in Fig. 4a. These dictionary elements were used to define the RFs of the prediction neurons in the first PC/BC-DIM processing stage, resulting in 413 prediction neurons at each pixel location in the input image. For the multi-scale task, training was performed on the 1050 car and non-car training images resized to six different scales. The dictionary consisted of 2465 elements representing non-car parts and 3601 elements representing car parts, resulting in 6066 first-stage prediction neurons at each pixel location.

Figure 4b shows two example test images for the single-scale task on which have been superimposed dots to show locations where there is a strong response from the sub-population of first processing stage prediction neurons that represent car parts. The size of the dot is proportional to the magnitude of the response of the prediction neuron. For prediction neurons whose RFs were defined using the same dictionary element, non-maximum suppression was performed over those prediction neuron responses, so that all response other than the local maximum were set to zero.

For the single-scale task, the number of second-stage prediction neurons was equal to the number of pixels in the input image. Each second-stage prediction neuron had the same weights (but at spatially sifted positions), equal to the summed response of all the first-stage prediction neurons to all the car images in the training set. However, to improve tolerance to position, these weights were smoothed across space by convolving them with a two-dimensional circular symmetric Gaussian function with a standard deviation of two pixels. Figure 4c shows the responses of all the second-stage prediction neurons for the two images shown in Fig. 4b. For the multi-scale task, the second processing-stage consisted of six sub-populations of prediction neurons (one for each scale), each sub-population contained one prediction neuron for each pixel in the test image. In this case, the weights were smoothed across space and scale using a three-dimensional Gaussian function.

To determine the location of cars predicted by the proposed method, the spatial distribution of prediction neuron responses (as illustrated in Fig. 4c) was analyzed to find the coordinates of spatially contiguous regions of strong activity. Such a region was defined as a contiguous neighborhood in which each neuron had an activity of more than 0.001, and which was completely surrounded by neurons with a response of 0.001 or less. The coordinates represented by such a region were then determined using population vector decoding [99]. This simply calculates the average of the coordinates represented by the neurons in the region, weighted by each neuron’s response. For the multi-scale task, the coordinates of regions of high activity were determined in the same way, but in a three-dimensional space (position and scale). The total sum of the response in each region was also recorded.

To quantitatively assess the performance of the proposed algorithm, the procedures advocated in [98] were followed. Specifically, for each region with a total response exceeding a threshold, the location (and scale) represented by that region were determined (as described in the preceding paragraph) and these values were compared to the true location (and scale) of each car provided in the ground-truth data. The comparison was performed using the java code supplied with UIUC cars data set. If the predicted parameter values were sufficiently close to the ground-truth, this was counted as a true-positive. If multiple regions of high activity corresponded to the same ground-truth parameters, only one match was counted as a true-positive, and the rest were counted as false-positives. All other regions of high activity that failed to match the ground-truth data were also counted as false-positives. Ground-truth parameters for which there was no corresponding values found by the proposed method were counted as false-negatives. The total number of true-positives (TP), the number of false-positives (FP), and the number of false-negatives (FN) were recorded over all test images, and were used to calculate recall (\(\frac {\text {TP}}{\text {TP}+\text {FN}}\)) and precision (\(\frac {\text {TP}}{\text {TP}+\text {FP}}\)). By varying the threshold applied to select regions of high activity, precision-recall curves were plotted to show how detection accuracy varied with threshold. To summarize performance, the f score (\(=\frac {2.\mathrm {recall.precision}}{\text {recall} + \text {precision}}=\frac {2\text {TP}}{2\text {TP}+\text {FP}+\text {FN}}\)) which measures the trade-off between precision and recall, was calculated at the threshold that gave the highest value. In addition, to allow comparison with previously published results, the equal error rate (EER) was also found. This is the percentage error when the threshold is set such that the number of false-positives equals the number of false-negatives.

The precision recall curve obtained on the UIUC single-scale cars dataset is shown in Fig. 5. The f score was 0.9975 and the EER was 0.5 %. Figure 5b, c shows the only two images in the test set on which the proposed method makes a mistake at the threshold for equal error rate. The results obtained on the UIUC multi-scale cars dataset are shown in Fig. 6. In this case, the f score was 0.9718 and the EER was 2.9 %. These results are compared to those of other published methods in Table 3. It can be seen that the proposed method is competitive with the state-of-the art, and particularly, that it outperforms the method described in [44]. That method is similar to the one proposed here, except that the first processing-stage described here was replaced by a process that found keypoints in the image, and matched (using the ZMNCC as the similarity metric) the image patches around these keypoints to elements in the dictionary. Hence, the method proposed here is simpler, in that both stages are implemented using PC/BC-DIM, rather than being implemented in completely different ways.

The algorithm described in [44] was inspired by the implicit shape model ISM;[100], which employs the generalised Hough transform [109‐111] to allow dictionary elements that match features in the image to cast votes for the possible location and scale of the to-be-recognised object. Once all the votes have been cast, ISM uses a minimum description length (MDL) criteria to reject false peaks caused by votes that come from image elements which have also voted for other peaks that are more likely to be the true ones. The second processing stage in the proposed model can also be thought of as implementing the voting process of the generalized Hough transform, but using explaining away (rather than MDL) to suppress false peaks [44]. In a previous section, the function of the second processing stage was described as being analogous to the function of the pooling stages in deep neural networks. There is therefore also an analogy between the Hough transform and pooling. Both attempt to allow recognition with tolerance to location, but the Hough transform is both less constrained and less arbitrary than the pooling used in deep networks.