A Deep Convolutional Spiking Neural Network for embedded applications

Convolutional neural network (CNN) is a special sort of ANNs that have fewer connections and parameters making their training easier while maintaining competent performances [34]. Figure 2 shows a classic CNN, LeNet, which includes four types of layers: an input layer, convolutional layers, pooling layers, and fully connected layers [35]. The capacity of the network can be controlled by varying its parameters such as the shape of the input (batch size, height, width, channels), the number of channels in the first and second convolution layers, the number of neurons in the hidden layer, and the kernel size for the convolution and pooling filters.

The initial phase of an SNN is to transform the original features into spikes to feed the spiking networks. The encoding strategy has a significant effect on network performance. Selecting the best coding strategy depends on the application, hardware limitations, and the neuron model [36]. In general, three encoding mechanisms have been popularised with respect to input data.

An ANN consists of at least two fundamental components: processing units (neurons) and connections (weights). Each neuron receives an input, uses its transfer function to process it, and then produces an output according to its activation function, as shown in Fig. 3a. The transfer function \(T{F}_{J}\) is the weighted sum of the inputs \({I}_{1\to M}\) with respect to the weights \({\omega }_{1\to M,J}\) of the links connected to the neuron \(J\). Accordingly [40]:

The output \({y}_{J}\) of the \(J{\text{th}}\) neuron is expressed as \({y}_{J}={{\text{AF}}}_{J}({{\text{net}}}_{J})\) where the activation function \({{\text{AF}}}_{J}\) is often the rectified linear units (ReLU) function.

The spiking neuron receives inputs from presynaptic neurons over multiple timesteps, while in ANNs, each neuron’s state is updated periodically [41]. The leaky integrate-and-fire (LIF) is a first-order model and one of the simplest and most used neuron models, thanks to its computational efficiency. The basic concept of the LIF model is that the neuron has a membrane potential that evolves through time due to incoming excitatory or inhibitory currents and a leakage parameter [42]. An output spike occurs whenever the membrane potential of a neuron crosses its threshold, and its membrane potential is then reset to its resting state, as shown in Fig. 3b. This behaviour is characterised as:where \(V (t)\) is the membrane potential, \(L\) is the leakage parameter, \(X(t)\) is the summation of all synaptic inputs at time t, \({V}_{{\text{rest}}}\) is the resting membrane potential, and \({V}_{threshold}\) is the membrane threshold. To propose a hardware friendly method, we use no leakage (L = 0), which gives the integrate-and-fire (IF) neuron model.

In this paper, we investigate a rate encoded SNNs, so the basic LIF model as presented in the previous section has an issue when used for rate-based computations that happen close to zero: the output result is dependent on the order of arriving spikes, which is not desirable in a rate-based context. Specific care should be given to low firing rates and low threshold values. To illustrate this, let us consider a LIF neuron J with two different inputs \({I}_{i}\) weighted by their respective weight \({w}_{i}\). In addition, assume that the \({{\text{V}}}_{{\text{resting}}}\) is set to zero. Let us suppose \({w}_{1}= a < 0\) and \({w}_{2}= b > 0.\) If \({I}_{1}\) spikes before \({I}_{2}\), the output is \({y}_{J} = b\) as \({y}_{J} = (0 - a) + b = 0 + b = b\). However, if the order is reversed, \({y}_{J} = b - a.\) This simple example highlights the limitations of this model. We applied an adapted model to fix this issue: setting the lower bound threshold \({\theta }_{threshold}^{down}\) to a negative value, \({\theta }_{{\text{threshold}}}^{{\text{down}}}=-{\theta }_{{\text{threshold}}}^{{\text{up}}}\) while keeping \({v }_{resting}=0\) and \({\theta }_{{\text{threshold}}}^{{\text{up}}}={\theta }_{{\text{threshold}}}\). This modification mathematically solves the problem around zero. As a matter of fact, \({y}_{J} = b - a\) in both cases now. Yet, the problem still exists around \({\theta }_{{\text{threshold}}}^{{\text{down}}}\). \({V}_{m}\) could be reset to \({v }_{{\text{resting}}}\) without firing or simply kept at \({\theta }_{{\text{threshold}}}^{{\text{down}}}\) (selected in this work).

Once the spike is generated, the membrane potential is reset. We discuss next two types of resets: reset to zero presented by Diehl et al. [43], always sets the membrane potential back to a baseline, typically zero. Reset by subtraction, or “linear reset mode” presented by Diehl et al. (2016); Cassidy et al. (2013), subtracts the threshold \({V}_{{\text{thr}}}\) from the membrane potential at the time when it exceeds the threshold [44, 45]. Accordingly, the adapted model inspired by similar considerations of work in [46‐48] is given in Eqs. (5–8).

If \({\uptheta }_{{\text{threshold}}}^{{\text{down}}}=0\) and \({\uptheta }_{{\text{threshold}}}^{{\text{up}}}={\uptheta }_{{\text{threshold}}}\), this model is equivalent to the basic LIF neuron model. In this work, the modified LIF model is used without leak, i.e. L = 0. We choose F_min = 50 Hz and F_max = 200 Hz in this work.

To convert the original features into spikes to feed the spiking networks, rate strategies are used for encoding information in SNNs, which is based on spike firing rate. To simulate neuron behaviours, a Poisson process is often used to generate spike trains. SNNs receive a series of spikes as input and produce a series of spikes as the output, which is usually referred to as spike trains. It is a process where events are assumed to be statistically independent which gives a good approximation of stochastic neuronal firing. The Poisson process is often used to convert the pixel illumination into a spike train: it produces on average a spiking rate proportional to the illumination of pixels while introducing noise [49]. Figure 4 shows an example of the rate coding.

In this research, we initiated the experiments with one of the standard deep learning models—Visual Geometry Group (VGG16) network [53]. Figure 6 shows the conversion of the VGG16-based ANN convolutional block to an SNN convolutional block. Some changes are needed to be able to transfer the VGG16 network to spiking neurons. We eliminate the bias and batch normalisation, as they lead to accuracy loss during the conversion process. We also swap the max pooling with average pooling in our design. The second step is to train the VGG16 network using the backpropagation method in order to have a robust and accurate trained model. We apply data augmentation techniques including resizing, rotation, horizontal flip, and vertical flip. To reduce overfitting and enhance the generalisation capability of our model, firstly, we use dropout to assist the regularisation process during training due to the absence of bias and batch normalisation. Secondly, we apply early stopping as a regularisation technique to stop the training after a number of epochs when the limited validation dataset no longer improves the performance of the model. Thus, the best weights are saved and updated during training. The final step is transferring the trained weights to a network of IF units. A weight–threshold balancing technique is used to ensure an accurate translation of continuous-valued neuron activations into the firing rates of a spiking neuron. To copy the behaviour of SoftMax to a spiking neuron, we apply an external spike generator. Figure 7 shows the conversion process of Deep CNN to Deep CSNN.

The EuroSAT is a dataset for LULC classification taken from the Sentinel-2 satellite [54]. It includes 10 different scene classes with 27,000 labelled images, and 13 spectral bands. Examples of EuroSAT dataset are shown in Fig. 8. The dataset is divided into 80/10/10 ratios for training, validation, and test, respectively. We apply several augment techniques to avoid overfitting during the training. These techniques include random zoom, resizing, rotation, horizontal flip, and vertical flip.

After training the deep convolutional model, we convert it into an SNN model by replacing the ReLU functions with spiking activations. Finally, we transfer the trained weights to a network of IF units and apply the weight–threshold balancing technique. The best overall model, which uses all 13 bands, accurately classified 94.88% of the testing set over T = 400 timesteps. Figure 9 shows the results of VGG16 (training and validation). Tables 1 and 2 show the corrected prediction result for each class and accuracy of Deep CNN and Deep CSNN models, respectively.

The training stopped at the 57th epoch due to an early stopping strategy. This method prevented the model from overfitting and saved computational time. Based on Table 1, the sea/lake, river, industrial, and forest classes showed the best performance above 96% accuracy on test dataset.

In this research, we propose a deep convolutional SNNs to make epileptic seizure predictions from the EGG samples. The proposed approach consists of the same procedure discussed in Sect. 4. Firstly, a number of modifications have been done to make the VGG16 model transferable to spiking model. After training the model with the ReLU activations function and getting the weights, the model transformed into a spiking domain by replacing the ReLU units with IF units. We also applied a weight–threshold balancing technique to ensure an accurate translation of continuous-valued neuron activations into the firing rates of a spiking neuron. Because the SNN layers only received binary spikes as inputs, the numerical EEG signals must be represented as spikes. We employ rate coding, which encodes each signal value normalised to (0, 1) as a spike train of length T. Figure 10 shows the conversion process of the Deep CNN to Deep SCNN for epileptic seizure prediction.

We used the EEG recordings provided by the children’s Hospital of Boston—Massachusetts Institute of Technology (CHB-MIT) to train and test the model for seizure prediction [56]. It includes 23 individuals and 969 h of scalp EEG recordings. The EEG recordings are divided into 24 sets and were recorded with a 16-bit resolution. Table 3 shows the details of the CHB-MIT EEG dataset.

Each patient’s data is divided into 70/15/15 ratios for training, validation, and test, respectively. We need to mention the raw recorded signals used in this work, as they need less computational resources from a hardware implementation point of view.

To evaluate the performance of Deep CSNN classifier, we used two well-known metrics, namely sensitivity and specificity. Sensitivity is the probability of seizure samples that the model correctly classifies as a seizure. Specificity is the proportion of non-seizure samples that the model correctly classifies as non-seizure.

The patient-specific Deep CNN algorithm was evaluated on the CHB-MIT database. Our results reveal that the sensitivity is 96.4% and the specificity is 99.66%. After converting Deep CNN to Deep CSNN, model was evaluated on test dataset. Our results reveal that the sensitivity is 95.06% and the specificity is 99.45% with T = 200. Figure 11 shows the sensitivity and specificity of the Deep CSNN model on test dataset. Table 4 shows the average sensitivity and specificity of Deep CNN and Deep CSNN models.