Switch or stay? Automatic classification of internal mental states in bistable perception

In this study, features were derived to capture the spectral content of the large-scale brain oscillations as they have been successfully used in the sensory and cognitive processing (Buzsáki and Watson 2012; Ward 2003) including bistable perception (İşoğlu-Alkaç et al. 1998; Isoglu-Alkaç et al. 2000; Strüber and Herrmann 2002). We used the wavelet transform that decomposes the signal into a number of scaled and shifted version of the basis function often called, ‘mother wavelet’. There exist several wavelet transforms depending on the characteristics of the mother wavelet. To extract the desired information, a mother wavelet is chosen in such a way so that it matches with the waveform of the signal. Morlet wavelet has been chosen in a number of EEG and MEG studies (Isoglu-Alkaç et al. 2000; Ghuman et al. 2011). In this work, therefore, we used complex Morlet wavelet (Cohen 2014) for feature extraction which is mathematically represented by a complex sine wave, multiplied by a Gaussian window as shown in Eq.(1).where \(A = 1/{(s\sqrt{\pi })}^{1/2}\) is the frequency band specific scaling factor, s is the standard deviation of the Gaussian wave, f is the frequency of the wavelet basis, \(t=nT_{s}\), \(T_{s}\) is the sampling period and n is the index of samples. Here, f was logarithmically scaled between 1 Hz and 80 Hz and sampling frequency, \(1/T_{s}\) was 1000 Hz.

Let, \(X_{g}(t)\) be the \(g\)th time series. We define \(Y_{g}(t,f)\) as the wavelet transform of signal \(X_{g}(t)\) which can be expressed as the inner product between the signal \(X_{g}(t)\) and complex Morlet wavelet cmw(t, f).In this study, the spectral information was divided into six frequency bands: delta band (1–4 Hz), theta band (4–8 Hz), alpha band (8–13 Hz), beta band (13–30 Hz), lower gamma band (30–50 Hz) and upper gamma band (50–80 Hz). To obtain frequency band specific activity, square operation was performed on \(Y_{g}(t,f)\) followed by averaging over the frequency points that lie in that particular frequency band. This is defined by Eq.(3).where \(N_{B_{b}}\) denotes the number of frequency bins in the \(b^{th}\) band (\(b = 1,2,\ldots ,6\)).

The averaging of the band-specific energies over time series (sensors) was performed in three ways to extract three types of features. For extracting global features, the averaging was performed over all the considered sensors. The left and right hemispheric features were extracted by averaging over the sensors that belong to the left and right hemisphere, respectively. To derive local features, sensors were grouped in 10 clusters (left and right frontal (LF, RF), left and right fronto-temporal (LFT, RFT), left parieto-temporal (LPT), right occipito-temporal (LOT), left and right parietal (LP, RP), posterior parietal (PP), left occipital (LO)) by applying Ward’s clustering algorithm (Ward 1963) to the position of the sensors. The aim of Ward’s algorithm is to construct the clusters in such a way that within-cluster variance is minimized. Thus, the local features were computed in 10 cortical regions by averaging over sensors that belonged to a particular cluster. As the perceptual moment was indicated by a button press, there is a need to exclude the motor-related activity to avoid its influence on the classification performance (Wang et al. 2013). Therefore, while computing the features, in the first place, we excluded 36 sensors from somatosensory and motor regions of both the hemispheres.

In the next step, the total time duration of T ms was divided into L number of segments, and for each segment, power was computed. The parameters T and L were empirically determined (see “Results” section). While the global and hemispheric features approximated the temporal dynamics of the power in six frequency bands over the whole cortical and hemispheric region, respectively, the local features captured the same but retaining individual attributes of 10 cortical regions. The detailed block diagram of the feature extraction technique is shown in Fig. 3.

The underlying neural sources were reconstructed using Linear Constraint Minimum Variance (LCMV) method (Van Veen et al. 1997). In this method, a bank of spatial filters was designed. Filter weights were chosen in such a way so that, it passed brain electrical activity from a specified location while attenuating activity originating at other locations by minimizing the filter output power subjected to a linear constraint.

For source analysis, forward models were constructed based on a standard structural T1-weighted MRI template “Colin27” (Holmes et al. 1998). For beamformer solution, the covariance matrix was calculated by considering the epochs—1800 ms to 0 ms and also 0 ms to 1800 ms where 0 ms indicated the time instant when the perceptual transition was reported. The dipoles were assumed to be located at the voxels within the head boundary (the only grey matter was considered) on a 3D grid with 5 mm spacing. This resulted in 11,780 positions where sources were to be localized. Subsequently, beamformer filters were designed to pass the signal of interest at these locations and attenuate the rest. Time series data were then projected through the resulting beamformer coefficients to produce time courses. According to Automated Anatomical Labelling (AAL), the grey matter of the brain was divided into 116 regions. For each brain region, the voxels that belong to that region were identified, and the corresponding time series were grouped. We considered the principal eigenvector of the grouped time series that represented a particular region (Friston et al. 2006). Thus, there were 116 time-series, each representing one region. Twenty-seven out of 116 regions were not considered which are often involved with the following functions: finger movement; contralateral finger, hand, and wrist movement; movement initiation and movement preparation¹. Thereby, 89 time-series were considered for further processing. The wavelet features were extracted from these 89 time-series by following the method explained in “Wavelet based features extraction” section. However, the step of averaging over time series was not considered in this case, since we wanted to capture the activity in these 89 brain regions, individually.

Due to the wide variability of perception switching rate across individuals, the number of trials differed from participant to participant, resulting in very few trials for some cases. Thus, personalized models were not considered, and we only performed classification at the group level by pooling trials across all participants (Table 2). We used SVM (Vapnik 2013) classifier with Radial Basis Function (RBF) as the kernel function, and also, ANN classifier. The application of SVM and ANN classifier has been found in various studies related to the classification of to neuro-signal (Alimardani et al. 2013; Subasi 2005). For ANN classifier, the number of nodes at the hidden layer was fixed at 10. Scale-conjugate gradient backpropagation algorithm was used to train the model. Mean square error was set to 10⁻⁵, and the hyperbolic tangent sigmoid transfer function was used as the activation function.

The performance of the classifier was evaluated using 10-fold cross-validation. It is considered more reliable compared to leave-one-out cross-validation (Varoquaux et al. 2017). Of note, the data to train the classifier and the data to test its performance were mutually exclusive. The performance of the classifier was measured by accuracy, sensitivity, and specificity; where sensitivity and specificity quantify how accurately the model was able to detect the transition, and maintenance states, respectively. All analysis was performed using customized scripts using MATLAB 2013a.

The participants indicated perceptual reversal by pressing a button. However, the reaction time might vary across participants as well as from trial-to-trial. Additionally, some transition occurs instantaneously whereas “other transitions comprise the dynamic mixture of both the percepts for variable periods before one percept dominates completely” (Knapen et al. 2011), thus leading to a variable duration of the transition state. To determine the effective time-duration that best represents the transition state, we considered the different extents of transition state varying from 600 to 1800 ms in steps of 300 ms. To retain the same feature dimension for both the classes, the length of the maintenance state was kept the same as the length of the transition state. Figure 4a shows the different transition and maintenance states considered for the analysis. For each case, the global feature was extracted from the alpha, beta, lower gamma, and upper gamma frequency band using segment length \(l= T/L\) of 300 ms and then used in classification. Table 3 shows the feature dimension for different length of the transition state. The classification performance, i.e. accuracy, sensitivity, and specificity concerning the length of the transition state, are shown on the left panel of Fig. 4b.

Since accuracy, sensitivity, and specificity for the length of 1200 ms were comparatively good and adequate for all the conditions and both classifiers, it was considered as the effective length of the transition state and used for further analysis. Next, we analyzed the performance variation with respect to the segment length, which was kept 300 ms throughout the previous analysis. Here, we varied the segment length from 100 to 600 ms (in steps of 100 ms), and the extracted features were used for classification (right panel of Fig. 4b). The classification performance remained relatively stable across various segment lengths. Since the segment length of 300 ms yielded the best accuracy, it was considered for further analysis.

For classification of mental states, three types of features were considered: (1) global features, (2) hemispheric (left and right hemispheric) features, and (3) local brain region-specific features. Along with the six conditions listed in Table 2, we also pooled the trials irrespective of types of condition, and their presentation mode is referred as ‘ACT’ (all conditions together) in this paper.

We computed global, hemispheric, and local features in the alpha, beta, lower gamma, and upper gamma frequency band for transition and maintenance states of a duration of 1200 ms. The feature dimension for global, hemispheric, and local features were 16, 16, and 160, respectively. A 10-fold cross-validation was performed to evaluate the performance. In the case of local features, we used PCA to reduce the feature dimension. The PCA was applied to the training data of 9 folds, and the new dimension was determined from the principal components that capture \(C_{v}\%\) (cumulative variance) of the total variance. The cumulative variance was varied from 95 to 99.9%, and the classification performances of training data were computed using 10-fold inner cross-validation. The \(C_{v}\) and the new dimension was determined from the value of the cumulative variance at which the best performance was obtained. Using this new dimension, the classification model was built from training data (9 folds), and the test performance was determined from the remaining one fold. This procedure was repeated in each run of 10 fold cross-validation, and average test performance was computed. Table 4 shows the classification performance of different features using SVM and ANN classifiers for each of the six conditions and also for ACT. We observed that the classification performance of all the features was substantially higher than the empirical chance level. Of note, while the theoretical chance level accuracy is 50% as there are only two classes, the chance level accuracy is liable to increase in the presence of a small number of data samples and is referred to as the empirical chance level. It was determined using the method in Combrisson and Jerbi (2015), which assumes that the classification errors obey a cumulative binomial distribution.

Among the wavelet features, the local features performed better than global and hemispheric features; the former yielded a better balance between sensitivity and specificity for both SVM and ANN classifiers. The sensitivity for global and hemisphere-specific features for all condition was higher than the indicating that the transition state was classified more accurately than the maintenance state at the global and hemispheric brain level. The left hemispheric features performed relatively better as compared to that of right hemispheric features indicating a greater contribution of left hemispheric features towards distinguishing the two classes. Alternatively, these two types of features may also capture complementary information. Thus, we performed an analysis by predicting the states by combining the scores obtained from both the features. The integration of the scores from left and right hemispheric features (\(S_{l}\) and \(S_{r}\)) was performed by the given equation,The parameter p was varied from 0 to 1, with a step size of 0.1. Lower the value of p, lesser the weight assigned to the left hemispheric features. As p was varied from 0 to 1, the classification performance (Fig. 5) attained maximum value for mid-range values of p, essentially indicating the contribution of both the features in classification.

The bias of global and hemispheric features towards classifying the transition state was balanced with the use of local features. We observed an average increase of approximately 9% in specificity as compared to that of the global and hemispheric features. The strength of this approach was its ability to detect both the transition and maintenance state with comparable accuracies. In the case of ANN classifier, a marginal relative improvement in performance as compared to other features was observed. However, the specificity for ACT condition increased by 4.66% and 3.18% using SVM and ANN classifier, respectively. Further, it was found that the reduced dimension of the feature vectors appeared in the range of 3–65 in case of SVM classifier and 3–37 in case of ANN classifier (Fig. 6). It was noted that the number of reduced feature dimension was lesser in each condition as compared to that of ACT condition. This indicated the existence of the variation among the different conditions, which was captured by a comparatively larger number of principle components in ACT.

The classification performance of the wavelet-based features was compared with the performance of the power spectral density features. The power spectral density was estimated in six frequency bands using the Welch’s method. The features were extracted in ten cortical regions and from four frequency band similar to wavelet-based local features. The performance of the features was evaluated using both SVM and ANN classifier, and the results are reported in Table 4. It can be observed that the wavelet-based feature outperformed the power spectral density features, indicating the strength of temporal information captured by the wavelet-based features in discriminating the hidden mental states. The standard deviations of classification accuracies for wavelet as well as power spectral density features were comparatively high for some of the conditions due to the smaller number of samples. However, the ACT condition, which consists of a larger number of samples, yielded a much less standard deviation of accuracy.

Since the process of transition is expected to occur rapidly, the duration of the transition state is supposed to be short-lived. However, the maintenance state could extend up to the start of the next transition state, and it can be much longer than the considered 1200 ms duration. It would, therefore, be interesting to investigate how precisely we could detect the maintenance state over this extended duration. We explored this possibility by shifting a 1200 ms sub window representation of maintenance state away from the button press with the step size of 100 ms to cover a zone from button press to 1800 ms (Fig. 7a). All these maintenance periods were pooled together to evaluate the performance of the classifier, and the detection performance for each of them was noted. Consequently, the number of trials for the maintenance state was seven times more than that of the transition state. The performance of the classifier was evaluated using local features in the classification framework as described in the previous section in conjunction with random subsampling (Kubat et al. 1997) approach to address the class data imbalance. The classification performance for ACT condition and average performance over all the individual conditions are reported in Fig. 7b. The average accuracy over the individual conditions and accuracy for ACT condition were respectively 74.10% and 77.59% using SVM classifier, 71.66% and 76.23% using ANN classifier. A balance between the sensitivity and specificity was observed in each case. Further, to investigate the effect of different sub-windows of maintenance state on classification performance, the average specificity across conditions and specificity for ACT were plotted against individual subwindows of the maintenance state (Fig. 7c). It was found that the specificity of the classifier first increased up to second sub window, maintained a good classification performance during second to fifth sub window and, decreased afterwards as maintenance state was shifted away from the moment of the perceptual switch. However, the average specificity across the condition and specificity of ACT did not decline below 69.14% and 71.94%, respectively using SVM classifier and were found to be 66.57% and 72.37%, respectively using ANN classifier.

In this section, we investigate the brain regions involved in the transition and maintenance state. For that purpose, we estimated the time series data in source domain reconstructed from MEG data recorded at sensor level and extracted the features in source domain by following the same feature extraction method as discussed in “Wavelet based features extraction” section and “Source reconstruction” section, respectively. This analysis in source domain would identify the neural sources in the brain rather than its projected effect on the scalp level. We have employed the framework used in the previous section (“Temporal evolution of internal mental states” section) that took into consideration the wider maintenance state duration. We considered local features extracted from reconstructed sources in six frequency bands and 89 brain regions for classification using SVM and ANN classifier. Here, we used F-ratio based feature selection technique instead of PCA, as in the latter, data was projected on the direction of principal components that leads to loss of information as regards to one specific source of the feature. F-ratio is computed as the ratio of between-class variance to the total within-class variance. A larger F-ratio indicates a greater separability between the classes, thus essentially implying more effective feature to discriminate the classes. The features with largest F-ratio were given highest priority. Thus, besides the classification performance, this analysis also found discriminative and dominant features by analyzing the features which were selected at the training stage. This, in turn, led to the identification of the underlying neural sources which responded differently during two mental states within a particular frequency band.

In this method, we were able to classify the mental states with an average (averaged over six conditions) accuracy of 73.55% and 71.19% using SVM and ANN classifier, respectively; 74.98% and 73.05% accuracy using SVM and ANN classifier, respectively when ACT condition was considered. We analyzed the source features that were selected at the training stage in each fold. Figure 8 shows the distribution of selected features over different frequency bands. The dominance of features from alpha, beta, lower gamma, and upper gamma frequency band was noticed, consolidating our earlier sensor space based findings. The distribution of the selected features over brain regions (Fig. 9) showed that the features were selected in the majority from parietal, occipital and cerebellum area. Thus, the sources of these areas were modulated differently during the state of transition and maintenance. Figure 10 shows the distribution of the selected features over anatomical brain regions for each of the highly contributing frequency bands for ACT conditions. The weights indicate the fraction of the number of selected features from that region. The weights of the excluded regions were set to zero for visualization. The rightmost panel shows the coarse labelling of grey matter according to AAL for reference. The features from parietal regions were found to be consistently selected in the majority from alpha, beta, lower and upper gamma frequency band. Besides this, the occipital and cerebellum regions were found to be modulated distinctly for two considered mental states in alpha, beta and lower gamma frequency bands. The analysis of selected features using SVM and ANN classifier showed a similar result. In Figs. 8, 9 and 10 we have presented the result of the analysis using SVM classifier which are similar for ANN classifier.

In the above sections, we evaluated the performance of different features under six conditions (three types of stimuli and two ways of presentation). For evaluating the performance for a condition, the data of that condition was divided into training and testing sets; training data was used to build the model and testing data to evaluate the model performance. Again, in the case of ACT, where the data from all the condition were pooled together, the data were treated irrespective of the corresponding condition and followed the same procedure to evaluate the performance. It is interesting to investigate the generalisation of classifier across different stimulus types. Therefore, in this section, we have considered the cross-conditional classification by building the model using the data of one stimulus and testing on the data of others. Instead of considering all the conditions separately, only three stimuli factors (Necker Cube, Stroboscopic Alternative Motion and Structure From Motion) were taken into account irrespective of the stimuli presentation style. This study reveals the cross-conditional generalization power of the features (Peelen et al. 2010) we used for classification purpose. Table 5 represents the cross-classification performances of sensor-based global, hemispheric and local features using SVM and ANN classifiers. This analysis was extended to senor and source based local features in the framework explained in “Temporal evolution of internal mental states” section and the result is presented in Table 6. It was found that cross-conditional classification performances were comparable with the classification result obtained in the previous three sections. We may conclude that brain responses captured by wavelet features are generalized in discriminating maintenance and transition states across different stimulus.