ATSFCNN: a novel attention-based triple-stream fused CNN model for hyperspectral image classification

Within CNN structures, a typical arrangement involves a stack of alternating convolution layers, pooling layers, fully connected layers and a loss layer [43]. The convolutional layer plays a crucial role in feature extraction by employing specific kernels. During this process, element-wise multiplication is conducted between input values within the kernel's scope and their corresponding values in the kernel itself. This operation is performed across the entire input, and if multiple kernels are utilized, each kernel undergoes the same procedure. Pooling layers are frequently employed to reduce data dimensionality. Initially introduced in [44], pooling was found to mitigate overfitting by randomly omitting half of the feature detectors during each training instance. Their experiments highlighted the efficacy of this technique, known as Dropout, particularly in tasks such as speech and object recognition. The fully connected layer, on the other hand, enables the extraction of deeper and more abstract features by reshaping feature maps into an n-dimensional vector [43]. The last fully connected layer serves as the loss layer that computes the loss or error which is a penalty for discrepancy between desired and actual output [45]. In the loss layer, various activation functions are theoretically available, including Sigmoid [46], Linear [47], Tanh [48], and ReLU [49].

As shown in figure 1, the theoretical architecture of a 1D-CNN model typically consists of several parts: the Input Layer, Convolutional Layer C1, Convolutional Layer C2, Max-Pool Layer M1, Dense Layer D1, and Output Layer.

Zoom In Zoom Out Reset image size

In a 1D-CNN model, the input layer consists of a 1D array with dimensions of $N_{1} \times 1$. The subsequent C1 Convolutional layer will have F different feature maps, where each feature map is an array with dimensions of $N_{2} \times 1$. If the kernel size is $S_{1} \times 1$, then their relationship satisfies equation (1)

$$\begin{equation} N_{1}-S_{1}+1 = N_{2}. \end{equation} \tag{ 1 }$$

The Convolutional Layer C2 has a similar structure to the Convolutional Layer C1, with the difference being that N₃ is smaller than N₂. When the kernel size is $S_{2} \times 1$, it satisfies a similar equation: $N_{2}-S_{2}+1 = N_{3}$. In the Max-pooling layer, if the kernel size is $S_{3} \times 1$, there exists the relation $N_{4} = N_{3}/S_{3}$. Subsequently, we need to flatten the results to a Dense Layer, and their relationship satisfies $N_{5} = F \times N_{4}$. Finally, for the Output Layer, its dimension N₆ will be the same as the number of different types in the dataset

$$\begin{equation} X_{a,b}^{m} = \left( \sum_{c = 1}^{F} \sum_{i = -\alpha}^{\alpha} \omega_{a,b,c}^{i} Y_{\left(a-1\right),c}^{m+i}\right) + B_{a,b} \end{equation} \tag{ 2 }$$

$$\begin{equation} Y_{a,b}^{m} = \sigma \left(X_{a,b}^{m}\right). \end{equation} \tag{ 3 }$$

The equations presented here denote that, in the ath layer and in the bth feature map, the result is obtained at point m along the 1D array. F represents the total number of all the feature maps in the layer of $(a-1)$th. α represents the extended width of the kernel, thus the width of the kernel is $(1+2\alpha)$. $Y_{(a-1),c}^{m+i}$ represents the activation values in the kernel area of the $(i-1)$th layer in each of the features, while $\omega_{a,b,c}^{i}$ represents the corresponding weights of this specific kernel. $B_{a,b}$ stands for the bias for the ath layer and the bth feature map. Once we obtain the sum of the weighted values and bias $X_{a,b}^{m}$, we apply the activation function σ (as equation (3) suggests), so that this result can be converted into $Y\,_{a,b}^{m}$, which represents the activation value in the ath layer and in the bth feature map.

Figure 2 illustrates the architecture of a 2D-CNN model. Similar to the 1D-CNN model, it consists of several layers, including the Input Layer, Convolutional Layer C1, Convolutional Layer C2, Max-Pool Layer M1, Dense Layer D1, and Output Layer.

Zoom In Zoom Out Reset image size

The 2D-CNN network begins with an input layer that takes in a 2D image with dimensions of $N_{1}\times N_{1}$. The first convolutional layer, C1, consists of F kernels with a size of $S_{1}\times S_{1}$, resulting in F feature maps with dimensions of $N_{2}\times N_{2}$, where $N_{2} = N_{1} - S_{1} + 1$. Similarly, the second convolutional layer, C2, also has F feature maps but with dimensions of $N_{3}\times N_{3}$, where $N_{3} = N_{2} - S_{2} + 1$, when using a kernel size of $S_{2}\times S_{2}$. For the Max-Pooling layer, with a kernel size of $S_{3}\times S_{3}$, the output size is calculated using $N_{4} = N_{3}/S_{3}$. The dense layer, D1, has a size of $N_{5} = F \times N_{4} \times N_{4}$, where F is the number of filters and N₄ is the output size of the Max-Pooling layer. Finally, the output layer, which corresponds to the number of classification types, has a size of N₆

$$\begin{equation} X_{a,b}^{m,n} = \left( \sum_{c = 1}^{F} \sum_{i = -\alpha}^{\alpha} \sum_{j = -\beta}^{\beta} \omega_{a,b,c}^{i,j} Y_{\left(a-1\right),c}^{m+i,n+j}\right) + B_{a,b} \end{equation} \tag{ 4 }$$

$$\begin{equation} Y_{a,b}^{m,n} = \sigma \left(X_{a,b}^{m,n}\right). \end{equation} \tag{ 5 }$$

The above equations describe how the result of a specific point (m, n) in the bth feature map of the ath layer is calculated. In the ath layer, F represents the total number of feature maps in the previous layer, which is $(a-1)$th layer. α and β represent the extended width and height of the kernel, respectively. Therefore, the size of the kernel is $(1+2{\alpha})\times(1+2{\beta})$. $Y\,_{(a-1),c}^{m+i,n+j}$ is the activation value in the kernel area of the $(i-1)$th layer for the cth feature map. $\omega_{a,b,c}^{i,j}$ is the corresponding weight for this specific kernel. $B_{a,b}$ represents the bias for the bth feature map in the ath layer. After obtaining the value of $X_{a,b}^{m,n}$, which is the sum of the weighted values and bias, the activation function σ (as equation (5)) is applied to obtain the activated value $Y\,_{a,b}^{m,n}$ in the bth feature map of the ath layer.

Figure 3 illustrates the architecture of a typical 3D-CNN model. This structure typically consists of several layers, including the Input Layer, Convolutional Layer C1, Convolutional Layer C2, Max-Pooling Layer M1, Dense Layer D1, and Output Layer.

Zoom In Zoom Out Reset image size

The input to a 3D-CNN model typically consists of a 3D dataset with the dimensions $N_{1} \times N_{1} \times F_{1}$. The convolutional layer C1 will have the size of $N_{2} \times N_{2} \times F_{2}$. When the kernel used for the convolutional process is of size $S_{1} \times S_{1} \times S_{1}$, the following two equations must be satisfied:

$$\begin{equation} N_{1}-S_{1}+1 = N_{2} \end{equation} \tag{ 6 }$$

$$\begin{equation} F_{1}-S_{1} = F_{2} \end{equation} \tag{ 7 }$$

The convolutional layer C2 has dimensions of $N_{2} \times N_{2} \times F_{2}$, with a kernel size of $S_{2} \times S_{2} \times S_{2}$ between C1 and C2. This requires satisfying the equations $N_{2}-S_{2}+1 = N_{3}$ and $F_{2}-S_{2} = F_{3}$. During the Max-Pooling process, with a kernel size of $S_{3} \times S_{3} \times S_{3}$, the equations $N_{3}/S_{3} = N_{4}$ and $F_{3}/S_{3} = F_{4}$ are used to calculate the resulting dimensions. The value of N₅ in the Dense Layer D1 is determined by $N_{5} = F_{4} \times N_{4} \times N_{4}$. Finally, the number of classification types determines the value of N₆ in the Output layer

$$\begin{equation} X_{a,b}^{m,n,p} = \left( \sum_{c = 1}^{F} \sum_{i = -\alpha}^{\alpha} \sum_{j = -\beta}^{\beta} \sum_{k = -\gamma}^{\gamma} \omega_{a,b,c}^{i,j,k} Y_{\left(a-1\right),c}^{m+i,n+j,p+k}\right) + B_{a,b} \end{equation} \tag{ 8 }$$

$$\begin{equation} Y_{a,b}^{m,n,p} = \sigma \left(X_{a,b}^{m,n,p}\right). \end{equation} \tag{ 9 }$$

The equations above show the result of point (m, n, p) in the 3D coordinate system in the ath layer and the bth feature map. F represents the total number of all the feature maps in the $(a-1)$th layer. α represents the extended width of the kernel; thus, the width of the kernel is (1 + 2α). β represents the extended height of the kernel; thus, the height of the kernel is (1 + 2β). γ represents the extended depth of the kernel; thus, the depth of the kernel is (1 + 2γ). $Y\,_{(a-1), c}^{m+i,n+j,p+k}$ represents the activation values in the kernel area of the (i − 1)th layer in each feature, while $\omega_{a,b,c}^{i,j,k}$ corresponds to the weights of this specific kernel. $B_{a,b}$ represents the bias for the ath layer and the bth feature map. Once we obtain $X_{a,b}^{m,n}$, which is the sum of the weighted values and bias, we need to apply the activation function σ (as suggested by equation (9)) to convert this result into $Y\,_{a,b}^{m,n}$, which is the activated value in the ath layer and the bth feature map.

Hyperspectral datasets contain both spectral and spatial information. However, 1D-CNNs only extract information from the spectral aspect while ignoring spatial information and information among adjacent slices. On the other hand, 2D-CNNs consider spatial information but fail to extract both spectral information and potential information among adjacent slices. To compensate for the disadvantage, we need to use 3D-CNNs, as they are better suited for capturing information among adjacent slices in hyperspectral datasets. In summary, the imperative lies in obtaining more comprehensive results in the process of feature extraction, which is significant in yielding superior and more robust outcomes during subsequent feature fusion and classification. Therefore, it remains essential to conduct a thorough examination of hyperspectral data from all three pertinent aspects to attain this objective.

In addition, even in the applications of former researches that try to combine information from 1D, 2D, and 3D aspects, they fail to explore effective methods for such feature fusion.

This article proposes a new model called attention-based triple-stream fused CNN (ATSFCNN) to address these issues. Firstly, we extract features from HSIs using the techniques of multi-scale 1D-CNN, 2D-CNN, and 3D-CNN. Secondly, we propose a new attention-based feature fusion method to combine features obtained from 1D-CNN, 2D-CNN, and 3D-CNN. Compared with other naive fusions among the features, our fusion method can stress the features which have a higher significance to the results. Furthermore, our model is scalable, as features generated from other neural networks can replace those generated by 1D, 2D, and 3D networks. In that case, our new feature fusion method proposed in this article from 1D, 2D, and 3D networks remains applicable.

The architecture of ATSFCNN is shown in figure 4. Its architecture consists of four different modules: Data Preprocessing Module, Feature Extraction Module, Feature Fusion Module, and Output Module.

Zoom In Zoom Out Reset image size

The data preprocessing module involves conducting principal component analysis (PCA) on the original hyperspectral dataset to preprocess it. This results in a reduction of the dataset's dimensionality from M×N×P to M×N×L, which serves as input for the 2D stream. In addition, the hyperspectral datacube is transformed into a series of 1D arrays. Each pixel in the 2D (M×N) image is extended in the depth direction, creating M×N 1D arrays. Each array has a dimension of 1×P, which is the input for the 1D stream. The 3D stream uses the original hyperspectral datacube with the size of M×N×P directly.

The choice to use the processed M×N×L datacube for 2D analysis and the original M×N×P hyperspectral datacube for 3D analysis is based on several factors. The 2D approach primarily focuses on 2D spatial information between adjacent pixels, while the 3D approach extracts information primarily from adjacent spectral channels. Therefore, in the case of 3D analysis, it is crucial to retain as many spectral channels as possible, whereas this is not necessary for 2D analysis. In fact, performing PCA on the data before using 2D-CNN can reduce dimensionality, decrease computational workload, and improve classification accuracy. This is demonstrated in later sections by comparing the results of 2D-CNN and 2D-CNN+PCA. As shown in the works of [50], the use of PCA in 2D-CNN can significantly improve HSI classification accuracy.

As depicted in figure 5, the feature extraction module employs the M×N×P dataset directly as input for 3D-CNN. Additionally, the processed M×N×L serves as input for 2D-CNN, while the M×N 1D arrays are used as input for 1D-CNN.

Zoom In Zoom Out Reset image size

In contrast to the conventional CNN model, our approach uses multi-scale feature extraction. The conventional CNN streams comprise Input_iD, ConviD_1, and ConviD_2 (i = 1,2,3). In our method, Multi-scale Feature_iD_1, Multi-scale Feature_iD_2, and Multi-scale Feature_iD_3 are used for Multi-scale feature extraction. Multi-scale Feature_iD_1 represents the convolutional result of Input_iD using a kernel different from that in ConviD_1. Multi-scale Feature_iD_2 is the concatenation of ConviD_1 and the convolutional result of Input_iD. Finally, Multi-scale Feature_iD_3 is the concatenation of ConviD_2 and the convolutional result of ConviD_1

$$\begin{equation} {FiD\_1} = Conv{(IiD}) \end{equation} \tag{ 10 }$$

$$\begin{equation} {FiD\_2} = Concat\left(Conv\left({IiD}\right), Conv\left({CiD\_1}\right)\right) \end{equation} \tag{ 11 }$$

$$\begin{equation} {FiD\_3} = Concat\left(Conv\left({CiD\_1}\right), Conv\left({CiD\_2}\right)\right). \end{equation} \tag{ 12 }$$

The feature fusion module, as depicted in figure 6, incorporates multi-scale features extracted from each stream. Equation (13) outlines the fusion process, which employs direct concatenation among the multi-scale features. This approach has been shown to be effective for multi-scale feature fusion, as demonstrated in the explications and experiments conducted in 2022 [51]. Their findings support the efficiency of concatenation for this purpose

Zoom In Zoom Out Reset image size

$$\begin{equation} {Fc\_i} = Concat\left({FiD\_1}, {FiD\_2}, {FiD\_3}\right). \end{equation} \tag{ 13 }$$

Deep fusion of features extracted from different dimensions presents a challenge due to differences in their dimensions. To address this, a flattening process is applied to each fused multi-scale feature, which is then passed through the same dense layer. This approach standardizes the extracted features from each stream into a 1D feature array of the same size. Subsequent to the Dense layer, each $\textit{FF}_\textit{i}$ (where i assume values of 1, 2, and 3) uniformly assumes dimensions of R × 1, wherein R signifies the extent along the array dimension originating from the Dense Layer

$$\begin{equation} {FF\_i} = Dense\left(Flatten\left({Fc\_i}\right)\right). \end{equation} \tag{ 14 }$$

Following the aforementioned standardization process, the fused multi-scale features undergo two subsequent sub-modules: the concatenation permutation module and the attention module.

3.5.1. Concatenation permutation module

The concatenation permutation module is illustrated in figure 7. Given the three streams of extracted features, there are six possible permutations of their concatenation: Fa_1 (1D-2D-3D), Fa_2 (1D-3D-2D), Fa_3 (2D-3D-1D), Fa_4 (2D-1D-3D), Fa_5 (3D-2D-1D), and Fa_6 (3D-1D-2D). The module considers all of these possibilities for concatenation.

Zoom In Zoom Out Reset image size

Due to the potential impact of the sequence of concatenation on the accuracy of feature fusion and classification, all six possible combinations of concatenation must be considered. In the experiments, one of these combinations is used at a time, with all six combinations ultimately being utilized

$$\begin{equation} {FF\_all} \in \left\{{Fa\_1}, {Fa\_2}, {Fa\_3}, {Fa\_4}, {Fa\_5}, {Fa\_6}\right\}. \end{equation} \tag{ 15 }$$

FF_all is the outcome that follows the execution of the Concatenation Permutation Module. Within this module, the three Multi-scale Fused Feature FF_i (R × 1) were concatenated, consequently leading to its dimensions expanding to $(3R) \times 1$.

3.5.2. Attention module

In 2017, some reseachers proposed a Transformer model, which marked a significant milestone in the application of attention mechanisms to the domain of images [52]. Their experiments demonstrated the model's ability to enhance significant information and remove redundancy. However, the Transformer model presented in their work was complex, requiring multiple hidden layers. Inspired by this, we propose a lightweight attention-based module in this article that achieves similar results.

As shown in figure 8, the Concatenated Fused Feature (FF_all) is inputted into the Channel Attention and Spatial Attention modules. Our model is inspired by the works of [53], who proposed a similar model of Channel Attention and Spatial Attention. However, there are two key differences between our work and theirs. Firstly, while their model operates solely on 2D images, we have adapted our model to the 1D case for feature fusion among the extracted features of three streams of hyperspectral datasets. Secondly, their models were directly tested on image detection, whereas in our article, we propose a novel and unique application of using the model to improve the classification performance of the fused features.

Zoom In Zoom Out Reset image size

3.5.2.1. Channel attention

Channel Attention focuses on the finding out the inter-channel information of the dataset. When constructing our model of Channel Attention, we utilize the Average Pooling as well as Max Pooling.

Initially, during the development of the Channel Attention model, researchers commonly employed Average Pooling as the primary method for feature extraction, as demonstrated in prior works such as [54, 55], which focused on Object Detection and demonstrated the effectiveness of this approach. However, subsequent studies, including [56, 57], have shown that incorporating Max Pooling in addition to Average Pooling can significantly improve model accuracy. Therefore, in the present study, we have opted to utilize both Average Pooling and Max Pooling in our Channel Attention models.

In Channel Attention, the fused feature undergoes both Max Pooling and Average Pooling operations. The outputs of these operations are then passed through a shared network, which generates the respective channel attention maps. To accomplish this, a MLP with a single hidden layer is utilized.

Compared with conventional MLP, a shared MLP is a type of neural network architecture that extracts pertinent features from various inputs, generating intermediate representations that subsequently interface with other facets of the model, such as classifiers or decision-making components. This architectural paradigm embodies the principle of sharing neural network components among diverse data streams, leveraging their collective prowess to amplify the model's overall effectiveness and predictive capacity. The genesis of the Shared MLP can be traced back to pioneering work in the domain of Natural Language Processing in 2017 [58], where its merits were first underscored. Subsequently, this paradigm found extensions in the realm of attention mechanisms [53], augmenting its relevance and utility.

Based on the idea of shared MLP, the process of Channel Attention can be expressed mathematically as shown in equations (16) and (17). CA_map materializes through an automated process that accentuates dominant features while simultaneously reducing the weights of less impactful attributes. Consequently, CA_map corresponds to the features in the FF_all, meticulously representing each feature in a one-to-one correspondence. Thus, the dimensions of the CA_map harmoniously mirror those of FF_all, spanning $(3R) \times 1$

$$\begin{equation} {CA\_map} = MLP\left(Maxpool\left({FF\_all}\right)\right)+ MLP\left(Avgpool\left({FF\_all}\right)\right) \end{equation} \tag{ 16 }$$

$$\begin{equation} {{FF\_CA}} = {FF\_all} \otimes {CA\_map} \end{equation} \tag{ 17 }$$

where ⊗ denotes the matrix multiplication.

3.5.2.2. Spatial attention

Spatial Attention focuses on exploiting the inter-spatial information of the dataset. As we have explained in the Channel Attention, in Spatial Attention, we still use the Max Pooling and the Average Pooling at the same time. The main difference is that we concatenate the results of pooling and add a convolutional layer after it for extracting the spatial layer. SA_map emerges as a consequence of an automated procedure that enhances salient features while concurrently diminishing the influence of less impactful attributes. As a result, SA_map impeccably aligns with the individual features present within the FF_all, ensuring a one-to-one representation of each feature. Thus, the dimensions of the SA_map harmoniously mirror those of FF_all, encompassing a spatial span of $(3R) \times 1$

$$\begin{equation} {SA\_map} = Concat\left(Maxpool\left({FF\_all}\right), Avgpool\left({FF\_all}\right)\right) \end{equation} \tag{ 18 }$$

$$\begin{equation} {FF\_SA} = {FF\_all} \otimes {SA\_map}. \end{equation} \tag{ 19 }$$

Channel Attention extracts the features more from the aspect of the channel, while Spatial Attention extracts the features more from the aspect of spatial relationship. Even though for 1D case, their difference may not be as distinguishing as the case of 2D. However, it is still useful to consider these two aspects for comprising more comprehensive information

$$\begin{equation} \textit{FF}_\textit{Merge} = \textit{FF}_\textit{CA} + \textit{FF}_\textit{SA}. \end{equation} \tag{ 20 }$$

After extracting features from the three streams, their fusion is carried out using equation (20). The resulting fused feature is subsequently utilized in the Output Module for further processing and analysis.

The Output Module is where the fused feature FF_Merge, obtained from the Feature Fusion Module as described in section 3.5, is processed using a Dense layer whose size is determined by the number of endmembers present in the dataset being used. This step is critical, as the size of the Dense layer must match the number of endmembers in order to facilitate accurate identification and classification.

In previous sections, we discussed the scalability of our ATSFCNN. Specifically, the neural networks used for extracting 1D, 2D, and 3D features can be replaced with other models, and the methods in the Feature Fusion section will still function. However, we must provide further details regarding the structures of the neural networks used in this article.

3.7.1. 1D-CNN

In our conducted experiments, the architecture of the 1D-CNN comprises a total of four hidden layers, as visually delineated in figure 9. The selection of hyperparameters governing this structure entails a multitude of potential permutations. To foster a stringent basis for performance assessment and result in comparability, we opted to standardize the architectural configuration across the 1D-CNN, 2D-CNN, and 3D-CNN models.

Zoom In Zoom Out Reset image size

Commencing with the foremost convolutional layer, we introduced 64 filters, each characterized by a kernel size of 3×1. The input to this layer is designated as T₁₁. Subsequently, the second convolutional layer integrates 32 filters of identical kernel dimensions, with T₁₂ as its designated input. A subsequent Flatten operation is executed to transform the output into a 1D array, with T₁₃ denoting the resultant array. Consequent to this, a Fully-Connected layer interfaces with the third layer, incorporating a ReLU activation function to exclude non-positive outcomes. This layer assumes dimensions of R×1.

Pertaining to the original hyperspectral dataset with dimensions expressed as M×N×P, as illustrated in figure 4, the dataset's inherent three-dimensional structure is reconfigured into a sequence of 1D arrays. Each individual pixel within the two-dimensional (M×N) image is projected along the depth dimension, yielding M×N distinct 1D arrays. Each of these arrays is characterized by a dimensionality of P×1, which, in turn, serves as the input for the 1D stream. This configuration thus imparts T₁₁ with dimensions of P×1. Given that the kernel dimensions are established at 3×1, in accordance with equation (1), T₁₂ assumes dimensions of (P-2)×1. Similarly, T₁₃ adopts dimensions of (P-4)×1. Within the confines of the Flatten layer, a compression process is initiated, whereby the entirety of the 32 filters within the second convolutional layer are compacted. The outcome of this operation, quantified as (P-4)×32, translates to (32P-128) in numerical terms. Consequently, the resultant output dimensions of the Flatten layer can be succinctly expressed as (32P-128)×1.

The architecture of our 1D-CNN model is primarily inspired by the works of [34, 59], and [60]. However, there is one fundamental difference between our model's architecture and theirs. In their respective studies, they have included a Maxpooling layer. [59, 60] added the Maxpooling layer after all the Convolutional layers, while [34] inserted the Maxpooling layer between the first and second Convolutional layer.

In our study, we decided not to include the Maxpooling layer due to the following consideration. In our 3D-CNN models, the hyperspectral dataset is divided into a series of small cubes measuring 5×5×P, and the kernel is 3×3×3. Without MaxPooling, the 1D-CNN, 2D-CNN, and 3D-CNN can have very similar structures, allowing for better comparison of their individual results. Adding the Maxpooling stage would require much larger small cubes as inputs for the 2D and 3D models. However, this step is not necessary for our study because our primary contribution is proposing a novel method for fusing features from different streams of HSIs for classification purposes.

3.7.2. 2D-CNN

In our conducted experiments, the architecture of the 2D-CNN encompasses a total of four hidden layers, as elucidated in figure 10. Commencing with the foremost convolutional layer, we introduced 64 filters, each characterized by a kernel size of 3×3. The input to this layer is designated as T₂₁. Subsequently, the second convolutional layer integrates 32 filters of identical kernel dimensions, with T₂₂ as its designated input. Subsequently, a Flatten operation is executed, resulting in the transformation of the output into a 1D array, with the resultant array designated as T₂₃. Consequent to this, a Fully-Connected layer interfaces with the third layer, incorporating a ReLU activation function to exclude non-positive outcomes. This layer assumes dimensions of R×1.

Zoom In Zoom Out Reset image size

With regards to the original hyperspectral dataset characterized by dimensions M×N×P, as portrayed in figure 4, via PCA, the inherent three-dimensional structure of the dataset undergoes a transformation into a datacube configuration of M×N×L.

In the field of 2D-CNN, it is practical to choose small image patches and use them to represent the class of the pixel in the center of the patches, as discussed by [35]. Specifically, to classify a pixel $p_{x,y}$ at location (x, y) on the image plane, we use a square patch of size s×s centered at pixel $p_{x,y}$. To avoid the issue of void values for image patches near the borders, we need to add padding. In this study, we choose to add zero values in the padding parts near the borders of the dataset.

In light of the M×N image being methodically encapsulated within the s×s image patch paradigm as elucidated earlier, it ensues that the input dimensionality pertinent to the 2DCNN-represented by T₂₁-precisely equates to s×s. An intricate facet to underscore is the existence of M×N such minuscule patches that embody the comprehensive representation of the entire dataset. Harmonizing this with the kernel dimensions, as per the precepts of equation (1), culminates in the derivation s-3+1 = (s-2). It follows that T₂₂ attains dimensions denoted as (s-2)×(s-2), with s signifying the size of the image patch. Similarly, T₂₃ assumes dimensions of (s-4)×(s-4), adhering to a parallel rationale.

Within the confines of the Flatten layer, a compression process is initiated, whereby the entirety of the 32 filters within the second convolutional layer are compacted. The outcome of this operation, quantified as (s-4)×(s-4)×32, translates to ($32s^2-256s+512$) in numerical terms. Consequently, the resultant output dimensions of the Flatten layer can be succinctly expressed as ($32s^2-256s+512$)×1.

3.7.3. 3D-CNN

In our conducted experiments, the architecture of the 2D-CNN encompasses a total of four hidden layers, as elucidated in figure 11. Commencing with the foremost convolutional layer, we introduced 64 filters, each characterized by a kernel size of 3×3×3. The input to this layer is designated as T₃₁. Subsequently, the second convolutional layer integrates 32 filters of identical kernel dimensions, with T₃₂ as its designated input. Subsequently, a Flatten operation is executed, resulting in the transformation of the output into a 1D array, with the resultant array designated as T₃₃. Consequent to this, a Fully-Connected layer interfaces with the third layer, incorporating a ReLU activation function to exclude non-positive outcomes. This layer assumes dimensions of R×1. Similar to the 2D case, T₃₁ has the size of s×s×s, T₃₂ has the size of (s-2)×(s-2)×(s-2), T₃₃ has the size of (s-4)×(s-4)×(s-4).

Zoom In Zoom Out Reset image size

The study utilizes thr ee hyperspectral datasets: Samson, Urban, and PaviaU. All these three datasets belong to the remote sensing field.

Samson contains 95×95 pixels with 156 channels, covering a wavelength range of 401 to 889 nm with a resolution of 3.13 nm. It includes three different endmembers: Soil, Tree, and Water. Urban comprises $307\times 307$ pixels with 162 channels, with a resolution of 10 nm. It consists of six endmembers: Asphalt, Grass, Tree, Roof, Metal, and Dirt. PaviaU has $610 \times 340$ pixels with 103 channels. Among all these pixels, only 42 776 pixels belong to the 9 endmembers: Asphalt, Meadows, Gravel, Trees, Painted metal sheets, Bare Soil, Bitumen, Self-Blocking Bricks, and Shadows. These three datasets are all labeled, with ground-truth classification information provided for each pixel.

In these three datasets, 70%, 5%, and 25% are randomly assigned as Training dataset, Validation Dataset, Testing Dataset. The Training dataset assumes a pivotal role as the part for model training. Subsequent to this, the Validation dataset operates as the crucible for refining model hyperparameters. Ultimately, the Testing dataset is enlisted to conduct a comprehensive evaluation of model performance.

The specific dataset partitions are demonstrated in tables 1–3.

The experiments were conducted using the Anaconda 22.9.0 environment and the Python language, along with the TensorFlow library toolkit. The results were generated on a PC located in the Imaging Department of C2RMF, featuring an Intel(R) Xeon(R) W-2275 CPU 3.30 GHz and an Nvidia GeForce RTX 3080 graphics card, with 128G of memory.

During the validation procedure, we need to determine the most suitable hyperparameters for our model. The two most predominant factors are the learning rate, the value of reduced dimension after PCA. Within this context, for each hyperparameter configuration, models exhibiting maximal classification performance within the validation subset shall be selected. Subsequently, during the assessment conducted on the testing dataset, the chosen model characterized by this specific constellation of hyperparameters will be engaged. During training, the batch size was set to 32, and the Binary Cross Entropy loss function was employed. All experiments utilized the Adam Optimizer, a choice underpinned by Adam's favorable attributes in terms of operational simplicity, invariance of the magnitudes of the parameter [61], excellent performance [62]. The number of epochs was set to 10, and the results of each model were repeated 10 times. The image patch size was fixed at 5 × 5 for all experiments. The experiments were conducted using the same hyperparameters, with a training/validation/testing ratio of 0.70:0.05:0.25.

First, we delve into the calibration of the learning rate, a pivotal facet of utmost significance. The determination of an apt learning rate value assumes paramount importance, as its judicious selection is instrumental in expediting convergence during training. An excessively low value might protract the convergence process significantly, whereas an overly high value can trigger deleterious divergence in the loss function dynamics [63]. In the quest to ascertain the most fitting learning rate, a comprehensive array of options is explored: {0.01, 0.003, 0.001, 0.0003, 0.0001, 0.00 003}. Across these three datasets, the experiments are uniformly executed over a span of 10 epochs. Drawing insights from the classification outcomes vis-a-vis the validation dataset, we identify the optimal learning rate values. For the Samson dataset, the learning rate of 0.001 emerges as the most suitable. Conversely, the Urban and PaviaU datasets manifest an optimal learning rate value of 0.0003.

Secondly, within the framework of our proposed methodology, we introduce an initial application of PCA to effect dimensional reduction in the context of the 2D channel. In this context, the identification of the most appropriate reduced dimension value constitutes a salient endeavor. As depicted in table 4, it becomes evident that for the Samson dataset, optimal performance materializes at a reduced dimension of 15, whereas the Urban dataset attains peak performance at a reduced dimension of 20. Conversely, the PaviaU dataset yields superior results with a reduced dimension of 10. Notably, the tabulated results in table 4 underscore a consistent trend wherein models devoid of PCA preprocessing consistently underperform in comparison to their PCA-incorporated counterparts. This trend resonates with the compelling necessity of integrating PCA as an indispensable facet of the data preprocessing pipeline.

The metrics used to assess the results of the classification in this section are OA, AA, and κ (kappa coefficient).

The OA is calculated as follows:

$$\begin{equation} {OA} = \frac{\sum_{i = 1}^{N} A_{i}}{N}\,. \end{equation} \tag{ 21 }$$

Here, A_i represents the number of pixels that belong to the ith class and were correctly classified as the ith class by the model. The variable N represents the total number of pixels in the testing dataset.

The AA is calculated as follows:

$$\begin{equation} \mathrm{AA} = \frac{1}{N}\sum_{i = 1}^{N}\frac{A_{i}}{N_{i}}. \end{equation} \tag{ 22 }$$

Here, A_i represents the number of pixels in the ith class that were correctly classified, and N_i represents the total number of pixels in the ith class. The variable N denotes the total number of pixels in the testing dataset.

κ (kappa coefficient) is a widely used statistical metric, it has been utilized in various fields including medical research [64], classification of thematic map [65], and voice recognition [66]. It is a powerful statistical tool because it is computed by weighting the measured accuracies, which represents the robust measure of the degree of agreement.

According to [67], the κ can be briefly expressed as follows:

$$\begin{equation} \kappa = \frac{\mathrm{Pr}\left(a\right) - \mathrm{Pr}\left(e\right)}{1 - \mathrm{Pr}\left(e\right)}\ \end{equation} \tag{ 23 }$$

where Pr(a) represents the actual observed agreement, and Pr(e) represents chance agreement.

In this part, we evaluated the performance of various CNN models for HSI classification on three datasets. The models evaluated included 1D-CNN, 2D-CNN, 2D-CNN + PCA, 3D-CNN, 3D-CNN + PCA, and the proposed ATSFCNN, all implemented in the Python language and TensorFlow library. We used OA, AA, and κ (kappa coefficient) as the evaluation metrics.

The results on the Samson dataset shown in table 5, indicate that the proposed ATSFCNN model outperformed other individual CNN models that use only one stream of the hyperspectral dataset. The OA, AA, and κ results showed that ATSFCNN had the highest OA, AA, and kappa coefficient of classification, as well as the smallest standard deviation compared to other methods. This indicates that the results of ATSFCNN are more stable and consistent, with better prediction ability even for individual classes.

In table 6, on the Urban dataset, ATSFCNN achieved the highest OA and the smallest standard deviation for the OA metric. Similarly, ATSFCNN outperformed other models in terms of AA and κ. The results in table 7 of the PaviaU dataset also showed that ATSFCNN's performances in OA, AA, and κ were much better than other models, with the highest average and the smallest standard deviation.

The illustrations denoted as figures 12–14 revealed the ground-truth results and the corresponding classification maps derived from various methodologies, including 1D-CNN, 2D-CNN, 2D-CNN + PCA, 3D-CNN, 3D-CNN + PCA, and ATSFCNN. The visual representations presented in these figures harmonize cohesively with the numerical values meticulously documented in tables 5–7. Remarkably, even within the context of a testing dataset that encompasses merely a quarter of the entire dataset, the conspicuous excellence of the proposed ATSFCNN model remains distinctly evident, as vividly depicted in table 7.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Our study's findings revealed that ATSFCNN, which fuses features from 1D-CNN, 2D-CNN, and 3D-CNN, achieved more accurate and stable predictions than models that use only one stream or part of the hyperspectral dataset. The superior OA of ATSFCNN can be attributed to its architecture, which considers spectral, spatial, and hidden information among adjacent channels in the hyperspectral data. In contrast, individual models such as 1D-CNN, 2D-CNN, and 3D-CNN only capture partial aspects of the data, which can lead to less accurate predictions.

These results demonstrate that, by fusing features from different streams using the attention mechanism, ATSFCNN incorporates information from all relevant aspects, resulting in more stable and accurate predictions with the smallest standard deviation. These results suggest the importance of using ATSFCNN in HSI classification.

Real-world applications often involve situations where labels are only available for a small proportion of the entire dataset. Therefore, it is crucial to examine the performance of our proposed model, ATSFCNN, under such conditions. To evaluate the robustness of the model, we conducted experiments with different proportions of training, validation, and testing data: 70/5/20, 45/5/50, 20/5/75, 10/5/85, 5/5/90. The notation T1/V/T2 indicates the percentage of the entire dataset used for the training, validation, and testing sets, respectively.

The results presented in table 8 demonstrate that the OA of all methods decreases slightly as the proportion of the testing set increases, which is expected due to the larger proportion of the training set typically leading to better model training. However, ATSFCNN consistently outperforms other methods across various proportions of training/validation/testing, highlighting its potential for real-world applications with limited labeled data. Notably, even when the training dataset accounts for only 5% of the entire dataset, ATSFCNN achieves a high classification accuracy of 96.65%. Furthermore, the relative performance of different methods remains consistent across various proportions of training/validation/testing, with ATSFCNN consistently achieving the highest OA and smallest standard deviation. This consistency suggests that ATSFCNN is a robust model for hyperspectral imaging classification, even in situations where labeled data is scarce.

Similarly, table 9 presents results of ATSFCNN's performance in predicting the Urban hyperspectral dataset across different proportions of training, validation, and testing data, showing that as the proportion of the training set decreases from 70/5/25 to 5/5/90, ATSFCNN consistently achieves the best OA among all models, with standard deviations that are almost consistently the smallest. Finally, table 10 confirms that ATSFCNN outperforms all other models in predicting Pavia University hyperspectral datasets, irrespective of the proportion of training, validation, and testing data used. These results emphasize the robustness of ATSFCNN in predicting hyperspectral datasets with limited labeled data.

From tables 8–10, two notable trends can be observed regarding the performance of different models on different datasets.

Firstly, the results of 2D-CNN and 3D-CNN fall short of 1D-CNN in the Samson and Urban datasets, while in the PaviaU dataset, this trend is reversed. This can be explained by the fact that 1D-CNN only extracts spectral information, requiring fewer hyperparameters to train the model and being better suited for situations where the number of endmembers is smaller. The Samson dataset has three different endmembers and the Urban dataset has six different endmembers, whereas the number of endmembers in the PaviaU dataset is much larger. Therefore, the simplicity of the 1D-CNN model may provide an advantage over the more complex 2D-CNN and 3D-CNN models in datasets with fewer endmembers.

Secondly, our proposed ATSFCNN consistently outperforms the other models on all three datasets. This can be attributed to the fact that the ATSFCNN incorporates and fuses features extracted from all possible streams, giving it an advantage over other models that only utilize information from one aspect. Additionally, the classification accuracy of ATSFCNN can still be improved by utilizing more complex CNN models with more hidden layers and various scales. However, we emphasize that the main contribution of our research lies in the development of a novel approach for integrating features from all relevant streams, considering all pertinent information from hyperspectral data, and fusing the features for prediction. The specific architectures used for the 1D, 2D, and 3D streams can be substituted with other CNN models, demonstrating the scalability of our proposed ATSFCNN approach.

Figure 15–17 serve as illustrative depictions of the classification maps, which align cohesively with the outcomes as delineated in tables 8–10. Simultaneously, abundance maps corresponding to the authentic ground truths are provided for reference. Notably, the classification maps resulting from the proposed ATSFCNN exhibit markedly reduced levels of noise in various regions, in stark contrast to the outcomes generated by methodologies exclusively reliant on singular streams (1D-CNN, 2D-CNN, 2D-CNN + PCA, 3D-CNN, 3D-CNN + PCA). A specific case in point, as illustrated in figure 17, distinctly underscores the superior classification precision of ATSFCNN in distinguishing between Bare Soil and Self-Blocking Bricks, outperforming all alternative methodologies.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size