Unifying Dual-Attention and Siamese Transformer Network for Full-Reference Image Quality Assessment

2.1 Traditional IQA Methods

In the previous few decades, a lot of research has been done on FR-IQA [2]. The aim of the FR-IQA is to predict the quality of a target image with full access to its reference image. Many FR-IQA methods follow a two-stage framework, which firstly calculates the distance between the distorted and reference image features and then translates the distance to the quality ratings. The most commonly used FR-IQA methods are MSE and PSNR [46], which have been successfully used in many visual applications due to their simplicity and easy use [47]. However, the two methods measure the differences between images by pixel-level differences, which differ from the perceptual consistency of the HVS [15].

The HVS can extract structural information from the scene in a highly adaptive manner. To conduct IQA from structural similarity, Wang et al. [35] proposed a useful technique called SSIM to evaluate quality of the distorted image in terms of brightness, structure, and contrast. Then, an improved method called Multi-Scale SSIM (MS-SSIM) [8] was proposed by extending SSIM to multiple scales. Subsequently, many FR-IQA methods gradually extract the structural information of images in the spatial domain, such as PSIM [16] and GMSD [21]. They both extract gradient features in the spatial domain to describe structural changes. However, the FR-IQA methods designed from the perspective of image structure are inefficient in predicting the quality of specific distortion types, such as blurring and compression. Based on these observations, researchers have started to investigate IQA methods based on low-level features, such as color, edge, and texture. For example, Zhang et al. [23] designed an efficient method called Feature SIMilarity (FSIM) to understand image scenes by some low-level features. In another work, Zhang et al. [22] proposed a visual evaluation model to calculate Visual Saliency-induced Index (VSI). Subsequently, some other useful IQA methods are proposed, such as Most Apparent Distortion (MAD) [18] and Normalized Laplacian PyramiD (NLPD) [17].

Furthermore, some IQA methods try to extract features from transform domain because image quality degradation can be also shown in the transform domain. For example, Sheikh et al. investigated the HVS with the Information Fidelity Criterion (IFC) index [20] and proposed the Visual Information Fidelity (VIF) [19] method for quantifying the information errors between the distorted image and the reference image.

2.2 CNN-Based IQA Methods

There have been some attempts to address FR-IQA tasks with CNN techniques due to the success of CNN in many vision applications. For example, Kang et al. [24] were the first to apply CNN to the IQA task without the use of any existing features. Similar to other computer vision tasks, this method combines feature learning and regression as complete optimization schemes. Specifically, several patches are cropped out of the image, and each patch is given a label corresponding to the visual quality score of the image for training. Later, visual sensitivity information of images is combined with CNN to predict image quality scores. In another work, Kim and Lee [27] exploited CNN to design a new FR-IQA method that learns human visual sensitivity to predict subjective scores by feeding distorted images and their objective error map into the network.

Recently, Zhang et al. [32] designed a new FR-IQA method called LPIPS (Learned Perceptual Image Patch Similarity). The trained deep features of LPIPS optimized by the Euclidean distance between the reference image and the distorted image are effective in FR-IQA. In another work, Prashnani et al. [31] presented a new FR-IQA framework called PieAPP, which is based on pairwise learning. The results of PieAPP show that pairwise learning helps to improve image quality prediction. Subsequently, Ding et al. [30] introduced a novel FR-IQA method that unifies structure and texture similarity. This method shows good performance in predicting visual qualities of textural images and natural images.

2.3 Transformer-Based IQA Methods

The Vision Transformer (ViT) [38] is the famous work about the Transformer in computer vision. This work has demonstrated that the Transformer may “fully” replace the convolutions regularly used in deep neural networks. In general, a pure Transformer architecture includes three important components, i.e., Multi-Head Attention (MHA), Multi-Layer Perceptron (MLP), and Layer Normalization (LN). The ViT exploits a pure Transformer to conduct image classification via taking an image as a sequence of patches. The ViT can learn with minimum inductive bias. Compared with CNNs, the relational representation learned by the Transformer is more general and robust than the local patterns of convolutional modules [48].

The ViT is an efficient technique based on self-attention. Since the self-attention layer collects global information from the whole input sequence, it can measure “perceptual loss” between the distorted image and the reference image when predicting image quality. This design might be helpful in the FR-IQA task because recent studies [42, 43, 44, 45] have proved the usefulness of deep features in the IQA task.

Inspired by ViT, You et al. [45] proposed the first Transformer-based IQA model named TRIQ. The TRIQ uses CNN to extract feature maps which are sent to a shallow Transformer encoder to solve blind IQA tasks. Taking the advantage of the inductive capability of CNN architecture and Transformer encoder for aggregated representation of attention mechanism, the TRIQ achieves outstanding performance. Subsequently, Cheon et al. [42] designed a new Transformer model named IQT for FR-IQA. In the IQT, the encoder-decoder architecture is employed to measure similarity. In another work, Ke et al. [44] used the Transformer to build a new method called MUSIQ (MUlti-Scale Image Quality). The MUSIQ can handle images with different resolutions by mapping input image to a multi-scale representation.

3.1 Feature Extraction Module

As depicted in Figure 1, the front part of the feature extraction module is a two-branch pre-trained ViT, which acts as a feature extraction backbone that mainly focuses on extracting low-level features of color, texture, shape information, and high-level features of semantic information. We crop out M patches for each input reference and distorted images separately. Let \(p_{i}^{ref}\) and \(p_{i}^{dist}\) be the \({i}\)th patches of the input reference and distorted images, where \(i\in \lbrace 1,2,\ldots , M \rbrace\). The \({i}\)th input patch \(p_{i}^{ref}\) of the reference image and the \({i}\)th input patch \(p_{i}^{dist}\) of the distorted image are input to the feature extractor to obtain the corresponding feature representations which are extracted from four intermediate layers of ViT, i.e., \(layer \in \lbrace 3,5,8,11\rbrace\). For these feature representations, each \(f_{layer} \in \mathbb {R}^{ c\times w\times h}\) has the same shape. Finally, we concatenate these four feature maps by channel dimension to get a total feature map \(f_{ref} \in \mathbb {R}^{C\times w\times h }\) and \(f_{dist} \in \mathbb {R}^{C\times w\times h }\), where \(C=c\times 4\).

To construct perceptual differences between the reference and distorted images, a simple subtraction operation between the feature maps \(f_{ref}\) and \(f_{dist}\) is conducted to get the difference map \(f_{diff}\) as follows: (1) \(\begin{align} f_{diff} = f_{ref} - f_{dist}. \end{align}\)

To illustrate the effectiveness of the difference map for FR-IQA, some visual examples are presented in Figure 2. In this figure, the first row shows a reference image and its distorted images with different distortions introduced by additive white Gaussian noise. The second row shows their corresponding feature maps, and the third row shows the difference maps. It can be seen that as more distortions are introduced to the image, there are significant changes in the difference map. This means that the difference map can be used for FR-IQA.

Fig. 2.

View Figure

After that, these feature maps are sent to a 1 × 1 convolutional layer to aggregate the information of each channel and downsampling.

3.2 Channel Attention Module

The channel attention module is made up of a Squeeze-and-Excite network. Firstly, the feature maps generated by the feature extraction module are squeezed to acquire global features at the channel level. The global features are then subjected to an excitation operation to learn the channel relationship and acquire the channel weights. Finally, the channel weights are multiplied by the input feature maps to generate the final features.

In summary, the channel attention module performs the channel rating action. This attention mechanism enables the network to prioritize relevant channel features while suppressing unimportant channel features. Note that the maps \(f_{ref}\), \(f_{dist}\), and \(f_{diff}\) are separately fed to the channel attention module. Take \(f_{diff}\) as an example for illustrating the formal descriptions of the whole process as follows: (2) \(\begin{align} z_k &= F_{sq}(f_{diff})=\frac{1}{h\times w} \sum _{i=1}^{h} \sum _{j=1}^{w} f_{diff}(k,i,j), (k=1, 2, \dots , C), \end{align}\) (3) \(\begin{align} z = [z_1, z_2, \dots , z_C], \end{align}\) (4) \(\begin{align} s = F_{ex}(z)=Sigmod(FC(Relu(FC(z)))), \end{align}\) (5) \(\begin{align} \tilde{f_{diff}} = (f_{diff}\cdot s)+f_{diff.} \end{align}\)

3.3 Feature Embedding Module

To feed these three feature maps into the spatial attention module for training, we need to perform feature embedding operations on them.

Since our siamese transformer network utilizes constant latent vectors sized D in all layers, the feature maps are reshaped by spatial dimension to convert to N without destroying the pixel information. Thus, the intermediate embedding features can be determined by the below equation, where \(N=h \times w\). (6) \( \begin{equation} z^{diff} = \left[ \tilde{f_{diff}^{1}}; \tilde{f_{diff}^{2}}; \cdots ; \tilde{f_{diff}^{N}} \right] \in \mathbb {R}^{N \times D }. \end{equation} \)

Similar to typical ViT [38] models, we prepare a learnable embedding called “quality token” \(f_{qua}\) to the sequence of \(z^{diff}\), whose state at the output of the network is viewed as the predicted quality. Here the function of “quality token” is similar to the “classification token” in the classical ViT. In addition, the position embedding \(f_{pos}^{diff} \in \mathbb {R}^{\left(N+1 \right)\times D }\) is also added to \(z^{diff}\) for maintaining positional information. Consequently, an embedding feature with positional information is available as follows: (7) \(\begin{equation} z_{0}^{diff} = \left[ f_{qua }; \tilde{f_{diff}^{1}}; \cdots ; \tilde{f_{diff}^{N}} \right] + f_{pos}^{diff}. \end{equation}\)

Similarly, the embedding features of \(f_{ref}\) and \(f_{dist}\) can be determined by the Equations (8) and (9), respectively. (8) \(\begin{align} &z_{0}^{ref} =\left[ f_{qua}; \tilde{f_{ref}^{1}};\cdots ; \tilde{f_{ref}^{N}} \right] + f_{pos}^{ref}, \end{align}\) (9) \(\begin{align} &z_{0}^{dist} =\left[ f_{qua}; \tilde{f_{dist}^{1}};\cdots ; \tilde{f_{dist}^{N}} \right] + f_{pos}^{dist}. \end{align}\)

A visual example of feature embedding is provided in Figure 3 to facilitate comprehension. The feature embedding module’s inputs are feature maps of size \(28 \times 28 \times 256\) output from the channel attention module. We perform a reshape operation on the feature map by the spatial dimension that produces the intermediate embedding features sized \(784 \times 256\). Finally, the final embedding features sized \(785 \times 256\) are obtained by embedding position and quality. Next, these embedded features are sent to a spatial attention module for training.

Fig. 3.

View Figure

3.4 Spatial Attention Module

The spatial attention module consists of a novel Siamese Transformer network and a new feature fusion block. It has three inputs, i.e., embedding features of the distorted image, embedding features of the reference image, and embedding features of the perceptual differences between distorted and reference images. Unlike conventional FR-IQA methods that only perform distance calculation between the reference and distorted images, the spatial attention module employs a Siamese Transformer network to learn information from perceptual differences between the distorted image features and the reference image features. This spatial attention module exploits the latent distance to calculate the perceptual differences between the reference and distorted images for quality prediction.

As shown in Figure 3, we connect each pixel in the feature map by channel dimension and use the concatenated vector as the token input to the siamese transformer network. As a result, the MHA employed in the transformer encoder and the transformer decoder will act as special spatial attention. Its role is to assign various weights to different pixels and thus allow the network to focus on significant features while suppressing irrelevant ones. This strategy can better learn perceptual image features.

In summary, the calculations of the spatial attention module can be formulated as Equations (10)–(23), where L denotes the number of the module layers. (10) \(\begin{align} &q_{l}= k_{l}= v_{l}=z_{l-1}^{diff}, \end{align}\) (11) \(\begin{align} &z_{l}^{\ast } = \mathrm{LN}\left(\mathrm{MSA}\left(q_{l},k_{l},v_{l} \right) + z_{l-1}^{diff} \right), \end{align}\) (12) \(\begin{align} &z_{l}^{diff} = \mathrm{LN}\left(\mathrm{MLP}\left(z_{l}^{\ast } \right) + z_{l}^{\ast } \right) ,l=1, 2, \ldots , L , \end{align}\) (13) \(\begin{align} &z_{L}^{diff}=\left[ z_{E_{qua}},z_{E_{0}},z_{E_{1}},\ldots ,z_{E_{N}} \right], \end{align}\) where MSA(\(\cdot\)) is the operation of Multi-head Self-Attention (MSA).

The inputs of one transformer decoder are \(z_{0}^{dist}\), and \(z_{L}^{diff}\) which is the output of the transformer encoder. Similarly, the inputs of the other transformer decoder are \(z_{0}^{ref}\) and \(z_{L}^{diff}\). The calculations of one single branch of the process of spatial attention module can be formulated as Equations (14)–(18). (14) \(\begin{align} &q_{l}= k_{l}= v_{l}=z_{l-1}^{ref}, \end{align}\) (15) \(\begin{align} &z_{l}^{\ast } = \mathrm{LN}\left(\mathrm{MSA}\left(q_{l},k_{l},v_{l} \right) + z_{l-1}^{ref} \right), \end{align}\) (16) \(\begin{align} &\hat{q_{l} } = z_{l}^{\ast },\hat{k_{l} }=\hat{v_{l} }=z_{L}^{diff}, \end{align}\) (17) \(\begin{align} &z_{l}^{\ast \ast }=\mathrm{LN}\left(\mathrm{MCA}\left(\hat{q_{l} }, \hat{k_{l} },\hat{v_{l} }\right) + z_{l}^{\ast } \right), \end{align}\) (18) \(\begin{align} &z_{l}^{ref }=\mathrm{LN}\left(\mathrm{MLP}\left(z_{l}^{\ast \ast } \right) +z_{l}^{\ast \ast } \right) ,l=1, 2, \ldots , L, \end{align}\) where MCA(\(\cdot\)) is the operation of Multi-head Cross-Attention (MCA).

Note that the outputs of the siamese transformer network are \(z_{L}^{ref\sim diff}\) and \(z_{L}^{dist\sim diff}\), which are defined by Equations (19)–(20). (19) \(\begin{align} &z_{L}^{ref\sim diff} = [ z_{R_{qua}},z_{R_{0}},z_{R_{1}},\ldots ,z_{R_{N}}], \end{align}\) (20) \(\begin{align} &z_{L}^{dist\sim diff} = [ z_{D_{qua}},z_{D_{0}},z_{D_{1}},\ldots ,z_{D_{N}}]. \end{align}\)

Obviously, the output features \(z_{L}^{ref\sim diff}\) and \(z_{L}^{dist\sim diff}\) are complementary. Therefore, it is necessary to aggregate the features of both branches. To do so, an additional feature fusion block is designed. Figure 4 depicts the diagram of our proposed feature fusion block. The proposed feature fusion block can efficiently fuse information from both branches of the siamese transformer network. It makes the proposed UniDASTN more effective in predicting image quality.

Fig. 4.

View Figure

Specifically, a sequence of tokens called \(\tilde{z_{i}}\) is constructed by concatenating all tokens in \(z_{L}^{ref\sim diff}\) and \(z_{L}^{dist\sim diff}\) except for the “quality token”. And then a new “quality token” \(\tilde{z_{qua}}\) is determined by the summation of \(z_{R_{qua}}\) and \(z_{D_{qua}}\). Consequently, a new sequence of tokens called \(\tilde{z}\) can be obtained by using \(\tilde{z_{qua}}\) and \(\tilde{z_{i}}\). Formal descriptions of these calculations are defined in Equations (21)–(23). Next, \(\tilde{z}\) is delivered to a transformer encoder for learning information from every token to help to improve the prediction performance. (21) \(\begin{align} &\tilde{z_{i} } = \mathrm{concatenate}\left(z_{R_{0} },\ldots ,z_{R_{N} },z_{D_{0} },\ldots ,z_{D_{N} } \right) , \end{align}\) (22) \(\begin{align} &\tilde{z_{qua}} = z_{R_{qua} } + z_{D_{qua} } , \end{align}\) (23) \(\begin{align} &\tilde{z} = \left[ \tilde{z_{qua}},\tilde{z_{i} } \right]. \end{align}\)

3.5 Prediction Head

The quality score is finally determined in the prediction head. The “quality token” \(\tilde{z_{qua}} \in \mathbb {R}^{1\times D }\) output by the spatial attention module contains the quality information. It is input to the prediction head. The prediction head is achieved by MLP which is composed of two fully connected layers. In the MLP, the first layer is activated by the ReLU, and the second layer only outputs one channel to predict a score, i.e., the final predicted score.

3.6 Loss Function

The novel loss function is designed to evaluate the predicted image scores and optimize the proposed model. Note that the MSE loss is sensitive to outliers. If just the MSE loss is utilized as the loss function for training, the training process is subject to substantial gradient changes caused by outliers, making loss convergence unstable.

To obtain more accurate image prediction scores, we design a novel joint loss function to optimize the proposed UniDASTN for stable training. Our loss function consists of three components, including the MSE loss \(L_{MSE}\), the bidirectional KL divergence loss \(L_{BKL}\), and the rank loss \(L_{RANK}\). The MSE loss \(L_{MSE}\) is defined as (24) \(\begin{equation} L_{MSE} = \sum _{i=1}^{K} MSE(p_i,\hat{q_i}) , \end{equation}\) where \(p_i\) and \(\hat{q_i}\) are the predicted score and the MOS of the ith image, \(MSE(\cdot ,\cdot)\) is the standard MSE, and K is the image number.

We select KL divergence as one component of the loss function because we want to use KL divergence to calculate the distance between the two distributions of prediction scores and MOS values in space. However, since KL divergence assesses the loss of information from one distribution compared to the other, it does not measure the proximity of two distributions in space. As a result, we improve the original KL divergence and create a bidirectional KL divergence for the IQA task to measure the differences between predicted scores and MOS values.

The bidirectional KL divergence loss \(L_{BKL}\) is defined as (25) \(\begin{equation} \begin{aligned}L_{BKL} &= KL\left(Q||\frac{1}{2}(Q+\hat{Q})\right)+KL\left(\hat{Q}||\frac{1}{2}(Q+\hat{Q})\right)\\ &=\sum _{i=1}^{K}Q\times log\frac{Q}{\frac{1}{2}(Q+\hat{Q})} +\sum _{i=1}^{K}\hat{Q}\times log\frac{Q}{\frac{1}{2}(Q+\hat{Q})} \end{aligned} \end{equation}\) where \(\hat{Q}\) and Q are the outputs of softmax regression defined as follows: (26) \(\begin{equation} \hat{Q} = \frac{\hat{e^{q_i}}}{\sum _{i=1}^{K}\hat{e^{q_i}} } , \end{equation}\) (27) \(\begin{equation} Q = \frac{e^{p_i} }{\sum _{i=1}^{K}e^{p_i} } . \end{equation}\)

Furthermore, only considering the regression ability of the network on the quality scores does not consistent with the perception of human vision. That is because, in terms of regression, the same degree of regression values is considered to achieve the same performance results. However, in terms of rank order, regression values of the same degree do not guarantee that the results predicted by the network for distorted images are of the same order as the MOS values.

Motivated by the concept of “rank loss”, we consider that the rank order of the prediction scores should be consistent with the ground truth for reflecting the rank order among the degraded images in the design of the loss function of the network. In a previous study [31], the network is trained by considering the probability of preference for each image pair with different distortions. In another study [29], the authors think that when the rank of the output scores does not agree with the rank of the ground truth scores regardless of their distortion types, the network should be penalized. Inspired by [29], we devise the following pairwise rank loss \(L_{r}(I_i,I_j)\) to determine the rank order of a pair of distorted images \(I_i\) and \(I_j\) as follows: (28) \(\begin{align} L_{r}(I_i,I_j) &= max\left(0,\frac{-(\hat{M}_i -\hat{M}_j)(M_i -M_j)}{|\hat{M}_i -\hat{M}_j |+ \varepsilon } \right)\times \mu , \end{align}\) (29) \(\begin{align} \mu &= (M_i -\hat{M}_i)^{2}+(M_j -\hat{M}_j)^{2}, \end{align}\) where \(\varepsilon\) is a small stability term and \(\mu\) is a penalty term. Note that \(L_{r}(I_i,I_j)\) is always 0 if the rank order of the predicted scores exactly matches that of MOS values. Otherwise, it is a positive number that penalizes the network. Therefore, we can obtain the total rank loss \(L_{RANK}\) by considering all pair rank losses as follows: (30) \(\begin{equation} \begin{aligned}L_{RANK} &= L_{r}(I_1,I_2) + L_{r}(I_1,I_3) + \dots + L_{r}(I_1,I_K)\\ &\quad +L_{r}(I_2,I_3) + L_{r}(I_2,I_4) + \dots + L_{r}(I_2,I_K)\\ &\quad +\dots \\ &\quad +L_{r}(I_i,I_{i+1}) + L_{r}(I_i,I_{i+2}) + \dots + L_{r}(I_i,I_K)\\ &\quad +\dots \\ &\quad +L_{r}(I_{K-1},I_K) \end{aligned} \end{equation}\)

Consequently, the total loss can be assessed as a weighted sum of these three loss items as follows: (31) \(\begin{equation} L_{total} = \alpha L_{MSE}(M,\hat{M}) + \beta L_{BKL}(Q,\hat{Q}) + \gamma L_{RANK} , \end{equation}\) where \(\alpha\), \(\beta ,\) and \(\gamma\) are the hyper-parameters for adjusting the weights.

4.1 Implementation Details of Our UniDASTN

The implementation details of the proposed UniDASTN are provided here. In the training phase, the hyper-parameters of the loss function are \(\alpha =1.0\), \(\beta =0.5,\) and \(\gamma =0.1\). Our used data augmentation strategies include horizontal flip, random rotation, and random cropping. Specifically, we do a horizontal flip with a probability of 0.5. With a probability of 0.25, we do rotations of 90, 180, or 270 degrees, as well as no rotation. Next, the images are randomly cropped to provide \(224\times 224\times 3\) image patches for the feature extraction backbone. The training is carried out with a batch size of 8 and employs an ADAM optimizer for 300 epochs using the designed joint loss function. An initial learning rate is set to 1e-4, and a weight decay of 1e-5 is used. We set the learning rate of each parameter by using a cosine annealing schedule, where \(\eta _{max}\) is set to the initial learning rate and \(\eta _{min}\) is set to 0 and the number of cosine learning rate period \(T_{cur}\) is 50. The value of \(\varepsilon\) used in the loss function is set to 1e-8.

In the testing phase, we perform test-time augmentation on the images by cropping into M overlapping patches of size \(224\times 224\times 3\) and predict the quality score by averaging the scores of all crops, where M is set to 20.

The hyper-parameters of the proposed UniDASTN are as follows. (1) The number of layers in the spatial attention module is 2, i.e., \(L=2\). (2) The head numbers of MSA and MCA are both 8. (3) The dimension of per head is 32. (4) The dimension of Transformer is 256, i.e., \(D=256\). (5) The dimension of the MLP in Transformer is 1024. (6) The dimension of the first layer in prediction head is 512, i.e., \(Dhead=512\).

4.2 Evaluation Criteria

In the following experiments, three popular IQA metrics are employed to quantify performance. These three metrics are: the Pearson Linear Correlation Coefficient (PLCC) [49], the Spearman Rank-order Correlation Coefficient (SRCC) [50], and the Kendall Rank-order Correlation Coefficient (KRCC) [51]. The PLCC is used to assess the ability of predicting subjective scores with low error. The SRCC and KRCC are used to measure the prediction monotonicity of an FR-IQA method. The ranges of PLCC, SRCC, and KRCC are all \([-1, 1]\). A positive value indicates a positive correlation between the two sets of data, and a higher value indicates better performance.

4.3 Datasets

We use five open databases to conduct extensive experiments. Specifically, the used databases include the TID2013 [52], the Categorical Subjective Image Quality (CSIQ) database [18], the LIVE IQA database [53], KADID-10k [54], and PIPAL [34]. These five databases are commonly used in IQA research. Table 1 summarizes the basic information of these databases. Traditional IQA databases are built for general IQA tasks, and they include some common types of distortions, such as JPEG compression, JPEG2000 compression, white noise injection and blurring.

4.4 Performance Comparison

To demonstrate the advantage, we conduct two types of experiments to compare our UniDASTN and some state-of-the-art FR-IQA methods. In the comparisons, conventional FR-IQA methods and deep learning-based FR-IQA methods are both selected. The first type is to validate the quality prediction performance on the same dataset. This evaluation experiment is to test the accuracy of the quality prediction of each objective IQA method. The other type is to validate each FR-IQA method across datasets. Such evaluation experiment is to verify the generalization ability of each objective FR-IQA method.

Evaluation on the same dataset. According to the suggestion [25, 29, 59], we randomly divide the dataset into training set (80% images) and test set (20% images) based on the reference image. During the network training, only the data from the training set are visible to the FR-IQA methods. We determine the weights loaded by model based on the training and use the test set to evaluate the final performance. We repeat the experiment five times with different seeds, and the results are averaged for a fair comparison.

We compare our UniDASTN method with some state-of-the-art FR-IQA methods on the traditional distorted types of datasets, i.e., LIVE, CSIQ, and TID2013. In this experiment, our UniDASTN is compared with a total of 17 FR-IQA methods, including 10 non-deep-learning-based methods and 7 deep-learning-based methods. The 10 non-deep-learning-based methods are PSNR [46], SSIM [35], MS-SSIM [8], FSIMc [23], VSI [22], MAD [18], VIF [19], NLPD [17], GMSD [21], and SCQI [55]. The seven deep-learning-based methods are DOG-SSIMc [65], DeepQA [27], DualCNN [66], WaDiQaM-FR [25], PieAPP [31], JND-SalCAR [29], and AHIQ [59].

The PLCC and SRCC results of these methods are shown in Table 2. Note that the results of the compared methods are taken from the corresponding original papers and [30]. In Table 2, the best results are in bold and the missing results are marked with “–”. It can be seen that our UniDASTN consistently outperforms all state-of-the-art FR-IQA methods in terms of PLCC and SRCC on three IQA datasets. Furthermore, Table 2 shows that our UniDASTN made significant progress in CSIQ and TID2013 when compared to other state-of-the-art FR-IQA methods. Although some existing methods have achieved high levels of visual quality score assessment for the LIVE dataset, our UniDASTN still makes some progress on the SRCC metric. These results illustrate that our UniDASTN achieves solid improvements over the previous methods and can cope well with traditional distortion types of images.

Furthermore, to validate the effectiveness of the proposed UniDASTN in predicting the quality of images generated by GAN-based image restoration methods, we also compare our UniDASTN method with some state-of-the-art FR-IQA methods on the GAN-based synthetic image dataset, i.e., the PIPAL dataset. The PIPAL dataset contains many traditional distortion types of images and some distorted images generated by GAN-based image restoration methods. In this experiment, our UniDASTN is compared with a total of 19 FR-IQA methods, including 14 non-deep-learning-based methods and 5 deep-learning-based methods. The 14 non-deep-learning-based FR-IQA methods are PSNR [46], NQM [57], UQI [69], SSIM [35], MS-SSIM [8], RFSIM [68], GSM [61], SRSIM [60], FSIM [23], VSI [22], NIQE [63], MA [62], PI [56], and Brisque [64]. The five deep-learning-based methods are LPIPS-Alex [32], LPIPS-VGG [32], DISTS [30], IQT [42], and AHIQ [59]. All deep-learning-based methods are trained on the PIPAL training set, and the images from the validation set of PIPAL are used for the test.

As can be seen from Table 3, the best results are in bold, and our UniDASTN achieves the second-best result in the SRCC performance comparison and the third-best result in the PLCC performance comparison. Although our UniDASTN shows a slightly lower performance on the PIPAL dataset, our UniDASTN still outperforms most compared methods. This experiment demonstrates that our UniDASTN is effective in assessing the quality of GAN-based synthetic images.

Evaluation on the cross-dataset. We conduct a cross-dataset experiment to evaluate the generality of our UniDASTN. In this experiment, the FR-IQA methods are trained on a single dataset and thus tested on several other databases to validate their generalization ability in IQA.

As recommended by [30, 31, 32, 42], we use the KADID-10k dataset for training the deep-learning-based FR-IQA methods, and evaluate their performances on the full set of the CSIQ, the LIVE, and the TID2013. For non-deep-learning-based FR-IQA methods, we directly assess their performance.

In this experiment, our UniDASTN is compared with a total of 14 FR-IQA methods, including 9 non-deep-learning-based methods and 5 deep-learning-based methods. The nine non-deep-learning-based FR-IQA methods are PSNR [46], SSIM [35], MS-SSIM [8], VSI [22], MAD [18], VIF [19], FSIMc [23], NLPD [17], and GMSD [21]. The five deep-learning-based methods are DISTS [30], IQT [42], PieAPP [31], LPIPS-Alex [32], and LPIPS-VGG [32].

The PLCC, SRCC, and KRCC results of these methods are shown in Table 4, where the best results are in bold. In this cross-dataset experiment, it can be seen that our UniDASTN consistently outperforms all compared FR-IQA methods in terms of PLCC, SRCC, and KRCC. In particular, for TID2013, our UniDASTN shows a significant performance improvement compared to some state-of-the-art FR-IQA methods. This experiment illustrates that our UniDASTN achieves satisfactory generalization ability.

In addition, it can be seen that the performances of some compared methods are not stable among different databases. For example, the VSI [22] achieves good results in all metrics on the TID2013 database, but its performances on the CSIQ and LIVE databases degrade significantly. Similarly, the deep-learning-based method called IQT [42] reaches good results for all metrics on the LIVE database. However, its performances on the other two databases also degrade slightly. For our UniDASTN, it can make consistent performance among the three databases. This also illustrates the effectiveness of our UniDASTN in evaluating the qualities of various distorted images.

Furthermore, Figure 5 displays scatter plots of subjective MOS values versus objective prediction scores for our UniDASTN and the 14 FR-IQA methods on the TID2013 database, as well as the fitted curves. In this figure, the x-axis is the objective prediction score generated by a specific FR-IQA method, the y-axis is the MOS value, and each point on the plot represents one image in the database. The straight line is fitted by simple linear regression. If the correlation is strong, the points in the graph are tightly clustered around the curve. If the correlation is weak, the points are dispersed around the curve. As can be seen from the Figure 5, the points of our UniDASTN are much closer to the fitted curve than those of the compared FR-IQA methods, implying a stronger consistency with the subjective scores.

Fig. 5.

View Figure

To analyze the statistical significance of our UniDASTN relative to the compared methods, we employ the left-tailed F-test [21, 53] to decide whether or not an FR-IQA method is statistically superior to another FR-IQA method. We perform a series of hypothesis tests on the residuals between the subjective quality scores and the objective quality scores obtained from each FR-IQA method after nonlinear regression. Figures 6(a)–(c) show the statistically significant results on the LIVE, CSIQ, and TID2013 databases, respectively. In this figure, the value of H = 1 for the left-tailed F-test represents that the FR-IQA method in that row is significantly superior to the method in the corresponding column with a confidence level of more than 95% at the 0.05 significance level. The value of H = 0 means that the FR-IQA method in that row is not significantly superior to the FR-IQA method in the corresponding column. The symbol “-” means that the two FR-IQA methods in the corresponding row and column are statistically indistinguishable. From the statistical significance results of the LIVE, CSIQ, and TID2013 databases, it can be seen that our UniDASTN significantly outperforms some state-of-the-art FR-IQA methods and achieves the best performances on the three databases.

Fig. 6.

View Figure

4.5 Ablation Study

To assess the effectiveness of the network components used in the UniDASTN, we perform a detailed performance analysis of networks with various network component combinations.

Effectiveness of our channel attention module. To assess the effectiveness of the proposed channel attention module, we perform an ablation study on the CSIQ and TID2013 datasets. The results are shown in Table 5, where the best results in bold, and N1–N2 represent our networks with different usage strategies of channel attention module. Specifically, the N1 network means that the channel attention module is not used, and the N2 network means that the channel attention module is used. From Table 5, we can see that the N2 network produces the best PLCC and SRCC outcomes. These experimental results demonstrate that the channel attention module plays a significant role in our UniDASTN. The use of the channel attention module can result in more accurate quality prediction results.

Effectiveness of our loss function. To assess the effectiveness of the proposed joint loss function, we perform an ablation study on the CSIQ and TID2013 datasets. The results are shown in Table 6, where the best results in bold, and M1–M5 represent our networks with different loss functions. Specifically, in the experiment, the M5 network with MSE loss, bidirectional KL divergence loss, and rank loss produces the best PLCC and SRCC outcomes. These experimental results demonstrate that both the bidirectional KL divergence loss and the rank loss contribute to the performance improvement of our UniDASTN.

In addition, the typical visual results of PSNR, SSIM, DISTS, and our UniDASTN are demonstrated in Figure 7. It can be seen that the quality rank of the distorted image predicted by our UniDASTN is consistent with the MOS result. This verifies that our UniDASTN can distinguish distortion types with similar scores.

Fig. 7.

View Figure

Effectiveness of our feature extraction backbone. To assess the effectiveness of our feature extraction backbones, we perform an ablation study on the CSIQ and TID2013 datasets. The results are shown in Table 7, where the best results are in bold, and M6–M8 represent our networks using different backbones. Specifically, the M6 network uses the ResNet50 as the feature extraction backbone, the M7 network uses the Inception-Resnet-V2 as the feature extraction backbone, and the M8 network uses the ViT-B\(/\)8 as the feature extraction backbone. It can be seen that the M8 network makes the best performances on the two datasets. These experimental results demonstrate that the selection of the ViT-B\(/\)8 is beneficial and effective in FR-IQA.

Effectiveness of our feature fusion technique. To assess the effectiveness of our feature fusion technique, we perform an ablation study on the CSIQ and TID2013 datasets using five different feature fusion techniques. Figure 8 shows the five techniques used for feature fusion, where Scheme (a) is the fusion technique reported in [43, 67], Scheme (b) is the fusion technique used in [58], Scheme (c) is the proposed feature fusion block, Scheme (d) is derived from Scheme (c) by adding the average pooling operation, and Scheme (e) is derived from Scheme (c) by replacing the operation of “concatenate” with “add”. Note that for the fusion techniques presented in Scheme (a) and Scheme (b), their outputs are the quality scores. These experimental results are provided in Table 8.

Fig. 8.

View Figure

Table 8 shows that the fusion technique of Scheme (c) makes the best PLCC and SRCC results. In a comparison of Scheme (a) and Scheme (c), an additional Transformer encoder used to fuse the extracted features can improve the prediction performance. This is because the additional transformer encoder can act as an aggregator to aggregate the information of each token. Moreover, in a comparison of Scheme (c) and Scheme (d) as well as Scheme (c) and Scheme (e), if the pooling operation is added or the operation of “concatenate” is replaced with “add”, the information of these tokens is inevitably lost and thus the FR-IQA performance decreases. These experimental results demonstrate that our used feature fusion technique can result in more accurate quality prediction.

Effectiveness of our siamese transformer network. To assess the effectiveness of our Siamese transformer network, we perform an ablation study on the CSIQ and TID2013 datasets. The results are shown in Table 9, where the best results are in bold, and the schemes 1–3 represent different input combinations used in the spatial attention module. There are three types of inputs used in our siamese transformer network, namely \(z_{0}^{ref}, z_{0}^{dist},\) and \(z_{0}^{diff}.\) If the number of inputs decreases to 2 (i.e., only \(z_{0}^{ref}\) and \(z_{0}^{diff}\) or \(z_{0}^{dist}\) and \(z_{0}^{diff}\) is input to the spatial attention module), there are two technical schemes in total. Specifically, the scheme 1 selects \(z_{0}^{ref}\) and \(z_{0}^{diff}\) as the inputs of the spatial attention module, and the scheme 2 uses \(z_{0}^{dist}\) and \(z_{0}^{diff}\) as the inputs of the spatial attention module. This means that, for the schemes 1–2, only one branch of the siamese transformer network is used in the spatial attention module. For comparison, the proposed scheme is denoted as the scheme 3 in which \(z_{0}^{ref}, z_{0}^{dist},\) and \(z_{0}^{diff}\) are all selected as the inputs of the spatial attention module. From Table 9, it can be seen that the scheme 3 makes the best results. These experimental results demonstrate that the features extracted by the two branches of our siamese transformer network are highly complementary. The scheme 1 and the scheme 2 only use the features of one branch. For the scheme 3, the features of two branches are both used and help to make good IQA whole performance.

To make easy understanding of the attention mechanism of Transformer [70], some visual examples of our Siamese transformer network are shown in Figure 9, where three distorted images and their corresponding attention maps of different heads are presented. In this experiment, the attention map of an image is obtained by averaging all attention weights in the self-attention module and resizing to the original image size. It can be found that different heads have different preferences in their attention maps. Through the collaboration of these heads, our UniDASTN can simulate the HVS to make an accurate quality prediction. Furthermore, visual examples of the two branches \(dist\sim diff\) (up branch) and \(ref\sim diff\) (down branch) of our Siamese transformer network are also illustrated. Figure 10 presents the attention maps of the Head 5, where the attention map of the up branch, the attention map of the down branch, and the attention map of two branches are all shown. It can be observed that the attentions of the two branches are spatially placed in different regions. This illustrates that the use of both branches can help in perceiving image quality.

Fig. 9.

View Figure

Fig. 10.

View Figure

Effectiveness of the inputs of siamese transformer network. To assess the effectiveness of our strategies for the input types, we perform an ablation study on the CSIQ and TID2013 datasets. The results are shown in Table 10, where the best results are in bold. Note that there are three embedding features feed into the siamese transformer network, namely \(z_{0}^{ref}, z_{0}^{dist},\) and \(z_{0}^{diff}\). Therefore, there are three schemes input of siamese transformer network in total. Specifically, for the Scheme 4, \(z_{0}^{diff}\) is input to the transformer encoder, and \(z_{0}^{dist}\) and \(z_{0}^{ref}\) are input to the transformer decoder. For the Scheme 5, \(z_{0}^{ref}\) is input to the transformer encoder, and \(z_{0}^{dist}\) and \(z_{0}^{diff}\) are input to the transformer decoder. For the Scheme 6, \(z_{0}^{dist}\) is input to the transformer encoder, and \(z_{0}^{ref}\) and \(z_{0}^{diff}\) are input to the transformer decoder. It can be seen that the Scheme 4 makes three best results and one second-best result. Therefore, the Scheme 4 is better than the Scheme 5 and the Scheme 6. These experimental results demonstrate that, as to the input of transformer encoder, the embedding features of “perceptual differences” are preferable to the reference or distortion embedding features. The aggregation of the embedding features of “perceptual differences” with the transformer encoder helps to improve IQA performance.