View-target relation-guided unsupervised 2D image-based 3D model retrieval via transformer

The purpose of 3D model retrieval is to measure the similarity between the query sample and the samples in the dataset and return the relevant 3D model according to the similarity order. The typical 3D model retrieval methods are mainly divided into model-based 3D model retrieval methods and image-based 3D model retrieval methods. Model-based methods usually extract features from 3D formats, such as point clouds and voxels. For example, [21] utilized the supervised convolutional network to encode the binary voxel grids for 3D model representation. Wu et al. [22] represented a model on a 3D voxel grid using probability distributions of binary variables. Qi et al. [23] utilized the neighbor points at multiple scales to capture the local structure information. In view-based methods, the 3D model is usually projected into a set of 2D view images. Su et al. [24] used a CNN to extract the feature of each view individually and then adopted max-pool operation on all view features to obtain a global model feature. Gao et al. [25] utilized patch-level features to learn the content information and spatial information within multiple views. Watanabe et al. [26] composed a compact representation of an image-based shape retrieval system employing a minimum number of views for each model. However, the above methods require a 3D model as the query, which is not convenient for users when they can only obtain an image of the 3D model. Therefore, many researchers seek retrieval approaches via image-based methods.

The object of domain adaptation is to establish knowledge transfer from the source domain to the target domain. Unsupervised domain adaptation, semi-supervised domain adaptation, and supervised domain adaptation are the three common categories of domain adaptation methods. Both source domain data and a few target domain data with labels are used during training in semi-supervised domain adaptation (SSDA) methods while all samples should be marked in supervised domain adaptation methods. Based on previous studies, [27] benefited from a subset of source domain data only containing samples highly relevant to the target domain by eliminating irrelevant source domain samples. To reduce the accuracy discrepancy across domains for better generalizations, invariant representations and invariant optimal predictors were jointly learned in [28]. Unsupervised domain adaptation (UDA) methods seek to narrow the domain gap between the source and target images without any labels of the target images and learning domain-invariant features is a common strategy for unsupervised domain adaptation. For example, [10] added an additional domain discriminator to the conventional CNN network to enforce the domain alignment. Zhou et al. [29] ensured the alignment at the class level by matching the class centroids of the source and target domains based on eliminating the offsets at the domain level. Long et al. [30] utilized the joint maximum mean difference criterion and used multiple domain-specific layers for adaptation. Multi-layer and multi-kernel maximum mean discrepancies were minimized between the source and target domain data to resolve domain transfer issues in [31]. Wang et al. [32] proposed a method that by designing a robust deep neural network, both source and target domains can be transformed to a shared feature space and the classifier trained on the source domain work well on the target domain. Later on, [33] trained a model on new target domains, while maintaining its performance on the previous target domains without forgetting. Hoyer et al. [34] proposed a method that by training with pseudo-labels fused from multiple resolutions and using an overlapping sliding window mechanism, the robustness of fine-grained pseudo-labels with respect to different contexts can be increased. The development of domain adaptation has also facilitated numerous close applications, such as fault detection [35, 36], frequency detection [37], and modulation classification [38, 39].

Transformer was first invented in 2017 in [40]. Later on, a lot of effort was dedicated to its variants. Among them, vision transformer (ViT) [41] was a representative work that proposed transformer stacks for the image classification task. Subsequently, many ViT-related structures have been proposed. In particular, [42] developed a novel ViT backbone with a safe training strategy for UDA. Also, the encoder–decoder framework of Transformer facilitated various vision tasks, such as image captioning [43, 44], relationship or attribute detection [45, 46]. For instance, [47] applied a Transformer-based dense captioning while [48] focused on textual context and higher word learning efficiency. Recently, due to the necessity of encoding spatial information, the Swin Transformer [49] structure was proposed and also applied in many tasks [50]. The application of the transformer-based method for unsupervised 2D image-based 3D model retrieval, however, is still extremely rare. Besides, we also elaborately design a shared view-guided attentive module to better arm the transformer-based architecture.

The goal of unsupervised 2D image-based 3D model retrieval is to create the cross-domain approach capable of precisely finding similar 3D models given a 2D image. In this task, the data of 2D images contain the images and their labels, denoted as \(S=\{X^S, Y^S\}\), while the data of the target domain only contain the 3D models without labels, denoted as \(T=\{X^t\}\). To learn better domain-invariant features without labels of 3D models, we present transformer-based 3D model retrieval network (T3DRN), which will be introduced in detail in Sect. 3.2.

We build up the transformer-based 3D model retrieval network on top of [42]. The T3DRN consists of ViT stacks (shown in Fig. 3) and a shared view-guided attentive model(SVAM), which will be elaborated in Sect. 3.3. T3DRN is composed of three ViT-block branches for 2D images, 3D model images and perturbed 3D model images, respectively. Each ViT block comprises of 4 transformer layers; the implementation of each Transformer layer is as follows. For each image either from 2D image gallery or 3D model gallery, the patch embedding layer first transforms it into a token sequence including a special class token and image tokens [42] to get visual features. Then, they are added by positional encoding in [40]. We adopt the positional encoding (PE) procedure with sin and cos functions.

It should be noted that PE operation only occurs at the bottom of each ViT block. The dimension of PE is the same as the input, so PE embedding can be added directly to the input. After the visual features are added with PE, the output is denoted as F, which is input into three linear projectors to attain three different vectors Q, K, V. These three vectors are fed into the ViT block, the \(l{\textrm{th}}\) layer in a ViT block is given by:where \(\varphi\) is layer normalization on residual output, PF represents the feed-forward layer which consists of two linear layers with a nonlinear activation function in between. \(\omega\) is the output of assembled multi-head attention with a layer normalization by \(\varphi\). \(M_1^l\) and \(M_2^l\) are the weights trained for the feed-forward layers, and \(b_1^l\) and \(b_2^l\) are bias vectors. \(F^l\) is the input of the \(l{\textrm{th}}\) encoding layer. \(f_t^l\) is given as the query to the encoding layer and l is the \(l{\textrm{th}}\) encoding layer. Note that \(F^0\) is the aforementioned visual feature F added by positional encodings. MA is a fine-grained component called multi-head attention, which is composed of H parallel partial dot-product attention components. Its realization is as follows:where \(\{h_j | j \in [1,H]\}\) refer to the index of each independent head. \(W_j^q\), \(W_j^K\), \(W_j^V\) denote the linear projectors to the input q, K, V for \(h_j\). \(W^O\) is the weight matrix for each head. It is noted that when the query comes from the decoder layer, and both the keys and values are from the encoder layer, it represents cross-module attention. In contrast, if the queries, keys, and values are all from encoder or decoder, this kind of multi-head attention is named self-attention. A is the scaled dot-product attention operation realized by the equation below.where \(q_i\in R^d\) is a query in all T queries that composes \(q_i\), a group of keys \(k_t\in R^d\) and values \(v_t\in R^d\), where \(t = 1,2,\ldots ,T\), the output of dot-product attention is the weighted sum of the \(v_t\) values. The weights are determined by the dot-products of query \(q_i\) and keys \(k_t\). Specifically, \(k_t\) and \(v_t\) are placed into respective matrices \(K = (k_1,\ldots , k_T)\) and \(V = (v_1,\ldots , v_T)\). d is the dimension of \(q_i\) and \(\sqrt{d}\) is to normalize the dot-product value.

In the end, with the output of l encoding layers, the encoded visual features, \(F^l\), is the final output for the ViT blocks. It is noted that we also adopted the random perturbation strategy and the safe training mechanism in [42]. It also consists of \(F^l_{2D}\), \(F^l_{3D}\) and \(F^l_{3Dpert}\), the latter two parts are as the input to the SVAM.

To guide the training process appropriately with view information, we first learn an adaptive view dictionary \(E=\{e_1,e_2,\ldots ,e_M\}\), where M is the total views in the 3D model dataset. With \(F^l_{3D}\) and \(F^l_{3Dpert}\), and E, we design the SVAM as follows:where F is the input features, in this scenario, it can be either \(F^l_{3D}\) or \(F^l_{3Dpert}\). \(E'\) is the corresponding embeddings of the view labels for each feature in F. MA, \(\varphi\) and PF are the same with Eq. 1. In this way, we can compute the view-guided features \(SVAM(F^l_{3D}, E^{'})\) and \(SVAM(F^l_{3Dpert},E^{'})\), denoted as \(F^v_{3D}\) and \(F^v_{3Dpert}\). They will be added with \(F^l_{3D}\) and \(F^l_{3Dpert}\) as follow for the downstream task:where \(\lambda\) is a balance coefficient between the original features and the view-guided features.

In this section, we show our training and optimization details. In order to enforce the 2D image data to be correctly classified, and the distributions of the representation of the 3D models to be similar with its perturbed counterparts, meanwhile confusing the data between two domains, multiple loss function items are leveraged during the Stochastic Gradient Descent [51] (SGD) at each training step in a training batch as follows:where \(L_{cls}\) is the classification binary cross-entropy loss function of the classifier for 2D image data, \(L_{tgt}\) is the KL divergence loss in [42], \(L_d\) is the domain adversarial loss in [10].

These experiments are carried out on an NVIDIA GTX 2080 Ti GPU with a memory of 11 GB. For the proposed method, \(\beta\) is set to 0.001 as [42], and the learning rate is set to 0.001. The image batch size is set to 32, the iteration in an epoch is 2,000, and the training epoch is 20. The perturbation coefficient \(\alpha\) is set as 0.2, and \(\lambda\) in Eq. 5 is set to 0.3. The confidence threshold \(\epsilon\) [42] is 0.5. For the safe training parameters, we keep the same with [42].