Distribution matching and structure preservation for domain adaptation

Distribution matching aims to reduce the distribution difference between the source and target domains. Maximum mean discrepancy (MMD) [20] is often used to measure the distribution difference. Based on MMD, joint distribution adaptation (JDA) [21] minimizes the class-wise distribution distance between the source and target domains to match their marginal and conditional distribution difference. Furthermore, to mine discriminative information within the two domains, domain invariant and class discriminative feature learning (DICD) [22] also minimizes the intra-class scatter and maximizes the inter-class dispersion simultaneously, while unsupervised metric transfer learning method (UMTL) [23] also maximizes the inter-class distance.

These above methods conduct distribution matching in a principal dimension reduction procedure to exploit a shared feature space. However, they still need to utilize a traditional classification method for label prediction of the target data. Moreover, the cross-domain discrepancy still exists in the reduced latent feature space. Our DMSP minimizes the class-wise distribution distance and maximizes the inter-class distance simultaneously as in [23], but in a different way. It conducts distribution matching through a classifier learning procedure, which directly obtains an adaptive classifier for the target data.

Local structure preservation aims to preserve the local geometric structure of given data. Graph regularization is devoted to preserving the local structure which encourages the samples from the same category to be close with each other. In DA problems, [24, 25] minimize graph regularization concerning the whole of the source and target data to preserve their space relationship, where the graph regularization can be regarded as inter-domain graph regularization. [26, 27] minimize such inter-domain graph regularization to ensure that the inferred data labels comply with the local structure of the source and target data.

The methods mentioned above use the inter-domain graph regularization to preserve the local structure of the whole data in different domains (i.e., inter-domain local structure). However, the difference across domains might cause the local manifold structures of the source data and target data to be different. Specifically, the adjacent samples from different domains might belong to different categories. As a result, forcibly minimizing the inter-domain graph regularization will degrade the DA performance. This paper proposes intra-domain graph regularization concerning source and target data separately, which can preserve the respective local manifold structures of source data and target data.

Adaptive classifier learning aims to directly obtain an adaptive classifier trained on source labeled samples that performs well on target data. Adaptation regularization based transfer learning (ARLT) [15] learns the adaptive classifier by jointly minimizing inter-domain graph regularization, the structural risk function, and class-wise distribution distance between domains. Clustering for domain adaptation (DAC) [28] further explores the cluster structure of target data. However, ARTL and DAC are designed only for the original feature where feature distortion will undermine the performance [19]. On the basis of ARTL, manifold dynamic distribution adaptation (MDDA) [29] learns the adaptive classifier on the Grassmann manifold (GM) to overcome feature distortion.

DMSP is also based on ARTL aiming to learn the adaptive classifier on the GM, but it is different from ARTL and MDDA. On the one hand, ARTL and MDDA utilize the inter-domain graph regularization to preserve the manifold consistency underlying the whole data, which will result in different inferred labels for adjacent samples due to different local manifold structures of the source data and the target data. In contrast, DMSP uses the intra-domain graph regularization to induce adjacent samples in the same domain to be inferred as the same label. On the other hand, ARTL and MDDA fail to consider the impact of discriminative information within domains on the discriminant structure of the adaptive classifier, while DMSP maximizes the inter-class distance to improve the discriminant structure.

In this section, we detail the proposed distribution matching and structure preservation for domain adaptation (DMSP) method.

Given the source domain \(\mathcal{D}_{s} { = }\left\{ {{\mathbf{x}}_{i} ,y_{i} } \right\}_{i = 1}^{{n_{s} }}\) and the target domain \(\mathcal{D}_{t} { = }\left\{ {{\mathbf{x}}_{j} } \right\}_{{j = n_{s} + 1}}^{{n_{s} + n_{t} }}\), where \({\mathbf{x}}_{i} \in {\mathbb{R}}^{1 \times m}\) is a source sample associated with its label \(y_{i} \in \left\{ {1,2, \ldots ,Cl} \right\}\) and \({\mathbf{x}}_{j} \in {\mathbb{R}}^{1 \times m}\) is a target sample, we assume that the two domains have different distributions, but share the same label space and feature space. The goal of DMSP is to train an adaptive classifier on \(\mathcal{D}_{s}\), which can perform well on \(\mathcal{D}_{t}\) with low expectation error.

DMSP works two steps. In the first step, DMSP adopts geodesic flow kernel (GFK) [30] to perform manifold feature learning on the Grassmann manifold (GM), which can avoid feature distortion in the original space and align the subspaces of the source and target domains. Specifically, the bases of the source and target subspaces are denoted as \({\mathbf{S}}_{1} \in {\mathbb{R}}^{m \times d}\) and \({\mathbf{S}}_{2} \in {\mathbb{R}}^{m \times d}\) respectively, where \(d\) is the dimension of the low-dimensional linear subspace. Considering \({\mathbf{S}}_{1}\) and \({\mathbf{S}}_{2}\) as two points on the GM, we know from the literature [30] that \(\forall {\mathbf{x}}_{i} ,{\mathbf{x}}_{j} \in {\mathbb{R}}^{m}\) in the original feature space can be projected into a manifold embedded feature space, denoted as \({\mathbf{z}}_{i} ,{\mathbf{z}}_{j}\), and their inner product is where \({\mathbf{G}}\) is the geodesic flow kernel matrix, \(\Psi \left( t \right)\) represents the geodesic flow from \({\mathbf{S}}_{1}\) to \({\mathbf{S}}_{2}\), \(t \in \left[ {0, \, 1} \right]\), \(\Psi \left( 0 \right) = {\mathbf{S}}_{1}\), \(\Psi \left( 1 \right) = {\mathbf{S}}_{2}\). According to (1), the distance between samples on the GM is calculated, and then based on the labeled information of the source samples, the initial pseudo labels \({\overline{\mathbf{Y}}}_{t} = \left[ {\overline{y}_{{n_{s} + 1}} , \ldots ,\overline{y}_{{n_{s} + n_{t} }} } \right]\) of the target samples are obtained using the 1-nearest neighbor (1-NN). In addition, the manifold embedded feature representations of the source and target samples can be obtained in explicit form from [19], that is, \({\mathbf{z}}_{i} = \sqrt {\mathbf{G}} {\mathbf{x}}_{i} ,i = 1,2, \ldots ,n_{s} + n_{t}\).

In the second step, DMSP learns an adaptive classifier by simultaneously optimizing the structure risk function, inter-domain distribution matching and intra-domain local structure preservation. The object function of DMSP is formulated aswhere \(k\left\langle { \cdot , \, \cdot } \right\rangle\) is the kernel function, \(\mathcal{H}_{k}\) represents the corresponding Hilbert space, \(\left\| f \right\|_{k}^{2}\) is the square norm of \(f\), the shrinkage regularization parameter \(\sigma > 0\), the intra-domain graph regularization parameter \(\gamma > 0\), the distribution adaptation regularization parameter \(\lambda > 0\), and the trade-off parameter \(\beta > 0\).

The first two items in formula (1) are the structural risk minimization on the source domain. According to the representer theorem [31], the structural risk minimization can be reformulated as where \(\left\| \cdot \right\|_{F}^{2}\) represents the F-norm square, \({\mathbf{Y}} = \left[ {y_{1} , \ldots ,y_{{n_{s} }} ,\overline{y}_{{n_{s} + 1}} , \ldots ,\overline{y}_{{n_{s} + n_{t} }} } \right]\) is the label matrix, for multi-class problems, \({\mathbf{Y}} \in {\mathbb{R}}^{{Cl \times \left( {n_{s} + n_{t} } \right)}}\), \({{\varvec{\upomega}}} = \left[ {\omega_{1} ,\omega_{2} , \cdots ,\omega_{{n_{s} + n_{t} }} } \right]^{{\text{T}}}\) is the coefficients, \({\mathbf{K}} \in {\mathbb{R}}^{{\left( {n_{s} + n_{t} } \right) \times \left( {n_{s} + n_{t} } \right)}}\) is the kernel matrix, its elements are \(\left( {\mathbf{K}} \right)_{ij} = k\left( {{\mathbf{z}}_{i} ,{\mathbf{z}}_{j} } \right)\), \({\mathbf{E}} = {\text{diag}}\left( {E_{1} ,E_{2} , \ldots ,E_{{n_{s} + n_{t} }} } \right)\), when \(1 \le i \le n_{s}\), \(E_{i} = 1\), when \(i \ge n_{s} + 1\), \(E_{i} = 0\), matrix \({\mathbf{E}}\) can remove unreliable pseudo labels in the target domain.

The third item in formula (1) is the proposed intra-domain graph regularization, which aims to preserve the respective local structures of the source and target domains, and to induce adjacent samples in the same domain to be inferred as the same label. Let \(\mathcal{N}_{p}^{s} \left( \cdot \right)\) and \(\mathcal{N}_{p}^{t} \left( \cdot \right)\) be the sets of p-nearest neighbors in source and target domains, respectively. The intra-domain graph regularization is defined aswhere

Let \({\mathbf{W}} = {\mathbf{W}}_{s} + {\mathbf{W}}_{t}\), intra-domain graph Laplacian matrix \({\mathbf{L}} = {\mathbf{I}} - {\mathbf{B}}^{{{{ - 1} \mathord{\left/ {\vphantom {{ - 1} 2}} \right. \kern-\nulldelimiterspace} 2}}} {\mathbf{WB}}^{{{{ - 1} \mathord{\left/ {\vphantom {{ - 1} 2}} \right. \kern-\nulldelimiterspace} 2}}}\),\({\mathbf{I}}\) is the identity matrix, \({\mathbf{B}} = {\text{diag}}\left( {B_{1} ,B_{2} , \ldots ,B_{{n_{s} + n_{t} }} } \right)\), \(B_{i} = \sum\nolimits_{j = 1}^{{n_{s} + n_{t} }} {{\mathbf{W}}_{ij} }\). The intra-domain graph regularization can be expressed as

The fourth item in formula (1) is the inter-domain distribution adaptation regularization, which simultaneously minimizes the class-wise distribution distance between domains and maximizes the inter-class distance in the source domain as in [23], where the class-wise distribution distance is formulated asand the inter-class distance is

Finally, the inter-domain distribution adaptation becomeswhere \(\mathcal{D}_{t}^{\left( 0 \right)} = \mathcal{D}_{t}\), \(\mathcal{D}_{s}^{\left( 0 \right)} = \mathcal{D}_{s}\), \(\mathcal{D}_{t}^{\left( c \right)} = \left\{ {{\mathbf{z}}_{j} :{\mathbf{z}}_{j} \in \mathcal{D}_{t} \wedge \overline{y}_{j} = c \ne 0} \right\}\), \(\mathcal{D}_{s}^{\left( c \right)} = \left\{ {{\mathbf{z}}_{i} :{\mathbf{z}}_{i} \in \mathcal{D}_{s} \wedge y_{i} = c \ne 0} \right\}\), \(n_{s}^{\left( c \right)} = \left| {\mathcal{D}_{s}^{\left( c \right)} } \right|\), \(n_{t}^{\left( c \right)} = \left| {\mathcal{D}_{t}^{\left( c \right)} } \right|\), \(n_{s}^{{\left( {\overline{c}} \right)}} = n_{s} - n_{s}^{\left( c \right)}\), \({\mathbf{M}} = \sum\nolimits_{c = 0}^{Cl} {{\mathbf{M}}_{c} }\), \({\overline{\mathbf{M}}} = \sum\nolimits_{c = 1}^{Cl} {{\overline{\mathbf{M}}}_{c} }\).

Substituting with formulas (3), (7) and (10), the object function of DMSP in formula (2) can be reformulated as

Setting \({{\partial L\left( {{\varvec{\upomega}}} \right)} \mathord{\left/ {\vphantom {{\partial L\left( {{\varvec{\upomega}}} \right)} {\partial {{\varvec{\upomega}}}}}} \right. \kern-\nulldelimiterspace} {\partial {{\varvec{\upomega}}}}} = {\mathbf{0}}\), we obtain the solution

Then, we can obtain the adaptive classifier \(f\left( {\mathbf{z}} \right) = \sum\nolimits_{i = 1}^{{n_{s} + n_{t} }} {\omega_{i} k\left( {{\mathbf{z}}_{i} ,{\mathbf{z}}} \right)}\). Like ARTL [15], DMSP iteratively updates the target pseudo labels, eventually optimizing itself.

Finally, DMSP is summarized in Algorithm 1. In addition, a chart showing the architecture of DMSP is given in Fig. 2.

In this section, we evaluate the performance of DMSP through extensive experiments on widely used cross-domain image datasets.

This paper uses public cross-domain classification datasets for experiments, including Office + Caltech, ImageCLEF -DA and Office-31. Office + Caltech dataset [30] consists of 4 domains, which are from Amazon, Webcam, DSLR and Caltech-256, respectively. These 4 domains are abbreviated as A10, W10, D10 and C10, with 10 common categories. Accordingly, 12 cross-domain classification tasks can be formed, i.e., A10 → C10, W10 → D10, etc. Note that \(\mathcal{D}_{s}\) → \(\mathcal{D}_{t}\) represents cross-domain classification task from \(\mathcal{D}_{s}\) to \(\mathcal{D}_{t}\).

Office-31 [32] consists of three domains: Amazon, Webcam and DSLR. These three domains are abbreviated as A31, W31 and D31, with 31 shared categories. Similarly, we can construct 6 tasks. ImageCLEF-DA is composed of three domains, which are, respectively, from Caltech-256, ImageNet ILSVRC 2012 and Pascal VOC 2012. These three domains are abbreviated as P12, I12, C12, with 12 common categories, and 6 tasks can be formed. For Office + Caltech, the SURF [30] features and Decaf6 [33] features are used for experiments. For ImageCLEF-DA and Office-31, we adopt the Resnet-50 features, extracted from the Resnet-50 model [34]. Figure 3. shows some exemplary images from these datasets and Table 1 lists the descriptions of these datasets.

We compare DMSP against the no adaptation baseline (i.e., 1-NN) and the state-of-the-art DA methods such as feature adaptation methods (i.e., transfer component analysis (TCA) [10], GFK [30], subspace alignment (SA) [12], JDA [21], group-lasso regularized optimal transport (OT-GL) [11], joint geometrical and statistical alignment (JGSA) [35], UTML [23], structure preservation and distribution alignment (SPDA) [24]), adaptive classifier learning methods (i.e., DCA [28], ARTL [15], MDDA [29]), deep DA methods (i.e., deep adaptation networks (DAN) [36], domain-adversarial neural network (DANN) [37], joint adaptation network (JAN) [38], collaborative and adversarial network (CAN) [39], conditional domain-adversarial network (CDAN) [40], domain-adversarial residual-transfer (DART) [41], multi-representation adaptation Network (MRAN) [42], and discriminative manifold propagation (DMP) [43]).

The dimension \(d\) of the low-dimensional subspace in DMSP method can be selected by the subspace disagreement measure proposed in [30]. In the comparative experiment, considering that the proposed method and ARTL share some of the same type parameters, for fair comparison, we uniformly set the corresponding parameters \(\sigma = 0.1\), \(\lambda = 10\), \(p = 10\), \(T = 10\), and the kernel function is set as Gaussian kernel function. In addition, for datasets using shallow features [i.e., Office + Caltech (SURF)], we set \(\gamma = 0.1\), while for datasets using deep features (i.e., Office + Caltech (Decaf6), ImageCLEF-DA and Office-31), we set \(\gamma = 1\). Furthermore, the trade-off parameter is set to \(\beta = 0.1\) on all datasets. The parameters of other DA methods are selected based on experience, and the experimental results under the parameters used in the original literature are compared. The best result is adopted.

In our experiments, we use classification Accuracy on the target domain as the evaluation measurement, which is widely adopted in existing literature [15, 21, 24, 25, 28, 29]: where \(f\left( {\mathbf{x}} \right)\) and \(y\) are the prediction label and truth label of \({\mathbf{x}}\), respectively.

The classification accuracy results of Office + Caltech using SURF and Decaf6 features are reported in Tables 2 and 3, respectively. We observe that the classification accuracy of no adaptation baseline (i.e., 1-NN) on Office + Caltech (SURF) and Office + Caltech (Decaf6) is lower than all the DA methods. Therefore, whether we use the shallow features or deep features, DA is necessary for cross-domain classification. Compared with those adaptive classifier learning methods (i.e., DCA, ARTL and MDDA), our DMSP (also an adaptive classifier learning method) preserves the intra-domain local structure and enhances the discriminant structure of the classifier, thus outperforming them on almost all the tasks. Another observation is that DMSP achieves the highest average classification accuracy on both Office + Caltech (SURF) and Office + Caltech (Decaf6). Even though the Decaf6 features yield an obvious improvement over the SURF features, DMSP still improves the final classification significantly, by more than 1.8% compared with the best comparison method (MDDA).

From Tables 2 and 3 we also note that on the tasks of W10 → C10 (SURF) and D10 → C10 (SURF), DMSP dominate over other DA methods except OT-GL. A possible explanation is that there is great difference between domains, and DMSP preserves the respective local structure of each domain but it does not outright advantage over sample-based matching in OT-GL. For the tasks W10 → D10 and D10 → W10, DMSP is beaten by JDA, JGSA, DCA and MDDA as reported in Tables 2 and 3. This is because D10 and W10 are closest pair of domains and our DMSP preserving the intra-domain local structure has not superiority. The fact that D10 and W10 are closest pair of domains is clearly found from the 1-NN accuracies. Although DMSP does not achieve the best performance in all tasks, DMSP performs better on most tasks (17/24). Particularly, on C10 → D10, A10 → W10 (SURF), A10 → C10 (SURF), C10 → W10 (Decaf6) and A10 → D10 (Decaf6), our DMSP improves by around 5% over the best comparison.

The results of ImageCLEF-DA and Office-31 datasets are also reported in Table 4. It can be found that using Resnet-50 features, DMSP also outperforms the related methods of ARTL and MDDA. This is because DMSP preserves the intra-domain local structure and makes the inter-class centroids separable. However, for the tasks D31 → W31 and W31 → D31, DMSP does not produce improvements in the classification accuracy, and is even beaten by the baseline method 1-NN. The main reason is that the two domain are very close and the advantage of DMSP (e.g., intra-domain local structure preservation) is not fully realized, instead it causes a negative impact. We also note although ARTL, MDDA and DMSP are non-deep DA methods, they outperform the deep DA methods (e.g., DAN, DANN, JAN, CAN, DART and MRAN), which demonstrates the significance of non-deep DA methods. Moreover, our DMSP achieves the highest average classification accuracy (88.6%), which is 0.3% higher than the best comparison. Especially, on the tasks of A31 → D31, DMSP improves by up to nearly 3 points over the best comparison result.

DMSP mainly consists of following components on top of the structural risk minimization: the inter-domain distribution adaptation regularization with inter-class distance maximization added, the intra-domain graph regularization and the embedded features. We, thus, performed an ablation study to investigate these components, where we remove each component of DMSP separately and observe how it changes in performance. Four variants of DMSP are as follows: (a) DMSP-L₁, removes the inter-domain distribution adaptation regularization; (b) DMSP-L₂, removes the intra-domain graph regularization; (c) DMSP-L₃, removes the inter-class distance maximization; (d) DMSP-original, uses the original features instead of the embedded features to train classifier.

The classification accuracy results of the variants on several tasks are reported in Table 5. We see that DMSP outperforms all of the variants, suggesting that each component is important to DMSP. Expectedly, the overall classification performance of DMSP-L₁ is the worst compared with other variants. This is because distribution matching is essential to reduce the distribution difference between domains. DMSP-original is obviously inferior to DMSP on several tasks (e.g., C10 → A10 (SURF), A10 → W10 (SURF), and D10 → A10 (SURF), which indicates that embedded features learned on the Grassmann manifold (GM) can effectively alleviate feature distortion and achieve subspace alignment for those domain pairs from Office + Caltech (SURF). DMSP-L₂ degrades the classification performance on these tasks, indicating that it is necessary to preserve intra-domain local structure for DMSP. The inter-class distance maximization can separate the inter-class centroids so as to enhance the discriminant structure of the final classifier, therefore removing it will degrade the classification performance. DMSP-L₃ has lower classification accuracy compared with DSMP indicating the necessity of inter-class distance maximization.

To verify the advantage of our proposed intra-domain graph regularization over the inter-domain graph regularization [15, 29], we use the inter-domain graph regularization instead of intra-domain graph regularization to obtain another variant of DMSP, denoted as DMSP-L₄, then observe the change in performance. The classification accuracy results of DMSP and DMSP-L₄ on several tasks are demonstrated in Fig. 4, where C10 → W10, A10 → W10, W10 → C10 are from Office + Caltech (SURF) dataset, C10 → D10, A10 → D10, D10 → A10 are from Office + Caltech (Decaf6) dataset. We find that the classification accuracy of DMSP on each task is higher than that of DMSP-L₄, whether based on shallow feature (i.e., SURF) or deep feature (i.e., Decaf6 and ResNet50). This is because intra-domain graph regularization aims to preserve the respective local structure of each domain while intra-domain graph regularization ignoring difference between domains aims to preserve the local structure of the whole data in different domains. Therefore, the intra-domain graph regularization is more conducive to enhancing the performance of final classification than the inter-domain graph regularization.

To further verify the performance of DMSP, we calculate and observe the average number of samples from a different class in each set of 10-nearest neighbors (ANDC) based on the label distribution in the target domain. The label distribution can be obtained by the final adaptive classifier learned by DMSP. If the final classifier is able to discriminate the target samples, the adjacent samples are predicted as the same label and the value of ANDC will be small. Figure 5 shows the ANDC values produced by DMSP and the related works (i.e., ARTL, MDDA) on several tasks from Office + Caltech (SURF). As can be observed, the ANDC values differ greatly. Especially, on tasks C10 → W10 and A10 → W10, the ANDC value of ARTL reaches almost 7. That is, on average, up to 7 of each 10-nearest neighbors are predicted to be different categories. The main reason is that ARTL learns the adaptive classifier in original space where feature distortion will negatively affect the performance of the classifier, and ignores the discriminative information within domains. MDDA learns the adaptive classifier on the Grassmann manifold (GM), but it also ignores the impact of the intra-domain local structure and the inter-class distance on the discriminant structure of the adaptive classifier. Therefore, the ANDC value of MDDA is smaller than that of ARTL, but larger than that of our DMSP, which jointly minimizes the structural risk function, the distribution adaptation regularization with inter-class distance maximization added and the intra-domain graph regularization, and learns the adaptive classifier on the GM.

Experiments are conducted on randomly selected tasks A10 → C10 with Decaf6 features, A10 → W10 with SURF features, C12 → p12 and W31 → A31 to analyze the parameter sensitivity and convergence of DMSP. Since we have verified that our proposed DMSP performs well, when \(\sigma = 0.1\), \(\lambda = 10\), \(p = 10\) are fixed to be the same as ARTL on all cross-domain classification tasks, we only evaluate the sensitivity of intra-domain graph regularization parameter \(\gamma\) and trade-off parameter \(\beta\). The classification accuracy curves of the selected tasks are provided in Fig. 6.

\(\gamma\) is the intra-domain graph regularization parameter. If \(\gamma\) is too small, the intra-domain local structure cannot be preserved; if it is too large, the discriminant information of domains and the distribution adaptation will be ignored. Figure 6a shows that DMSP achieves better DA performance in the range \(\gamma \in \left[ {0.1, \, 1} \right]\). Besides, \(\beta\) is the trade-off parameter between inter-domain and inter-class distribution difference. Figure 6b reveals that DMSP has better DA performance in the range \(\beta \in \left[ {0.1, \, 0.4} \right]\).

Finally, we check the convergence property of DMSP through empirical analysis. The convergence curves of the classification accuracy of DMSP are shown in Fig. 7. It can be observed that the classification accuracy of DMSP in each task increases steadily with more iterations and tends to stabilize within 10 iterations, indicating that DMSP converges within a couple of iterations.