Top

Neural Processing Letters

Published in:

14-08-2023

Self-training and Multi-level Adversarial Network for Domain Adaptive Remote Sensing Image Segmentation

Authors: Yilin Zheng, Lingmin He, Xiangping Wu, Chen Pan

Published in: Neural Processing Letters | Issue 8/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Unsupervised domain adaptive (UDA) image segmentation has received more and more attention in recent years. Domain adaptive methods can align the features of data in different domains, so that segmentation models can be migrated to data in other domains without incurring additional labeling costs. The traditional adversarial training uses the global alignment strategy to align the feature space, which may cause some categories to be incorrectly mapped. At the same time, in high-spatial resolution remote sensing images (RSI), the same category from different scenes (such as urban and rural areas) may have completely different distributions, which severely limits the accuracy of UDA. In order to solve these problems, in this paper: (1) a multi-level adversarial network at category-level is proposed, aiming at integrating feature information in different dimensions, studying the joint distribution at category-level, and aligning each category with adaptive adversarial loss in different dimensional spaces. (2) Use covariance regularization to optimize self-training. A method combining self-training with adversarial training is proposed, optimizes the domain adaptation effect, reduces the negative impact of false pseudo-label iteration caused by self-training. We demonstrated the latest performance of semantic segmentation on challenging LoveDA datasets. Experiments on “urban-to-rural” and “rural-to-urban” show that our method has better performance than the most advanced methods.

previous article A Robust Extreme Learning Machine Based on Adaptive Loss Function for Regression Modeling

next article Edge Detection-Guided Balanced Sampling

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Inglada J (2007) Automatic recognition of man-made objects in high resolution optical remote sensing images by svm classification of geometric image features. Isprs J Photogramm Remote Sens 62(3):236–248CrossRef

Maloof MA, Langley P, Binford TO et al (2003) Improved rooftop detection in aerial images with machine learning. Mach Learn 53(1–2):157–191CrossRef

Pal SK, Ghosh A, Shankar BU (2000) Segmentation of remotely sensed images with fuzzy thresholding, and quantitative evaluation. Int J Remote Sens 21(11):2269–2300CrossRef

Sirmaek B, Unsalan C (2009) Urban-area and building detection using sift keypoints and graph theory. IEEE Trans Geosci Remote Sens 47(4):1156–1167CrossRef

Trias-Sanz R, Stamon G, Louchet J (2008) Using colour, texture, and hierarchial segmentation for high-resolution remote sensing. Isprs J Photogramm Remote Sens 63(2):156–168CrossRef

Turker M, Koc-San D (2015) Building extraction from high-resolution optical spaceborne images using the integration of support vector machine (svm) classification, hough transformation and perceptual grouping. Int J Appl Earth Obs Geoinf 34:58–69

Deng Z, Sun H, Zhou S et al (2018) Multi-scale object detection in remote sensing imagery with convolutional neural networks. ISPRS J Photogramm Remote Sens 145:3–22. https://doi.org/10.1016/j.isprsjprs.2018.04.003CrossRef

Xia G, Bai X, Ding J, et al (2018) DOTA: a large-scale dataset for object detection in aerial images. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018. Computer Vision Foundation/IEEE Computer Society, pp 3974–3983

Zheng Z, Zhong Y, Wang J, et al (2020) Foreground-aware relation network for geospatial object segmentation in high spatial resolution remote sensing imagery. In: 2020 IEEE/CVF conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. Computer Vision Foundation/IEEE, pp 4095–4104, https://doi.org/10.1109/CVPR42600.2020.00415, https://openaccess.thecvf.com/content_CVPR_2020/html/Zheng_Foreground-Aware_Relation_Network_for_Geospatial_Object_Segmentation_in_High_Spatial_CVPR_2020_paper.html

10.

Pang J, Li C, Shi J et al (2019) \(\mathscr {R}\)\({}^{\text{2 }}\)-cnn: fast tiny object detection in large-scale remote sensing images. IEEE Trans Geosci Remote Sens 57(8):5512–5524. https://doi.org/10.1109/TGRS.2019.2899955CrossRef

11.

Deng Z, Sun H, Zhou S et al (2019) Learning deep ship detector in SAR images from scratch. IEEE Trans Geosci Remote Sens 57(6):4021–4039. https://doi.org/10.1109/TGRS.2018.2889353CrossRef

12.

Chen L, Yang Y, Wang J, et al (2016) Attention to scale: Scale-aware semantic image segmentation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016. IEEE Computer Society, pp 3640–3649

13.

Chen LC, Papandreou G, Kokkinos I et al (2018) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848CrossRef

14.

Chen L, Papandreou G, Schroff F, et al (2017) Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587

15.

Chen L, Zhu Y, Papandreou G, et al (2018a) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari V, Hebert M, Sminchisescu C, et al (eds) Computer Vision—ECCV 2018—15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, Lecture Notes in Computer Science, vol 11211. Springer, pp 833–851

16.

Yang M, Yu K, Zhang C, et al (2018) Denseaspp for semantic segmentation in street scenes. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018. Computer Vision Foundation/IEEE Computer Society, pp 3684–3692

17.

Wang J, Zheng Z, Ma A, et al (2021a) LoveDA: a remote sensing land-cover dataset for domain adaptive semantic segmentation. In: Vanschoren J, Yeung S (eds) Proceedings of the neural information processing systems track on datasets and benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021

18.

Liu W, Luo Z, Cai Y et al (2021) Adversarial unsupervised domain adaptation for 3d semantic segmentation with multi-modal learning. ISPRS J Photogramm Remote Sens 176:211–221. https://doi.org/10.1016/j.isprsjprs.2021.04.012CrossRef

19.

Huang L, Fu Q, He M et al (2021) Detection algorithm of safety helmet wearing based on deep learning. Concurr Comput Pract Exp 33(13):e6234. https://doi.org/10.1002/cpe.6234CrossRef

20.

Huang L, Chen C, Yun J et al (2022) Multi-scale feature fusion convolutional neural network for indoor small target detection. Front Neurorobotics 16:881021. https://doi.org/10.3389/fnbot.2022.881021CrossRef

21.

Jiang D, Li G, Tan C et al (2021) Semantic segmentation for multiscale target based on object recognition using the improved faster-rcnn model. Future Gener Comput Syst 123:94–104. https://doi.org/10.1016/j.future.2021.04.019CrossRef

22.

Sun Y, Zhao Z, Jiang D et al (2022) Low-illumination image enhancement algorithm based on improved multi-scale retinex and abc algorithm optimization. Front Bioeng Biotechnol 10:865820. https://doi.org/10.3389/fbioe.2022.865820CrossRef

23.

Yun J, Jiang D, Sun Y et al (2022) Grasping pose detection for loose stacked object based on convolutional neural network with multiple self-powered sensors information. IEEE Sens J. https://doi.org/10.1109/JSEN.2022.3190560CrossRef

24.

Liu Y, Jiang D, Xu C et al (2022) Deep learning based 3d target detection for indoor scenes. Appl Intell 53(9):10218–10231CrossRef

25.

Jiang D, Li G, Sun Y et al (2021) Manipulator grabbing position detection with information fusion of color image and depth image using deep learning. J Amb Intell Hum Comput 12(12):10809–10822CrossRef

26.

Liu Y, Jiang D, Duan H et al (2021) Dynamic gesture recognition algorithm based on 3d convolutional neural network. Comput Intell Neurosci 12:1–12

27.

Zhou ZH, Li M (2005) Tri-training: exploiting unlabeled data using three classifiers. IEEE Trans Knowl Data Eng 17(11):1529–1541CrossRef

28.

Goodfellow I, Pouget-Abadie J, Mirza M, et al (2014) Generative adversarial nets. In: Neural Information Processing Systems, pp 2672–2680

29.

Yu W, Bai J, Jiao L (2020) Background subtraction based on gan and domain adaptation for vhr optical remote sensing videos. IEEE Access 8:119144–119157. https://doi.org/10.1109/ACCESS.2020.3004495CrossRef

30.

Li X, Du Z, Huang Y et al (2021) A deep translation (gan) based change detection network for optical and sar remote sensing images. ISPRS J Photogramm Remote Sens 179:14–34. https://doi.org/10.1016/j.isprsjprs.2021.07.007CrossRef

31.

Hoffman J, Tzeng E, Park T, et al (2018) Cycada: Cycle-consistent adversarial domain adaptation. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018, Proceedings of Machine Learning Research, vol 80. PMLR, pp 1994–2003

32.

Tsai YH, Hung WC, Schulter S, et al (2018) Learning to adapt structured output space for semantic segmentation. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR) pp 7472–7481

33.

Luo Y, Zheng L, Guan T, et al (2019) Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In: The IEEE conference on computer vision and pattern recognition (CVPR), pp 2507–2516

34.

Wang H, Shen T, Zhang W, et al (2020) Classes matter: a fine-grained adversarial approach to cross-domain semantic segmentation. In: The European conference on computer vision (ECCV), pp 642–659

35.

Wang X, Jin Y, Long M, et al (2019) Transferable normalization: towards improving transferability of deep neural networks. In: Neural information processing systems, pp 1951–1961

36.

Zhao Y, Zhong Z, Zhao N, et al (2022) Style-hallucinated dual consistency learning for domain generalized semantic segmentation. In: Computer Vision—ECCV 2022—17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, Lecture Notes in Computer Science, vol 13688. Springer, pp 535–552

37.

Huang J, Guan D, Xiao A et al (2022) Multi-level adversarial network for domain adaptive semantic segmentation. Pattern Recognit 123:108384. https://doi.org/10.1016/j.patcog.2021.108384CrossRef

38.

Ning M, Lu D, Wei D, et al (2021) Multi-anchor active domain adaptation for semantic segmentation. In: 2021 IEEE/CVF international conference on computer vision, ICCV 2021, Montreal, QC, Canada, October 10–17, 2021. IEEE, pp 9092–9102

39.

Cheng Y, Wei F, Bao J, et al (2021) Dual path learning for domain adaptation of semantic segmentation. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10–17, 2021. IEEE, pp 9062–9071

40.

Lai X, Tian Z, Xu X, et al (2022) Decouplenet: decoupled network for domain adaptive semantic segmentation. In: Computer Vision—ECCV 2022—17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, Lecture Notes in Computer Science, vol 13693. Springer, pp 369–387

41.

Lian Q, Lv F, Duan L, et al (2019) Constructing self-motivated pyramid curriculums for cross-domain semantic segmentation: a non-adversarial approach. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE, pp 6757–6766

42.

Zou Y, Yu Z, Kumar BVKV, et al (2018) Domain adaptation for semantic segmentation via class-balanced self-training. arXiv:1810.07911

43.

Mei K, Zhu C, Zou J et al (2020) Instance adaptive self-training for unsupervised domain adaptation. In: Vedaldi A, Bischof H, Brox T et al (eds) Computer Vision–ECCV 2020, vol 12371. Springer, Cham, pp 415–430CrossRef

44.

Liu Y, Zhang S, Li Y, et al (2021d) Learning to adapt via latent domains for adaptive semantic segmentation. In: Beygelzimer A, Dauphin Y, Liang P, et al (eds) Advances in neural information processing systems 34: annual conference on neural information processing systems 2021, NeurIPS 2021, December 6–14, 2021, virtual, pp 1167–1178

45.

Liu W, Liu J, Luo Z et al (2022) Weakly supervised high spatial resolution land cover mapping based on self-training with weighted pseudo-labels. Int J Appl Earth Obs Geoinf 112:102931. https://doi.org/10.1016/j.jag.2022.102931CrossRef

46.

Hoyer L, Dai D, Van Gool L (2022b) HRDA: Context-aware high-resolution domain-adaptive semantic segmentation. In: Computer Vision—ECCV 2022—17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXX, Lecture Notes in Computer Science, vol 13690. Springer, pp 372–391

47.

Hoyer L, Dai D, Van Gool L (2022a) DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, pp 9914–9925

48.

Liu Y, Deng J, Gao X, et al (2021b) Bapa-net: boundary adaptation and prototype alignment for cross-domain semantic segmentation. In: 2021 IEEE/CVF international conference on computer vision, ICCV 2021, Montreal, QC, Canada, October 10–17, 2021. IEEE, pp 8781–8791

49.

Wang W, Ma L, Chen M et al (2021) Joint correlation alignment-based graph neural network for domain adaptation of multitemporal hyperspectral remote sensing images. IEEE J Sel Top Appl Earth Obs Remote Sens 14:3170–3184. https://doi.org/10.1109/JSTARS.2021.3063460CrossRef

50.

Pan F, Shin I, Rameau F et al (2020) Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. IEEE conference on computer vision and pattern recoginition (CVPR). Computer Vision Foundation, IEEE, pp 3763–3772

51.

Shen W, Wang Q, Jiang H, et al (2021) Unsupervised domain adaptation for semantic segmentation via self-supervision. In: IEEE international geoscience and remote sensing symposium, IGARSS 2021, Brussels, Belgium, July 11–16, 2021. IEEE, pp 2747–2750

52.

Deng X, Yang HL, Makkar N, et al (2019a) Large scale unsupervised domain adaptation of segmentation networks with adversarial learning. In: IGARSS 2019—2019 IEEE International Geoscience and Remote Sensing Symposium, pp 4955–4958

53.

Wu L, Lu M, Fang L (2022) Deep covariance alignment for domain adaptive remote sensing image segmentation. IEEE Trans Geosci Remote Sens 60:1–11. https://doi.org/10.1109/TGRS.2022.3163278CrossRef

54.

Richter SR, Vineet V, Roth S et al (2016) Playing for data: Ground truth from computer games. In: Part II (ed) Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings. Springer, pp 102–118CrossRef

55.

Cordts M, Omran M, Ramos S, et al (2016) The cityscapes dataset for semantic urban scene understanding. IEEE, pp 3213–3223

56.

He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: CVPR. IEEE Computer Society, pp 770–778

57.

Deng J, Dong W, Socher R et al (2009) Imagenet: a large-scale hierarchical image database. In: Florida USA (ed) 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20–25 June 2009, Miami. IEEE Computer Society, pp 248–255

58.

Yu F, Koltun V (2016) Multi-scale context aggregation by dilated convolutions. In: 4th International conference on learning representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, conference track proceedings, arXiv:1511.07122

59.

Radford A, Metz L, Chintala S (2016) Unsupervised representation learning with deep convolutional generative adversarial networks. Comput Sci. arXiv:1511.06434

60.

Maas AL, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic models. Int Conf Mach Learn 30(1):3

61.

Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Lechevallier Y, Saporta G (eds) 19th International conference on computational statistics, COMPSTAT 2010, Paris, France, August 22–27, 2010–keynote, invited and contributed papers. Physica-Verlag, pp 177–186

62.

Kingma D, Ba J (2015) Adam: a method for stochastic optimization. Computer Science http://arxiv.org/abs/arXiv:1412.6980

63.

Maaten LV, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(2605):2579–2605MATH

64.

Bi X, Chen D, Huang H et al (2023) Combining pixel-level and structure-level adaptation for semantic segmentation. Neural Process Lett. https://doi.org/10.1007/s11063-023-11220-5CrossRef

65.

Li W, Yang X, Li Z (2023) Mlcb-net: a multi-level class balancing network for domain adaptive semantic segmentation. Multimed Syst. https://doi.org/10.1007/s00530-023-01055-4CrossRef

66.

Zhu S, Tian Y (2023) Shape robustness in style enhanced cross domain semantic segmentation. Pattern Recognit 135:109143. https://doi.org/10.1016/j.patcog.2022.109143CrossRef

67.

Zhang Y, Tian S, Liao M et al (2023) A hybrid domain learning framework for unsupervised semantic segmentation. Neurocomputing 516:133–145. https://doi.org/10.1016/j.neucom.2022.10.005CrossRef

Title: Self-training and Multi-level Adversarial Network for Domain Adaptive Remote Sensing Image Segmentation
Authors: Yilin Zheng
Lingmin He
Xiangping Wu
Chen Pan
Publication date: 14-08-2023
Publisher: Springer US
Published in: Neural Processing Letters / Issue 8/2023
Print ISSN: 1370-4621
Electronic ISSN: 1573-773X
DOI: https://doi.org/10.1007/s11063-023-11341-x

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 8/2023

Supervised Learning Strategy for Spiking Neurons Based on Their Segmental Running Characteristics

EfficientU-Net: A Novel Deep Learning Method for Breast Tumor Segmentation and Classification in Ultrasound Images

A Novel Spatio-Temporal Network of Multi-channel CNN and GCN for Human Activity Recognition Based on BAN

Vision-Aware Language Reasoning for Referring Image Segmentation

Analysis of Markovian Jump Stochastic Cohen–Grossberg BAM Neural Networks with Time Delays for Exponential Input-to-State Stability

Mixed Radial Basis Function Neural Network Training Using Genetic Algorithm