Skip to main content
SpringerLink
Log in

DGCA: high resolution image inpainting via DR-GAN and contextual attention

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The most image inpainting algorithms often have existed problems such as blurred image, texture distortion and semantic inaccuracy, and the image inpainting effect is limited for images with large missing regions and resolution level. To solve above problems, the paper proposes an improved two-stage image inpainting network based on parallel network and contextual attention. Firstly, the improved deep residual network is used to perform generative pixels filling on the missing area, and the first-stage adversarial network is used to complete the edges information. Then, the color features of the filling map are extracted, the edge map is fused and complemented, and the fusion map is used as the conditional label of the second-stage adversarial network. Finally, the image repairing result has obtained through the two-stage network with the contextual attention module. The experiments on public datasets can show that the proposed algorithm can obtain a more realistic repairing effect.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

All data generated or analysed during this study are included in this published article and its supplementary information files. The authors would like to thank Places2, CelebA, Façade and Oxford Building datasets, which allowed us to train and evaluate the proposed model.

References

  1. Barnes C, Shechtman E, Finkelstein A, Goldman D (2009) PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):24

    Article  Google Scholar 

  2. Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp 417–424

  3. Chen T, Zhang X, Hamann B, Wang D, Zhang H (2022) A multi-level feature integration network for image inpainting. Multimed Tools Appl 81:38781–38802

    Article  Google Scholar 

  4. Ding D, Ram S, Rodriguez J (2019) Image inpainting using nonlocal texture matching and nonlinear filtering. IEEE Trans Image Process 28(4):1705–1709

    Article  MathSciNet  Google Scholar 

  5. Doersch C, Singh S, Gupta A, Sivic J, Efros A (2012) What makes Paris look like Paris? ACM Trans Graphics 31(4):101

    Article  Google Scholar 

  6. Fang Y, Li Y, Tu X, Tan T, Wang X (2020) Face completion with hybrid dilated convolution. Signal Process Image Commun 80:115664

    Article  Google Scholar 

  7. Gao S, Cheng M, Zhao K, Zhang X, Yang M, Torr P (2019) Res2Net: a new multi-scale backbone architecture. IEEE Trans Pattern Anal Mach Intell 43(2):652–662

    Article  Google Scholar 

  8. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Proceedings of Advances in Neural Information Processing Systems, pp 2672–2680

  9. Guo Q, Gao S, Zhang X, Yin Y, Zhang C (2018) Patch-based image inpainting via two-stage low rank approximation. IEEE Trans Visual Comput Graphics 24(6):2023–2026

    Article  Google Scholar 

  10. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp 7132–7141

  11. Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Trans Graphics 36(4):107

    Article  Google Scholar 

  12. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of International Conference on Machine Learning, pp 448–456

  13. Jiang B, Huang W, Yang C, Huang Y (2022) Image inpainting based on cross-hierarchy global and local aware network. Multimed Tools Appl.https://doi.org/10.1007/s11042-022-14245-5

  14. Jiang Y, Yang F, Bian Z, Lu C, Xia S (2022) Mask removal: Face inpainting via attributes. Multimed Tools Appl 81:29785–29797

    Article  Google Scholar 

  15. Johnson J, Alahi A, Li F (2016) Perceptual losses for real-time style transfer and super-resolution. In: Proceedings of European Conference on Computer Vision, pp 694–711

  16. Kingm D, Ba J (2014) Adam: A method for stochastic optimization. In Proceedings of 4th International Conference on Learning Representation, pp 58–64

  17. Korhonen J, Junyong Y (2012) Peak signal-to-noise ratio. In Proceedings of International Workshop on Quality of Multimedia Experience Electronics Letters, pp 37–38

  18. Liu G, Reda F, Shih K, Wang T, Tao A, Catanzaro B (2018) Image inpainting for irregular holes using partial convolutions, In: Proceedings of European Conference Computer Vision, pp 89–105

  19. Liu W, Xu D, Tsang I, Zhang W (2019) Metric learning for multi-output tasks. IEEE Trans Pattern Anal Mach Intell 41(2):408–422

    Article  Google Scholar 

  20. Pan J, Sun D, Zhang J, Tang J, Yang J, Tai Y, Yang M (2022) Dual convolutional neural networks for low-level vision. Int J Comput Vision 130:1440–1458

    Article  Google Scholar 

  21. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros A (2016) Context encoders: feature learning by inpainting. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp 2536–2544

  22. Pen H, Wang Q, Wang Z (2021) Boundary precedence image inpainting method based on self-organizing maps. Knowl-Based Syst 216:106722

    Article  Google Scholar 

  23. Qin Z, Zeng Q, Zong Y, Xu F (2021) Image inpainting based on deep learning: A review. Displays 69:102028

    Article  Google Scholar 

  24. Quan W, Zhang R, Zhang Y, Li Z, Wang J, Yan D (2022) Image inpainting with local and global refinement. IEEE Trans Image Process 31:2405–2420

    Article  Google Scholar 

  25. Song Y, Yang C, Lin Z, Liu X, Huang Q, Li H, Kuo C (2018) Contextual-based image inpainting: infer, match and translate. In: Proceedings of European Conference on Computer Vision, pp 3–18

  26. Wan R, Shi B, Li H, Duan L, Kot A (2021) Face image reflection removal. Int J Comput Vision 129:385–399

    Article  MathSciNet  Google Scholar 

  27. Wang Y, Tao X, Qi X, Shen X, Jia J (2018) Image inpainting via generative multi-column convolutional neural networks. In: Proceedings of Advances in Neural Information Processing Systems, pp 329–338

  28. Wang N, Li J, Zhang L, Du B (2019) MUSICAL: multi-scale image contextual attention networks. In: Proceedings of International Joint Conference on Artificial Intelligence, pp 3748–3754

  29. Wang Z, Bovik A, Sheikh H, Simoncelli E (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  30. Wang N, Zhang Y, Zhang L (2021) Dynamic selection network for image inpainting. IEEE Trans Image Process 30:1784–1798

    Article  Google Scholar 

  31. Xie C, Liu S, Li C, Cheng M, Zuo W, Liu X, Wen S, Ding E (2019) Image inpainting with learnable bidirectional attention maps. In: Proceedings of IEEE International Conference on Computer Vision, pp 8858–8867

  32. Xu R, Guo M, Wang J, Li X, Zhou B, Loy C (2021) Texture memory-augmented deep patch-based image inpainting. IEEE Trans Image Process 30:9112–9124

    Article  Google Scholar 

  33. Zeng Y, Gong Y, Zhang J (2021) Feature learning and patch matching for diverse image inpainting. Pattern Recogn 119:108036

    Article  Google Scholar 

  34. Zhang L, Chang M, Chen R (2023) Image inpainting based on sparse representation using self-similar joint sparse coding. Multimed Tools Appl.https://doi.org/10.1007/s11042-023-14337-w

  35. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of European Conference on Computer Vision, pp 294–310

  36. Zhang Y, Ding F, Kwong S, Zhu G (2021) Feature pyramid network for diffusion-based image inpainting detection. Inf Sci 572:29–42

    Article  MathSciNet  Google Scholar 

  37. Zhang X, Wang X, Shi C, Yan Z, Li X, Kong B, Lyu S, Zhu B, Lv J, Yin Y, Song Q, Wu X, Mumtaz I (2022) DE-GAN: domain embedded GAN for high quality face image inpainting. Pattern Recogn 124:108415

    Article  Google Scholar 

  38. Zheng C, Cham T, Cai J (2019) Pluralistic image completion. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp 1438–1447

  39. Zheng C, Cham T, Cai J (2021) Pluralistic free-form image completion. Int J Comput Vision 129:2786–2805

    Article  Google Scholar 

  40. Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A (2018) Places: A 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell 50(6):1452–1464

    Article  Google Scholar 

  41. Zhu M, He D, Li X, Li C, Li F, Liu X, Ding E, Zhang Z (2021) Image inpainting by end-to-end cascaded refinement with mask awareness. IEEE Trans Image Process 30:4855–4866

    Article  Google Scholar 

  42. Zhuo L, Tan S, Li B, Huang J (2022) ISP-GAN: inception sub-pixel deconvolution-based lightweight GANs for colorization. Multimed Tools Appl 81:24977–24994

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by A Project Supported by Scientific Research Fund of Hunan Provincial Education Department under Grant 22A0701, 2022 Institute Scientific Research Project of Hunan University of Information Technology under Grant XXY022ZD01, College Students’ Innovative Entrepreneurial Training Plan Program of Hunan University of Information Technology under Grant X202213836002, China University Innovation Funding—Beslin Smart Education Project under Grant 2022BL055.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Hunan University of Information Technology, Changsha, Hunan, China

    Yuantao Chen

  2. Mountain Yuelu Breeding Innovation Center Limited, Changsha, China

    Runlong Xia

  3. Hunan Provincial Science and Technology Affairs Center, Changsha, China

    Runlong Xia

  4. Hunan ZOOMLION Intelligent Technology Corporation Limited, Changsha, China

    Kai Yang

  5. Hunan WUJO High-Tech Material Corporation Limited, Loudi, China

    Ke Zou

Authors
  1. Yuantao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Runlong Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ke Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuantao Chen.

Ethics declarations

Conflicts of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Xia, R., Yang, K. et al. DGCA: high resolution image inpainting via DR-GAN and contextual attention. Multimed Tools Appl 82, 47751–47771 (2023). https://doi.org/10.1007/s11042-023-15313-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15313-0

Keywords

Search

Navigation