nach oben

Erschienen in:

2020 | OriginalPaper | Buchkapitel

Semantic Image Completion Through an Adversarial Strategy

verfasst von : Patricia Vitoria, Joan Sintes, Coloma Ballester

Erschienen in: Computer Vision, Imaging and Computer Graphics Theory and Applications

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Image completion or image inpainting is the task of filling in missing regions of an image. When those areas are large and the missing information is unique such that the information and redundancy available in the image is not useful to guide the completion, the task becomes even more challenging. This paper proposes an automatic semantic inpainting method able to reconstruct corrupted information of an image by semantically interpreting the image itself. It is based on an adversarial strategy followed by an energy-based completion algorithm. First, the data latent space is learned by training a modified Wasserstein generative adversarial network. Second, the learned semantic information is combined with a novel optimization loss able to recover missing regions conditioned by the available information. Moreover, we present an application in the context of face inpainting, where our method is used to generate a new face by integrating desired facial attributes or expressions from a reference face. This is achieved by slightly modifying the objective energy. Quantitative and qualitative top-tier results show the power and realism of the presented method.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Localization and Grading of Building Roof Damages in High-Resolution Aerial Images

Nächstes Kapitel An Enhanced Louvain Based Image Segmentation Approach Using Color Properties and Histogram of Oriented Gradients

Adler, J., Lunz, S.: Banach wasserstein gan. In: Advances in Neural Information Processing Systems, pp. 6754–6763 (2018)

Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. arXiv:1701.07875 (2017)

Aujol, J.F., Ladjal, S., Masnou, S.: Exemplar-based inpainting from a variational point of view. SIAM J. Math. Anal. 42(3), 1246–1285 (2010)MathSciNetCrossRef

Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv:1607.06450 (2016)

Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, pp. 417–424. ACM Press/Addison-Wesley Publishing Co., New York (2000). https://doi.org/10.1145/344779.344972

Burlin, C., Le Calonnec, Y., Duperier, L.: Deep image inpainting (2017)

Cao, F., Gousseau, Y., Masnou, S., Pérez, P.: Geometrically guided exemplar-based inpainting. SIAM J. Imaging Sci. 4(4), 1143–1179 (2011)MathSciNetCrossRef

Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised diverse colorization via generative adversarial networks. In: Ceci, M., Hollmén, J., Todorovski, L., Vens, C., Džeroski, S. (eds.) ECML PKDD 2017, Part I. LNCS (LNAI), vol. 10534, pp. 151–166. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71249-9_10CrossRef

Chan, T., Shen, J.H.: Mathematical models for local nontexture inpaintings. SIAM J. Appl. Math. 62(3), 1019–1043 (2001)MathSciNetMATH

10.

Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based inpainting. IEEE Trans. IP 13(9), 1200–1212 (2004)

11.

Demir, U., Unal, G.: Patch-based image inpainting with generative adversarial networks. arXiv preprint arXiv:1803.07422 (2018)

12.

Deng, Y., et al.: Graph laplace for occluded face completion and recognition. IEEE Trans. IP 20(8), 2329–2338 (2011)MathSciNetMATH

13.

Fedorov, V., Arias, P., Facciolo, G., Ballester, C.: Affine invariant self-similarity for exemplar-based inpainting. In: Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pp. 48–58 (2016)

14.

Fedorov, V., Facciolo, G., Arias, P.: Variational framework for non-local inpainting. Image Process. Line 5, 362–386 (2015). https://doi.org/10.5201/ipol.2015.136MathSciNetCrossRef

15.

Getreuer, P.: Total variation inpainting using split bregman. Image Process. Line 2, 147–157 (2012). https://doi.org/10.5201/ipol.2012.g-tviCrossRef

16.

Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

17.

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: Advances in Neural Information Processing Systems, pp. 5769–5779 (2017)

18.

Hwang, B.W., Lee, S.W.: Reconstruction of partially damaged face images based on a morphable face model. IEEE TPAMI 25(3), 365–372 (2003) CrossRef

19.

Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4), 107:1–107:14 (2017). https://doi.org/10.1145/3072959.3073659. http://doi.acm.org/10.1145/3072959.3073659CrossRef

20.

Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

21.

Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2, p. 4 (2017)

22.

Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: CVPR, vol. 1, p. 3 (2017)

23.

Lin, D., Tang, X.: Quality-driven face occlusion detection and recovery. In: IEEE CVPR

24.

Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: ICCV (2015)

25.

Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: ICCV, pp. 2813–2821 (2017)

26.

Masnou, S., Morel, J.M.: Level lines based disocclusion. In: Proceedings of IEEE ICIP (1998)

27.

Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning

28.

Nguyen, A., Yosinski, J., Bengio, Y., Dosovitskiy, A., Clune, J.: Plug & play generative networks: conditional iterative generation of images in latent space. arXiv:1612.00005 (2016)

29.

Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: CVPR, June 2016

30.

Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH 2003 (2003)

31.

Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434 (2015)

32.

Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text to image synthesis. In: Proceedings of The 33rd International Conference on Machine Learning, pp. 1060–1069 (2016)

33.

Song, Y., et al.: Contextual-based image inpainting: infer, match, and translate. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018, Part II. LNCS, vol. 11206, pp. 3–18. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_1CrossRef

34.

Vitoria., P., Sintes., J., Ballester., C.: Semantic image inpainting through improved wasserstein generative adversarial networks. In: Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, pp. 249–260. INSTICC, SciTePress (2019). https://doi.org/10.5220/0007367902490260

35.

Vo, H.V., et al.: Structural inpainting. In: 2018 ACM Multimedia Conference (2018)

36.

Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. IP 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861CrossRef

37.

Wang, N., et al.: A comprehensive survey to face hallucination. Int. J. CV 106(1), 9–30 (2014)

38.

Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: CVPR, vol. 1, p. 3 (2017)

39.

Yeh, R.A., Chen, C., Lim, T.Y., Schwing, A.G., Hasegawa-Johnson, M., Do, M.N.: Semantic image inpainting with deep generative models. In: CVPR, vol. 2, p. 4 (2017)

40.

Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention

Titel: Semantic Image Completion Through an Adversarial Strategy
verfasst von: Patricia Vitoria
Joan Sintes
Coloma Ballester
Verlag: Springer International Publishing
Buch: Computer Vision, Imaging and Computer Graphics Theory and Applications
Print ISBN: 978-3-030-41589-1

Electronic ISBN: 978-3-030-41590-7

Copyright-Jahr: 2020
DOI: https://doi.org/10.1007/978-3-030-41590-7_22

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"