Skip to main content
Erschienen in: Machine Vision and Applications 4/2023

01.07.2023 | Original Paper

ECM: arbitrary style transfer via Enhanced-Channel Module

verfasst von: Xiaoming Yu, Gan Zhou

Erschienen in: Machine Vision and Applications | Ausgabe 4/2023

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Arbitrary style transfer is to fuse the style of one image with the content of another image. With the development of artificial intelligence, many style transfer methods have emerged, which focus on stylizing content features in different aspects, such as loss functions, attention mechanism, and functional instance normalization layer. However, there are still problems such as the lack of local detail information or the insufficient degree of style fusion in the generated images. To get more realistic images, we propose an Enhanced-Channel Module, which vectorizes content feature maps and style feature maps to generate content-aware channel weights. The channel weights are multiplied by the content feature maps as the map offsets to complete the style injection. Meanwhile, in order to utilize the multilayer feature maps, we train the network in a progressive rendering way using multilayer content features and deep style features. The comparison experiments with the state-of-the-art methods prove that the proposed method can generate high-quality stylization results in artistic style transfer and video style transfer tasks, and the ablation experiments prove the effectiveness of the different components.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Gatys, L. A., Ecker, A. S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR, pp. 2414–2423 (2016) Gatys, L. A., Ecker, A. S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR, pp. 2414–2423 (2016)
2.
Zurück zum Zitat Gatys, L., Ecker, A. S., Bethge, M.: Texture synthesis using convolutional neural networks. NeurIPS 28 (2015) Gatys, L., Ecker, A. S., Bethge, M.: Texture synthesis using convolutional neural networks. NeurIPS 28 (2015)
3.
Zurück zum Zitat Ulyanov, D., Lebedev, V., Vedaldi, A., Lempitsky, V.: Texture networks: feed-forward synthesis of textures and stylized images. arXiv:1603.03417 (2016) Ulyanov, D., Lebedev, V., Vedaldi, A., Lempitsky, V.: Texture networks: feed-forward synthesis of textures and stylized images. arXiv:​1603.​03417 (2016)
4.
Zurück zum Zitat Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. arXiv:1607.08022 (2016) Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. arXiv:​1607.​08022 (2016)
5.
Zurück zum Zitat Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. In: CVPR, pp. 6924–6932 (2017) Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. In: CVPR, pp. 6924–6932 (2017)
6.
Zurück zum Zitat Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Stylebank: an explicit representation for neural image style transfer. In: CVPR, pp. 1897–1906 (2017) Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Stylebank: an explicit representation for neural image style transfer. In: CVPR, pp. 1897–1906 (2017)
7.
Zurück zum Zitat Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.-H.: Diversified texture synthesis with feed-forward networks. In: CVPR, pp. 3920–3928 (2017) Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.-H.: Diversified texture synthesis with feed-forward networks. In: CVPR, pp. 3920–3928 (2017)
8.
Zurück zum Zitat Zhang, H., Dana, K.: Multi-style generative network for real-time transfer. In: ECCV Workshops (2018) Zhang, H., Dana, K.: Multi-style generative network for real-time transfer. In: ECCV Workshops (2018)
9.
Zurück zum Zitat Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV, pp. 1501–1510 (2017) Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV, pp. 1501–1510 (2017)
10.
Zurück zum Zitat Park, D.Y., Lee, K.H.: Arbitrary style transfer with style-attentional networks. In: CVPR, pp. 5880–5888 (2019) Park, D.Y., Lee, K.H.: Arbitrary style transfer with style-attentional networks. In: CVPR, pp. 5880–5888 (2019)
11.
Zurück zum Zitat Li, X., Liu, S., Kautz, J., Yang, M.-H.: Learning linear transformations for fast image and video style transfer. In: CVPR, pp. 3809–3817 (2019) Li, X., Liu, S., Kautz, J., Yang, M.-H.: Learning linear transformations for fast image and video style transfer. In: CVPR, pp. 3809–3817 (2019)
12.
Zurück zum Zitat Deng, Y., Tang, F., Dong, W., Sun, W., Huang, F., Xu, C.: Arbitrary style transfer via multi-adaptation network. In: ACM MM, pp. 2719–2727 (2020) Deng, Y., Tang, F., Dong, W., Sun, W., Huang, F., Xu, C.: Arbitrary style transfer via multi-adaptation network. In: ACM MM, pp. 2719–2727 (2020)
13.
Zurück zum Zitat Deng, Y., Tang, F., Dong, W., Huang, H., Ma, C., Xu, C.: Arbitrary video style transfer via multi-channel correlation. In: AAAI, vol. 35, pp. 1210–1217 (2021) Deng, Y., Tang, F., Dong, W., Huang, H., Ma, C., Xu, C.: Arbitrary video style transfer via multi-channel correlation. In: AAAI, vol. 35, pp. 1210–1217 (2021)
14.
Zurück zum Zitat Chen, H., Wang, Z., Zhang, H., Zuo, Z., Li, A., Xing, W., Lu, D.: Artistic style transfer with internal-external learning and contrastive learning. NeurIPS 34, 26561–26573 (2021) Chen, H., Wang, Z., Zhang, H., Zuo, Z., Li, A., Xing, W., Lu, D.: Artistic style transfer with internal-external learning and contrastive learning. NeurIPS 34, 26561–26573 (2021)
15.
Zurück zum Zitat Zhang, Y., Tang, F., Dong, W., Huang, H., Ma, C., Lee, T.-Y., Xu, C.: Domain enhanced arbitrary image style transfer via contrastive learning. In: ACM SIGGRAPH (2022) Zhang, Y., Tang, F., Dong, W., Huang, H., Ma, C., Lee, T.-Y., Xu, C.: Domain enhanced arbitrary image style transfer via contrastive learning. In: ACM SIGGRAPH (2022)
16.
Zurück zum Zitat Wu, Z., Zhu, Z., Du, J., Bai, X.: CCPL: contrastive coherence preserving loss for versatile style transfer. In: ECCV, pp. 189–206. Springer (2022) Wu, Z., Zhu, Z., Du, J., Bai, X.: CCPL: contrastive coherence preserving loss for versatile style transfer. In: ECCV, pp. 189–206. Springer (2022)
17.
Zurück zum Zitat Xu, W., Long, C., Wang, R., Wang, G.: Drb-gan: A dynamic resblock generative adversarial network for artistic style transfer. In: ICCV, pp. 6383–6392 (2021) Xu, W., Long, C., Wang, R., Wang, G.: Drb-gan: A dynamic resblock generative adversarial network for artistic style transfer. In: ICCV, pp. 6383–6392 (2021)
18.
Zurück zum Zitat Kolkin, N., Salavon, J., Shakhnarovich, G.: Style transfer by relaxed optimal transport and self-similarity. In: CVPR, pp. 10051–10060 (2019) Kolkin, N., Salavon, J., Shakhnarovich, G.: Style transfer by relaxed optimal transport and self-similarity. In: CVPR, pp. 10051–10060 (2019)
20.
Zurück zum Zitat Wang, X., Oxholm, G., Zhang, D., Wang, Y.-F.: Multimodal transfer: a hierarchical deep convolutional neural network for fast artistic style transfer. In: CVPR, pp. 5239–5247 (2017) Wang, X., Oxholm, G., Zhang, D., Wang, Y.-F.: Multimodal transfer: a hierarchical deep convolutional neural network for fast artistic style transfer. In: CVPR, pp. 5239–5247 (2017)
21.
Zurück zum Zitat Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV, pp. 694–711. Springer (2016) Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV, pp. 694–711. Springer (2016)
22.
Zurück zum Zitat Li, C., Wand, M.: Precomputed real-time texture synthesis with markovian generative adversarial networks. In: ECCV, pp. 702–716. Springer (2016) Li, C., Wand, M.: Precomputed real-time texture synthesis with markovian generative adversarial networks. In: ECCV, pp. 702–716. Springer (2016)
23.
Zurück zum Zitat Lin, T., Ma, Z., Li, F., He, D., Li, X., Ding, E., Wang, N., Li, J., Gao, X.: Drafting and revision: Laplacian pyramid network for fast high-quality artistic style transfer. In: CVPR, pp. 5141–5150 (2021) Lin, T., Ma, Z., Li, F., He, D., Li, X., Ding, E., Wang, N., Li, J., Gao, X.: Drafting and revision: Laplacian pyramid network for fast high-quality artistic style transfer. In: CVPR, pp. 5141–5150 (2021)
24.
Zurück zum Zitat Jing, Y., Liu, X., Ding, Y., Wang, X., Ding, E., Song, M., Wen, S.: Dynamic instance normalization for arbitrary style transfer. In: AAAI, vol. 34, pp. 4369–4376 (2020) Jing, Y., Liu, X., Ding, Y., Wang, X., Ding, E., Song, M., Wen, S.: Dynamic instance normalization for arbitrary style transfer. In: AAAI, vol. 34, pp. 4369–4376 (2020)
25.
Zurück zum Zitat Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.-H.: Universal style transfer via feature transforms. NeurIPS 30 (2017) Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.-H.: Universal style transfer via feature transforms. NeurIPS 30 (2017)
27.
Zurück zum Zitat Gu, S., Chen, C., Liao, J., Yuan, L.: Arbitrary style transfer with deep feature reshuffle. In: CVPR, pp. 8222–8231 (2018) Gu, S., Chen, C., Liao, J., Yuan, L.: Arbitrary style transfer with deep feature reshuffle. In: CVPR, pp. 8222–8231 (2018)
28.
Zurück zum Zitat Sheng, L., Lin, Z., Shao, J., Wang, X.: Avatar-net: multi-scale zero-shot style transfer by feature decoration. In: CVPR, pp. 8242–8250 (2018) Sheng, L., Lin, Z., Shao, J., Wang, X.: Avatar-net: multi-scale zero-shot style transfer by feature decoration. In: CVPR, pp. 8242–8250 (2018)
29.
Zurück zum Zitat Wu, Z., Song, C., Zhou, Y., Gong, M., Huang, H.: Efanet: exchangeable feature alignment network for arbitrary style transfer. In: AAAI, vol. 34, pp. 12305–12312 (2020) Wu, Z., Song, C., Zhou, Y., Gong, M., Huang, H.: Efanet: exchangeable feature alignment network for arbitrary style transfer. In: AAAI, vol. 34, pp. 12305–12312 (2020)
30.
Zurück zum Zitat Huo, J., Jin, S., Li, W., Wu, J., Lai, Y.-K., Shi, Y., Gao, Y.: Manifold alignment for semantically aligned style transfer. In: ICCV, pp. 14861–14869 (2021) Huo, J., Jin, S., Li, W., Wu, J., Lai, Y.-K., Shi, Y., Gao, Y.: Manifold alignment for semantically aligned style transfer. In: ICCV, pp. 14861–14869 (2021)
31.
Zurück zum Zitat Wang, Z., Zhao, L., Chen, H., Qiu, L., Mo, Q., Lin, S., Xing, W., Lu, D.: Diversified arbitrary style transfer via deep feature perturbation. In: CVPR, pp. 7789–7798 (2020) Wang, Z., Zhao, L., Chen, H., Qiu, L., Mo, Q., Lin, S., Xing, W., Lu, D.: Diversified arbitrary style transfer via deep feature perturbation. In: CVPR, pp. 7789–7798 (2020)
32.
33.
Zurück zum Zitat Saito, K., Saenko, K., Liu, M.-Y.: Coco-funit: few-shot unsupervised image translation with a content conditioned style encoder. In: ECCV, pp. 382–398. Springer (2020) Saito, K., Saenko, K., Liu, M.-Y.: Coco-funit: few-shot unsupervised image translation with a content conditioned style encoder. In: ECCV, pp. 382–398. Springer (2020)
34.
Zurück zum Zitat Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: ECCV, pp. 740–755. Springer (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: ECCV, pp. 740–755. Springer (2014)
35.
Zurück zum Zitat Phillips, F., Mackintosh, B.: Wiki art gallery, Inc.: a case for critical thinking. Issues Account. Educ. 26(3), 593–608 (2011) Phillips, F., Mackintosh, B.: Wiki art gallery, Inc.: a case for critical thinking. Issues Account. Educ. 26(3), 593–608 (2011)
36.
Zurück zum Zitat Yao, Y., Ren, J., Xie, X., Liu, W., Liu, Y.-J., Wang, J.: Attention-aware multi-stroke style transfer. In: CVPR, pp. 1467–1475 (2019) Yao, Y., Ren, J., Xie, X., Liu, W., Liu, Y.-J., Wang, J.: Attention-aware multi-stroke style transfer. In: CVPR, pp. 1467–1475 (2019)
37.
Zurück zum Zitat Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018) Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018)
38.
Zurück zum Zitat Chang, H.-Y., Wang, Z., Chuang, Y.-Y.: Domain-specific mappings for generative adversarial style transfer. In: ECCV, pp. 573–589. Springer (2020) Chang, H.-Y., Wang, Z., Chuang, Y.-Y.: Domain-specific mappings for generative adversarial style transfer. In: ECCV, pp. 573–589. Springer (2020)
39.
Zurück zum Zitat Yu, X., Chen, Y., Liu, S., Li, T., Li, G.: Multi-mapping image-to-image translation via learning disentanglement. NeurIPS 32 (2019) Yu, X., Chen, Y., Liu, S., Li, T., Li, G.: Multi-mapping image-to-image translation via learning disentanglement. NeurIPS 32 (2019)
Metadaten
Titel
ECM: arbitrary style transfer via Enhanced-Channel Module
verfasst von
Xiaoming Yu
Gan Zhou
Publikationsdatum
01.07.2023
Verlag
Springer Berlin Heidelberg
Erschienen in
Machine Vision and Applications / Ausgabe 4/2023
Print ISSN: 0932-8092
Elektronische ISSN: 1432-1769
DOI
https://doi.org/10.1007/s00138-023-01428-9

Weitere Artikel der Ausgabe 4/2023

Machine Vision and Applications 4/2023 Zur Ausgabe

Premium Partner