Skip to main content
Top
Published in: Neural Processing Letters 7/2023

08-03-2023

Lightweight Image Super-Resolution with ConvNeXt Residual Network

Authors: Yong Zhang, Haomou Bai, Yaxing Bing, Xiao Liang

Published in: Neural Processing Letters | Issue 7/2023

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Single image super-resolution based on convolutional neural networks has been very successful in recent years. However, as the computational cost is too high, making it difficult to apply to resource-constrained devices, a big challenge for existing approaches is to find a balance between the complexity of the CNN model and the quality of the resulting SR. To solve this problem, various lightweight SR networks have been proposed. In this paper, we propose lightweight and efficient residual networks (IRN), which differ from previous lightweight SR networks that aggregate more powerful features by improving feature utilization through complex layer-connection strategies. The main idea is to simplify feature aggregation by using simple and efficient residual modules for feature learning, thus achieving a good trade-off between the computational cost of the model and the quality of the resulting SR. In addition, we revisit the impact of the activation function in the model and observe that different activation functions have an impact on the performance of the model. The experiment results show that IRN outperforms previous state-of-the-art methods in benchmark tests while maintaining a relatively low computational cost. The code will be available at https://​github.​com/​kptx666/​IRN.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European conference on computer vision (ECCV). 184–199 Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European conference on computer vision (ECCV). 184–199
2.
go back to reference Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: international conference for learning representations (ICLR) Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: international conference for learning representations (ICLR)
3.
go back to reference Kim J, Lee JK, Lee KM (2016) Accurate image superresolution using very deep convolutional networks. In: IEEE conference on computer vision and pattern recognition (CVPR). 1646–1654 Kim J, Lee JK, Lee KM (2016) Accurate image superresolution using very deep convolutional networks. In: IEEE conference on computer vision and pattern recognition (CVPR). 1646–1654
4.
go back to reference Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR). 1637–1645 Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR). 1637–1645
5.
go back to reference Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: CVPR Workshops. IEEE Computer Society, pp 1132–1140 Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: CVPR Workshops. IEEE Computer Society, pp 1132–1140
6.
go back to reference Ahn N, Kang B, Sohn K (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In: ECCV (10). Lecture Notes in Computer Science, vol. 11214, pp 256–272. Springer Ahn N, Kang B, Sohn K (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In: ECCV (10). Lecture Notes in Computer Science, vol. 11214, pp 256–272. Springer
7.
go back to reference Hui Z, Wang X, Gao X (2018) Fast and accurate single image super-resolution via information distillation network. In: IEEE conference on computer vision and pattern recognition (CVPR). 723–731 Hui Z, Wang X, Gao X (2018) Fast and accurate single image super-resolution via information distillation network. In: IEEE conference on computer vision and pattern recognition (CVPR). 723–731
8.
go back to reference Hui Z, Gao X, Yang Y, Wang X (2019) Lightweight image super-resolution with information multi-distillation network. In: ACM multimedia. ACM pp 2024–2032 Hui Z, Gao X, Yang Y, Wang X (2019) Lightweight image super-resolution with information multi-distillation network. In: ACM multimedia. ACM pp 2024–2032
9.
go back to reference Li W, Zhou K, Qi L, Jiang N, Lu J, Jia J (2020) Lapar: linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond. Adv Neural Inf Process Syst 33:20343–20355 Li W, Zhou K, Qi L, Jiang N, Lu J, Jia J (2020) Lapar: linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond. Adv Neural Inf Process Syst 33:20343–20355
10.
go back to reference Chu X, Zhang B, Ma H, Xu R, Li Q (2021) Fast, accurate and lightweight super-resolution with neural architecture search. In: 2020 25th international conference on pattern recognition (ICPR). IEEE. (pp 59–64) Chu X, Zhang B, Ma H, Xu R, Li Q (2021) Fast, accurate and lightweight super-resolution with neural architecture search. In: 2020 25th international conference on pattern recognition (ICPR). IEEE. (pp 59–64)
11.
go back to reference Liu Z, Mao H, Wu CY, Feichtenhofer C, Darrell T, Xie S (2022) A convnet for the 2020s. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 11976–11986) Liu Z, Mao H, Wu CY, Feichtenhofer C, Darrell T, Xie S (2022) A convnet for the 2020s. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 11976–11986)
12.
go back to reference Liu J, Tang J, Wu G (2020) Residual feature distillation network for lightweight image super-resolution. In: European conference on computer vision. Springer, Cham. (pp 41–55) Liu J, Tang J, Wu G (2020) Residual feature distillation network for lightweight image super-resolution. In: European conference on computer vision. Springer, Cham. (pp 41–55)
13.
go back to reference Dong C, Loy CC, Tang X (2016) Accelerating the superresolution convolutional neural network. In: European conference on computer vision (ECCV). 391–407 Dong C, Loy CC, Tang X (2016) Accelerating the superresolution convolutional neural network. In: European conference on computer vision (ECCV). 391–407
14.
go back to reference Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017_ Deep laplacian pyramid networks for fast and accurate super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR). 624–632 Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017_ Deep laplacian pyramid networks for fast and accurate super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR). 624–632
15.
go back to reference Lai W-S, Huang J-B, Ahuja N, Yang M-H (2018) Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Lai W-S, Huang J-B, Ahuja N, Yang M-H (2018) Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence
16.
go back to reference Wang Y, Perazzi F, McWilliams B, Sorkine-Hornung A, Sorkin-Hornung O, Schroers C (2018) A fully progressive approach to single-image super-resolution. In: IEEE conference on computer vision and pattern recognition workshop (CVPRW). 977–986 Wang Y, Perazzi F, McWilliams B, Sorkine-Hornung A, Sorkin-Hornung O, Schroers C (2018) A fully progressive approach to single-image super-resolution. In: IEEE conference on computer vision and pattern recognition workshop (CVPRW). 977–986
17.
go back to reference Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR. IEEE Computer Society pp 1874–1883 Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR. IEEE Computer Society pp 1874–1883
18.
go back to reference He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition (CVPR). 770–778 He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition (CVPR). 770–778
19.
go back to reference Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2020) Residual dense network for image restoration. IEEE Trans Pattern Anal Mach Intell 43(7):2480–2495CrossRef Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2020) Residual dense network for image restoration. IEEE Trans Pattern Anal Mach Intell 43(7):2480–2495CrossRef
20.
go back to reference Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE conference on computer vision and pattern recognition (CVPR). 4700–4708 Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE conference on computer vision and pattern recognition (CVPR). 4700–4708
21.
go back to reference Tong T, Li G, Liu X, Gao Q (2017) Image super-resolution using dense skip connections. In: IEEE international conference on computer vision (ICCV). 4799–4807 Tong T, Li G, Liu X, Gao Q (2017) Image super-resolution using dense skip connections. In: IEEE international conference on computer vision (ICCV). 4799–4807
22.
go back to reference Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: IEEE conference on computer vision and pattern recognition (CVPR). 7132–7141 Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: IEEE conference on computer vision and pattern recognition (CVPR). 7132–7141
23.
go back to reference Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: European conference on computer vision (ECCV). 286–301 Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: European conference on computer vision (ECCV). 286–301
24.
go back to reference Zhang Y, Li K, Li K, Zhong B, Fu Y (2019) Residual non-local attention networks for image restoration. In: international conference on learning representations (ICLR) Zhang Y, Li K, Li K, Zhong B, Fu Y (2019) Residual non-local attention networks for image restoration. In: international conference on learning representations (ICLR)
25.
go back to reference Guo Y, Chen J, Wang J, Chen Q, Cao J, Deng Z, Xu Y, Tan M (2020) Closed-loop matters: dual regression networks for single image super-resolution. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 5407–5416) Guo Y, Chen J, Wang J, Chen Q, Cao J, Deng Z, Xu Y, Tan M (2020) Closed-loop matters: dual regression networks for single image super-resolution. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 5407–5416)
26.
go back to reference Liang J, Cao J, Sun G, Zhang K, Van Gool L, Timofte R (2021) Swinir: image restoration using swin transformer. In: proceedings ofthe IEEE/CVF international conference on computer vision, pp 1833–1844 Liang J, Cao J, Sun G, Zhang K, Van Gool L, Timofte R (2021) Swinir: image restoration using swin transformer. In: proceedings ofthe IEEE/CVF international conference on computer vision, pp 1833–1844
27.
go back to reference Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: proceedings of the IEEE/CVF international conference on computer vision, pp 10012–10022 Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: proceedings of the IEEE/CVF international conference on computer vision, pp 10012–10022
28.
29.
go back to reference Mehta N, Murala S (2021) MSAR-Net: multi-scale attention based light-weight image super-resolution. Pattern Recogn Lett 151:215–221CrossRef Mehta N, Murala S (2021) MSAR-Net: multi-scale attention based light-weight image super-resolution. Pattern Recogn Lett 151:215–221CrossRef
30.
go back to reference Wang Q, Wu B, Zhu P, Li P, Zuo W, Hu Q (2020) Supplementary material for ‘ECA-Net: efficient channel attention for deep convolutional neural networks. In: proceedings of the 2020 IEEE/CVF conference on computer vision and pattern recognition, IEEE, Seattle, WA, USA (pp 13–19) Wang Q, Wu B, Zhu P, Li P, Zuo W, Hu Q (2020) Supplementary material for ‘ECA-Net: efficient channel attention for deep convolutional neural networks. In: proceedings of the 2020 IEEE/CVF conference on computer vision and pattern recognition, IEEE, Seattle, WA, USA (pp 13–19)
31.
go back to reference Woo S, Park J, Lee J-Y, Kweon IS (2018) CBAM: convolutional block attention module. In: the European conference on computer vision (ECCV). 3–19 Woo S, Park J, Lee J-Y, Kweon IS (2018) CBAM: convolutional block attention module. In: the European conference on computer vision (ECCV). 3–19
32.
go back to reference Dai T, Cai J, Zhang Y, Xia ST, Zhang L (2019). Second-order attention network for single image super-resolution. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 11065–11074) Dai T, Cai J, Zhang Y, Xia ST, Zhang L (2019). Second-order attention network for single image super-resolution. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 11065–11074)
33.
go back to reference Wang X, Girshick R, Gupta A, He K (2018) Non-local Neural Networks. In: IEEE conference on computer vision and pattern recognition (CVPR). 7794–7803 Wang X, Girshick R, Gupta A, He K (2018) Non-local Neural Networks. In: IEEE conference on computer vision and pattern recognition (CVPR). 7794–7803
35.
go back to reference Yu J, Fan Y, Yang J, Xu N, Wang Z, Wang X, Huang T (2018) Wide activation for efficient and accurate image super-resolution. arXiv preprint arXiv:1808.08718 Yu J, Fan Y, Yang J, Xu N, Wang Z, Wang X, Huang T (2018) Wide activation for efficient and accurate image super-resolution. arXiv preprint arXiv:​1808.​08718
36.
go back to reference Liu J, Zhang W, Tang Y, Tang J, Wu G (2020) Residual feature aggregation network for image superresolution. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2359–2368 Liu J, Zhang W, Tang Y, Tang J, Wu G (2020) Residual feature aggregation network for image superresolution. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2359–2368
37.
go back to reference Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: dataset and study. In: proceedings ofthe IEEE conference on computer vision and pattern recognition workshops, pp 126–135 Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: dataset and study. In: proceedings ofthe IEEE conference on computer vision and pattern recognition workshops, pp 126–135
38.
go back to reference Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel ML (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 2012. 5 Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel ML (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 2012. 5
39.
go back to reference Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: international conference on curves and surfaces, pp 711–730. Springer, 2010. 5 Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: international conference on curves and surfaces, pp 711–730. Springer, 2010. 5
40.
go back to reference Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: proceedings Eighth IEEE international conference on computer vision. ICCV 2001, IEEE, vol. 2, pp 416–423 Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: proceedings Eighth IEEE international conference on computer vision. ICCV 2001, IEEE, vol. 2, pp 416–423
41.
go back to reference Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: proceedings ofthe IEEE conference on computer vision and pattern recognition, pp 5197–5206 Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: proceedings ofthe IEEE conference on computer vision and pattern recognition, pp 5197–5206
42.
go back to reference Matsui Y, Ito K, Aramaki Y, Fujimoto A, Ogawa T, Yamasaki T, Aizawa K (2017) Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications 76(20):21811–21838CrossRef Matsui Y, Ito K, Aramaki Y, Fujimoto A, Ogawa T, Yamasaki T, Aizawa K (2017) Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications 76(20):21811–21838CrossRef
43.
go back to reference Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612CrossRef Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612CrossRef
45.
go back to reference Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In Icml 4, 7 Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In Icml 4, 7
46.
go back to reference Maas AL, Hannun AY, Ng AY et al. (2013) Rectifier nonlinearities improve neural network acoustic models. In: Proc. icml, Citeseer. vol. 30, p 3 Maas AL, Hannun AY, Ng AY et al. (2013) Rectifier nonlinearities improve neural network acoustic models. In: Proc. icml, Citeseer. vol. 30, p 3
47.
go back to reference Tai Y, Yang J, Liu X, Xu C (2017) MemNet: a persistent memory network for image restoration. In: IEEE international conference on computer vision (ICCV). 4539–4547 Tai Y, Yang J, Liu X, Xu C (2017) MemNet: a persistent memory network for image restoration. In: IEEE international conference on computer vision (ICCV). 4539–4547
48.
go back to reference Tai Y, Yang J, Liu X (2017) Image superresolution via deep recursive residual network. In: proceedings ofthe IEEE conference on computer vision and pattern recognition, pp 3147–3155 Tai Y, Yang J, Liu X (2017) Image superresolution via deep recursive residual network. In: proceedings ofthe IEEE conference on computer vision and pattern recognition, pp 3147–3155
49.
go back to reference Choi J-S, Kim M (2017) A deep convolutional neural network with selection units for super-resolution. In: CVPRW, pp 154–160 Choi J-S, Kim M (2017) A deep convolutional neural network with selection units for super-resolution. In: CVPRW, pp 154–160
50.
go back to reference Zhang K, Zuo W, Zhang L (2018) Learning a single convolutional super-resolution network for multiple degradations. In: CVPR, pp 3262–3271 Zhang K, Zuo W, Zhang L (2018) Learning a single convolutional super-resolution network for multiple degradations. In: CVPR, pp 3262–3271
51.
go back to reference Li Z, Yang J, Liu Z, Yang X, Jeon G, Wu W (2019) Feedback network for image super-resolution. In: CVPR, pp 3867–3876 Li Z, Yang J, Liu Z, Yang X, Jeon G, Wu W (2019) Feedback network for image super-resolution. In: CVPR, pp 3867–3876
52.
go back to reference Mehta N, Murala S (2022) Con-Net: a consolidated light-weight image restoration network. IEEE Trans Broadcast 68(4):862–875CrossRef Mehta N, Murala S (2022) Con-Net: a consolidated light-weight image restoration network. IEEE Trans Broadcast 68(4):862–875CrossRef
53.
go back to reference Zhang X, Zeng H, Zhang L (2021) Edge-oriented convolution block for real-time super resolution on mobile devices. In: Shen HT, Zhuang Y, Smith JR, Yang Y, Cesar P, Metze F, Prabhakaran B (eds), MM ’21: ACM multimedia conference, Virtual Event, China, ACM. October 20–24, 2021, pp 4034–4043 Zhang X, Zeng H, Zhang L (2021) Edge-oriented convolution block for real-time super resolution on mobile devices. In: Shen HT, Zhuang Y, Smith JR, Yang Y, Cesar P, Metze F, Prabhakaran B (eds), MM ’21: ACM multimedia conference, Virtual Event, China, ACM. October 20–24, 2021, pp 4034–4043
54.
go back to reference Guangwei G et al (2022) "Feature distillation interaction weighting network for lightweight image super-resolution." Proceedings of the AAAI conference on artificial intelligence. Vol. 36(1) Guangwei G et al (2022) "Feature distillation interaction weighting network for lightweight image super-resolution." Proceedings of the AAAI conference on artificial intelligence. Vol. 36(1)
Metadata
Title
Lightweight Image Super-Resolution with ConvNeXt Residual Network
Authors
Yong Zhang
Haomou Bai
Yaxing Bing
Xiao Liang
Publication date
08-03-2023
Publisher
Springer US
Published in
Neural Processing Letters / Issue 7/2023
Print ISSN: 1370-4621
Electronic ISSN: 1573-773X
DOI
https://doi.org/10.1007/s11063-023-11213-4

Other articles of this Issue 7/2023

Neural Processing Letters 7/2023 Go to the issue