Skip to main content
Top
Published in: Cluster Computing 2/2019

09-12-2017

Pyramidal RoR for image classification

Authors: Ke Zhang, Liru Guo, Ce Gao, Zhenbing Zhao

Published in: Cluster Computing | Special Issue 2/2019

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

The Residual Networks of Residual Networks (RoR) exhibits excellent performance in the image classification task, but sharply increasing the number of feature map channels makes the characteristic information transmission incoherent, which losses a certain of information related to classification prediction, limiting the classification performance. In this paper, a Pyramidal RoR network model is proposed by analysing the characteristics of RoR and combining with the PyramidNet. Firstly, based on RoR, the Pyramidal RoR network model with channels gradually increasing is designed. Secondly, we analysed the effect of different residual block structures on performance, and chosen the residual block structure which best favoured the classification performance. Finally, we add an important principle to further optimize Pyramidal RoR networks, drop-path is used to avoid over-fitting and save training time. In this paper, image classification experiments were performed on CIFAR-10/100, SVHN and Adience datasets, and we achieved the current lowest classification error rates were 2.96, 16.40 and 1.59% on CIFAR-10/100 and SVHN, respectively. Experiments show that the Pyramidal RoR network optimization method can improve the network performance for image classification and effectively suppress the gradient disappearance problem in DCNN training.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015) LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
2.
go back to reference Zou, W.Y., Wang, X.Y., Sun, M., Lin, Y.: Generic object detection with dense neural patterns and regional. arXiv preprint arXiv:1404.4316 (2014) Zou, W.Y., Wang, X.Y., Sun, M., Lin, Y.: Generic object detection with dense neural patterns and regional. arXiv preprint arXiv:​1404.​4316 (2014)
3.
go back to reference Krizhenvshky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional networks. In: Proceedings of the Advances in Neural Information Processing System, pp. 1097–1105 (2012) Krizhenvshky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional networks. In: Proceedings of the Advances in Neural Information Processing System, pp. 1097–1105 (2012)
4.
go back to reference Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2014) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2014)
5.
go back to reference Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:​1409.​1556 (2014)
6.
go back to reference Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recogniton, pp. 1–9 (2015) Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recogniton, pp. 1–9 (2015)
8.
go back to reference He, K., Sun, J.: Convolutional neural networks at constrained time cost, In: Proceedings of the IEEE Conference on Computer and Vision Pattern Recognition, pp. 5353–5360 (2015) He, K., Sun, J.: Convolutional neural networks at constrained time cost, In: Proceedings of the IEEE Conference on Computer and Vision Pattern Recognition, pp. 5353–5360 (2015)
12.
go back to reference Zhang, K., Sun, M., Han, X., et al.: Residual networks of residual networks: multilevel residual networks. IEEE Trans. Circuit Syst. Video Technol. 99, 1 (2016) Zhang, K., Sun, M., Han, X., et al.: Residual networks of residual networks: multilevel residual networks. IEEE Trans. Circuit Syst. Video Technol. 99, 1 (2016)
13.
go back to reference Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus), arXiv preprint arXiv:1511.07289 (2015) Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus), arXiv preprint arXiv:​1511.​07289 (2015)
14.
go back to reference Trottier, L., Giguere, P., Chaib-draa, B.: Parametric exponential linear unit for deep convolutional neural networks, arXiv preprint arXiv:1605.09322 (2016) Trottier, L., Giguere, P., Chaib-draa, B.: Parametric exponential linear unit for deep convolutional neural networks, arXiv preprint arXiv:​1605.​09322 (2016)
16.
go back to reference Krizhenvshky, A., Hinton, G.: Learning multiple layers of features from tiny images, M.Sc. thesis, Deptartment of Computer Science, University of Toronto, Toronto, ON, Canada (2009) Krizhenvshky, A., Hinton, G.: Learning multiple layers of features from tiny images, M.Sc. thesis, Deptartment of Computer Science, University of Toronto, Toronto, ON, Canada (2009)
17.
go back to reference Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning, In: Proceedings of the NIPS Workshop Deep Learning and Unsupervised Feature Learning, pp. 1–9 (2011) Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning, In: Proceedings of the NIPS Workshop Deep Learning and Unsupervised Feature Learning, pp. 1–9 (2011)
18.
go back to reference Eidinger, E., Enbar, R., Hassner, T.: Age and gender estimation of unfiltered faces[J]. IEEE Trans. Inf. Forensics Secur. 9(12), 2170–2179 (2014) Eidinger, E., Enbar, R., Hassner, T.: Age and gender estimation of unfiltered faces[J]. IEEE Trans. Inf. Forensics Secur. 9(12), 2170–2179 (2014)
19.
go back to reference Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (2014) Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (2014)
21.
go back to reference Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines, In: Proceedings of the International Conference on ICML pp. 807–814 (2010) Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines, In: Proceedings of the International Conference on ICML pp. 807–814 (2010)
22.
go back to reference Han, D., Kim, J., Kim, J.: Deep pyramidal residual networks, In: Proceedings of the International Conference on CVPR (2017) Han, D., Kim, J., Kim, J.: Deep pyramidal residual networks, In: Proceedings of the International Conference on CVPR (2017)
23.
24.
go back to reference Wang, F., Jiang, M., Qian C, et al.: Residual Attention Network for Image Classification, In: Proceedings of the International Conference on CVPR (2017) Wang, F., Jiang, M., Qian C, et al.: Residual Attention Network for Image Classification, In: Proceedings of the International Conference on CVPR (2017)
25.
go back to reference Chen, Y., Li, J., Xiao, H, et al.: Dual Path Networks, In: Proceedings of the International Conference on CVPR (2017) Chen, Y., Li, J., Xiao, H, et al.: Dual Path Networks, In: Proceedings of the International Conference on CVPR (2017)
26.
go back to reference Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:1502.03167 (2015) Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:​1502.​03167 (2015)
27.
go back to reference Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:1502.03167 (2015) Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:​1502.​03167 (2015)
28.
go back to reference Hinton, G., Srivastava, N., Krizhevsky, A., Weinberger, K.: Improving neural networks by preventing co-adaptation of feature detectors, arXiv preprint arXiv:1207.0580 (2012) Hinton, G., Srivastava, N., Krizhevsky, A., Weinberger, K.: Improving neural networks by preventing co-adaptation of feature detectors, arXiv preprint arXiv:​1207.​0580 (2012)
29.
go back to reference Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
32.
go back to reference Larsson, G., Maire, M., Shakhnarovich, G.: FractalNet: ultra-deep neural networks without residuals, arXiv preprint arXiv:1605.07648 (2016) Larsson, G., Maire, M., Shakhnarovich, G.: FractalNet: ultra-deep neural networks without residuals, arXiv preprint arXiv:​1605.​07648 (2016)
33.
34.
36.
go back to reference Xie, S., Girshick, R., Dollr P., et al.: Aggregated residual transformations for deep neural networks, arXiv preprint arXiv:1611.05431 (2016) Xie, S., Girshick, R., Dollr P., et al.: Aggregated residual transformations for deep neural networks, arXiv preprint arXiv:​1611.​05431 (2016)
37.
go back to reference Yamada, Y., Iwamura, M., Kise, K.: Deep pyramidal residual networks with separated stochastic depth[J]. arXiv preprint arXiv:1612.01230 (2016) Yamada, Y., Iwamura, M., Kise, K.: Deep pyramidal residual networks with separated stochastic depth[J]. arXiv preprint arXiv:​1612.​01230 (2016)
Metadata
Title
Pyramidal RoR for image classification
Authors
Ke Zhang
Liru Guo
Ce Gao
Zhenbing Zhao
Publication date
09-12-2017
Publisher
Springer US
Published in
Cluster Computing / Issue Special Issue 2/2019
Print ISSN: 1386-7857
Electronic ISSN: 1573-7543
DOI
https://doi.org/10.1007/s10586-017-1443-x

Other articles of this Special Issue 2/2019

Cluster Computing 2/2019 Go to the issue

Premium Partner