Abstract
Pooling operations for convolutional neural networks provide the opportunity to greatly reduce network parameters, leading to faster training time and less data overfitting. Unfortunately, many of the common pooling methods such as max pooling and mean pooling lose information about the data (i.e., they are lossy methods). Recently, spectral pooling has been utilized to pool data in the spectral domain. By doing so, greater information can be retained with the same network parameter reduction as spatial pooling. Spectral pooling is currently implemented in the discrete Fourier domain, but it is found that implementing spectral pooling in the discrete cosine domain concentrates energy in even fewer spectra. Although Discrete Cosine Transforms Spectral Pooling Layers (DCTSPL) require extra computation compared to normal spectral pooling, the overall time complexity does not change and, furthermore, greater information preservation is obtained, producing networks which converge faster and achieve a lower misclassification error.
This work was supported by the National Science Centre, Krakow, Poland, under grant No. 2015/17/B/ST6/01880.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cooley, J.W., Tukey, J.W.: An algorithm for the machine calculation of complex Fourier series. Math. Comput. 19(90), 297–301 (1965)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Highlander, T., Rodriguez, A.: Very efficient training of convolutional neural networks using fast Fourier transform and overlap-and-add. arXiv:1601.06815 [cs], January 2016
Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y.: What is the best multi-stage architecture for object recognition? In: 2009 IEEE 12th International Conference on Computer Vision, pp. 2146–2153, September 2009
Ko, J.H., Mudassar, B., Na, T., Mukhopadhyay, S.: Design of an energy-efficient accelerator for training of convolutional neural networks using frequency-domain computation. In: Proceedings of the 54th Annual Design Automation Conference 2017, DAC 2017, pp. 59:1–59:6. ACM, New York (2017)
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Mathieu, M., Henaff, M., LeCun, Y.: Fast training of convolutional networks through FFTs. arXiv:1312.5851 [cs], December 2013
Olejczak, A., Korniak, J., Wilamowski, B.M.: Discrete cosine transformation as alternative to other methods of computational intelligence for function approximation. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10245, pp. 143–153. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59063-9_13
Rippel, O., Snoek, J., Adams, R.P.: Spectral representations for convolutional neural networks. arXiv:1506.03767 [cs, stat], June 2015
Scherer, D., Müller, A., Behnke, S.: Evaluation of pooling operations in convolutional architectures for object recognition. In: Diamantaras, K., Duch, W., Iliadis, L.S. (eds.) Part III, ICANN 2010. LNCS, vol. 6354, pp. 92–101. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15825-4_10
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Wang, Y., Xu, C., You, S., Tao, D., Xu, C.: CNNpack: packing convolutional neural networks in the frequency domain. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 253–261. Curran Associates, Inc., Red Hook (2016)
Watson, A.B.: Image compression using the discrete cosine transform. Math. J. 4(1), 81–88 (1994)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Smith, J.S., Wilamowski, B.M. (2018). Discrete Cosine Transform Spectral Pooling Layers for Convolutional Neural Networks. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2018. Lecture Notes in Computer Science(), vol 10841. Springer, Cham. https://doi.org/10.1007/978-3-319-91253-0_23
Download citation
DOI: https://doi.org/10.1007/978-3-319-91253-0_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-91252-3
Online ISBN: 978-3-319-91253-0
eBook Packages: Computer ScienceComputer Science (R0)