Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Arabian Journal for Science and Engineering
Published in: Arabian Journal for Science and Engineering 2/2022

25-06-2021 | Research Article-Computer Engineering and Computer Science

Hand Gesture Recognition from 2D Images by Using Convolutional Capsule Neural Networks

Authors: Osman Güler, İbrahim Yücedağ

Published in: Arabian Journal for Science and Engineering | Issue 2/2022

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Object classification and recognition are an important research area widely used in computer vision and machine learning. With the use of deep learning methods in the field of object recognition, there have been important developments in recent years. Object recognition and its sub-branches face recognition, motion recognition, and hand gesture recognition are now used effectively in devices used in daily life. Hand sign classification and recognition are an area that researchers are working on and trying to develop for human–computer interaction. In this study, a hybrid model was created by using a capsule network algorithm with a convolutional neural network for object classification. A dataset, named HG14, containing 14 different hand gestures was created. To measure the success of the proposed model in object recognition, training was carried out on HG14, FashionMnist, and Cifar-10 datasets. Also, VGG16, ResNet50, DenseNet, and CapsNet models were used to classify the images in HG14, FashionMnist, and Cifar-10 datasets. The results of the training were compared and evaluated. The proposed hybrid model achieved the highest accuracy rates with 90% in the HG14 dataset, 93.88% in the FashionMnist dataset, and 81.42% in the Cifar-10 dataset. The proposed model was found to be successful in all studies compared to other models.
previous article Design of Backpropagated Intelligent Networks for Nonlinear Second-Order Lane–Emden Pantograph Delay Differential Systems
next article Impact of Propagation Path Loss by Varying BTS Height and Frequency for Combining Multiple Path Loss Approaches in Macro-Femto Environment

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now
Literature
1.
Maung, T.H.H.: Real-time hand tracking and gesture recognition system using neural networks. Proc. World Acad. Sci.: Eng. Technol. 50, 466–470 (2009)
2.
Trigueiros, P.; Ribeiro, F.; Reis, L.P.: Hand gesture recognition system based in computer vision and machine learning. In: Developments in medical ımage processing and computational vision, pp. 355–377. Springer, Cham (2015)CrossRef
3.
Limonchik, B.; Amdur, G.: 3D model-based data augmentation for hand gesture recognition (2017)
4.
Sarkar, A.; Gepperth, A.; Handmann, U.; Kopinski, T.: Dynamic hand gesture recognition for mobile systems using deep lstm. In: International conference on ıntelligent human computer ınteraction, pp. 19–31. Springer, Cham (2017)CrossRef
5.
Shastry, K. R.; Ravindran, M.; Srikanth, M. V. V. N. S.; Lakshmikhanth, N.: Survey on various gesture recognition techniques for interfacing machines based on ambient intelligence (2010). arXiv preprint arXiv:​1012.​0084.
6.
Pala T.; Yücedağ İ.; Kahraman H. T.; Güvenç U.; Sönmez Y.: Haar wavelet neural network model. In: 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), pp. 1–8. IEEE (2018a)
7.
Pala T.; Güvenç U.; Kahraman H. T.; Yücedağ İ.; Sönmez Y.: Comparison of pooling methods for handwritten digit recognition problem. In: 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), pp. 1–5. IEEE (2018b)
8.
Korkmaz M.; Yücedağ İ.: Visual object detection with deep learning. In 1. International Technological Sciences and Design Symposium, pp.412–418 27–29 June- Giresun/Turkey (2018)
9.
10.
LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRef
11.
Krizhevsky, A.; Sutskever, I.; Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012)
12.
Zeiler, M.D.; Fergus, R.: Visualizing and understanding convolutional networks. In: European conference on computer vision, pp. 818–833. Springer, Cham (2014)
13.
Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A.: Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9. (2015)
14.
He, K.; Zhang, X.; Ren, S.; Sun, J.: Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. (2016)
15.
Simonyan, K.; Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint arXiv:​1409.​1556.
16.
Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K. Q.: Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, (2017)
17.
Sun, Y., Xue, B., Zhang, M., Yen, G. G.: Automatically designing CNN architectures using genetic algorithm for image classification (2018). arXiv preprint arXiv:​1808.​03818.
18.
Molchanov P.; Gupta, S.; Kim, K.; Kautz J.: Hand gesture recognition with 3D convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 1–7 (2015)
19.
Tang, A.; Lu, K.; Wang, Y.; Huang, J.; Li, H.: A real-time hand posture recognition system using deep neural networks. ACM Trans. Intell. Syst. Technol. (TIST) 6(2), 21 (2015)
20.
Wu, D.; Pigou, L.; Kindermans, P.J.; Le, N.D.H.; Shao, L.; Dambre, J.; Odobez, J.M.: Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38(8), 1583–1597 (2016)CrossRef
21.
Molchanov P.; Yang X.; Gupta S.; Kim K.; Tyree S.; Kautz J.: Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4207–4215 (2016)
22.
John V.; Boyali A.; Mita S.; Imanishi M.; Sanma N.: Deep learning-based fast hand gesture recognition using representative frames. In: 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–8. IEEE (2016)
23.
Koller O.; Ney H.; Bowden R.: Deep hand: How to train a CNN on 1 million hand images when your data is continuous and weakly labelled. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3793–3802 (2016).
24.
Arenas, J.O.P.; Moreno, R.J.; Murillo, P.C.U.: Hand gesture recognition by means of region-based convolutional neural networks. Contem Eng Sci 10(27), 1329–1342 (2017)CrossRef
25.
Bao, P.; Maqueda, A.I.; del-Blanco, C.R.; García, N.: Tiny hand gesture recognition without localization via a deep convolutional network. IEEE Trans. Consum. Electr. 63(3), 251–257 (2017)CrossRef
26.
Cao C.; Zhang Y.; Wu Y.; Lu H.; Cheng J.: Egocentric gesture recognition using recurrent 3d convolutional neural networks with spatiotemporal transformer modules. In: Proceedings of the IEEE International Conference on Computer Vision (2017).
27.
Fernández, D.N.; Kwolek, B.: Hand posture recognition using convolutional neural network. In: Iberoamerican congress on pattern recognition, pp. 441–449. Springer, Cham (2017)
28.
Roy K.; Mohanty A.; Sahay R. R.: Deep learning based hand detection in cluttered environment using skin segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 640–649 (2017)
29.
Dadashzadeh A.; Targhi A. T; Tahmasbi M.: HGR-Net: a two-stage convolutional neural network for hand gesture segmentation and recognition (2018). arXiv preprint arXiv:​1806.​05653.
30.
Devineau G.; Moutarde F.; Xi W.; Yang J.: Deep learning for hand gesture recognition on skeletal data. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 106–113. IEEE (2018)
31.
Nikolaev, E.I.; Dvoryaninov, P.V.; Lensky, Y.Y.; Drozdovsky, N.S.: Using virtual data for training deep model for hand gesture recognition. J. Phys. Conf. Series 1015(4), 042045 (2018)CrossRef
32.
Mathe E.; Mitsou A.; Spyrou E.; Mylonas P.: arm gesture recognition using a convolutional neural network. In 2018 13th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP), pp. 37–42 (2018)
33.
Alonso, D.G.; Teyseyre, A.; Berdun, L.; Schiaffino, S.: A deep learning approach for hybrid hand gesture recognition. In: Mexican international conference on artificial ıntelligence, pp. 87–99. Springer, Cham (2019)
34.
Li, G.; Tang, H.; Sun, Y.; Kong, J.; Jiang, G.; Jiang, D.; Liu, H.: Hand gesture recognition based on convolution neural network. Clust. Comput. 22(2), 2719–2729 (2019)CrossRef
35.
Nuzzi, C.; Pasinetti, S.; Lancini, M.; Docchio, F.; Sansoni, G.: Deep learning-based hand gesture recognition for collaborative robots. IEEE Instrum. Meas. Mag. 22(2), 44–51 (2019)CrossRef
36.
Côté-Allard, U.; Fall, C.L.; Drouin, A.; Campeau-Lecours, A.; Gosselin, C.; Glette, K.; Gosselin, B.: Deep learning for electromyographic hand gesture signal classification using transfer learning. IEEE Trans. Neural Syst. Rehabil. Eng. 27(4), 760–771 (2019)CrossRef
37.
Skaria, S.; Al-Hourani, A.; Lech, M.; Evans, R.J.: Hand-gesture recognition using two-antenna Doppler radar with deep convolutional neural networks. IEEE Sens. J. 19(8), 3041–3048 (2019)CrossRef
38.
39.
Hoang, V.T.: HGM-4: A new multi-cameras dataset for hand gesture recognition. Data Brief 30, 105676 (2020)CrossRef
40.
Nuzzi, C.; Pasinetti, S.; Pagani, R.; Coffetti, G.; Sansoni, G.: HANDS: an RGB-D dataset of static hand-gestures for human-robot interaction. Data Brief 35, 106791 (2021)CrossRef
41.
Sabour, S.; Frosst, N.; Hinton, G. E.: Dynamic routing between capsules. Adv. Neural Inf. Process. Syst., pp. 3856–3866 (2017)
42.
43.
Güler, O.; Yücedağ, İ.: Derin Öğrenme İle El Hareketi Tanıma Üzerine Yapılan Çalışmaların İncelenmesi. In 21. Akademik Bilişim konferansı, Ordu. (2019)
44.
Song, H.A.; Lee, S.Y.: Hierarchical representation using NMF. In: International conference on neural information processing, pp. 466–473. Springer, Berlin (2013)CrossRef
45.
Wang, L.; Sng, D.: Deep learning algorithms with applications to video analytics for a smart city: a survey (2015). arXiv preprint arXiv:​1512.​03131.
46.
Savaş, S.; Topaloğlu, N.; Kazcı, Ö.; Koşar, P. N.: Performance comparison of carotid artery ıntima media thickness classification by deep learning methods. In: HORA 2019-International Congress on Human-Computer Interaction Optimization and Robotic Applications SETSCI Conference Proceedings 4(5), 125–131 (2019). https://​doi.​org/​10.​36287/​setsci.​4.​5.​025
47.
Nagi, J.; Ducatelle, F.; Di Caro, G. A.; Cireşan, D.; Meier, U.; Giusti, A.; Nagi, F.; Schmidhuber, J.; Gambardella, L. M.: Max-pooling convolutional neural networks for vision-based hand gesture recognition. In: 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), pp. 342–347. IEEE (2011)
48.
İnik, Ö.; Ülker, E.: Derin Öğrenme ve Görüntü Analizinde Kullanılan Derin Öğrenme Modelleri. Gaziosmanpaşa Bilimsel Araştırma Dergisi 6(3), 85–104 (2017)
49.
Mobiny, A.; Van Nguyen, H.: Fast capsnet for lung cancer screening. In: International conference on medical ımage computing and computer-assisted intervention, pp. 741–749. Springer, Cham (2018)
50.
Kinli, F.; Ozcan, B.; Kirac, F. L.: Fashion image retrieval with capsule networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0–0 (2019)
51.
Xiao, H.; Rasul, K.; Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithm (2017). arXiv preprint arXiv:​1708.​07747.
52.
53.
Chauhan, R.; Ghanshala, K. K.; Joshi, R. C.: Convolutional neural network (CNN) for ımage detection and recognition. In 2018 First International Conference on Secure Cyber Computing and Communication (ICSCCC), pp. 278–282. IEEE (2018)
54.
Çayir, A.; Yenidoğan, I.; Dağ, H.: Feature extraction based on deep learning for some traditional machine learning methods. In 2018 3rd International Conference on Computer Science and Engineering (UBMK), pp. 494–497. IEEE (2018)
55.
Zhong, G.; Zheng, Y ; Zhang, X. Y.; Wei, H.; Ling, X.: Convolutional discriminant analysis. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 1456–1461. IEEE (2018).
56.
Do Rosario, V.M.; Borin, E.; Breternitz, M.: The multi-lane capsule network. IEEE Signal Process. Lett. 26(7), 1006–1010 (2019)CrossRef
57.
Luo, L.; Duan, S.; Wang, L.: R-CapsNet: An improvement of capsule network for more complex data. In 2019 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 2124–2129. IEEE (2019).
58.
Langroudi, H. F.; Carmichael, Z.; Gustafson, J. L.; Kudithipudi, D.: PositNN framework: tapered precision deep learning ınference for the edge. In 2019 IEEE Space Computing Conference (SCC), pp. 53–59. IEEE (2019)
59.
Bhatnagar, S.; Ghosal, D.; Kolekar, M. H.: Classification of fashion article images using convolutional neural networks. In: 2017 Fourth International Conference on Image Information Processing (ICIIP), pp. 1–6. IEEE (2017)
60.
Agarap, A. F.: An architecture combining convolutional neural network (CNN) and support vector machine (SVM) for image classification (2017). arXiv preprint arXiv:​1712.​03541.
61.
JE, S. D.; Priyanka, R.; RK, S.; HR, S.: Apparel classification using CNN. Int. J. Sci. Res. Rev. 07(03) (2019)
62.
Greeshma, K.V.; Sreekumar, K.: Hyperparameter optimization and regularization on fashion-MNIST classification. Int. J. Recent Technol. Eng. (IJRTE) 8(2), 3713–3719 (2019)CrossRef
Metadata
Title
Hand Gesture Recognition from 2D Images by Using Convolutional Capsule Neural Networks
Authors
Osman Güler
İbrahim Yücedağ
Publication date
25-06-2021
Publisher
Springer Berlin Heidelberg
Published in
Arabian Journal for Science and Engineering / Issue 2/2022
Print ISSN: 2193-567X
Electronic ISSN: 2191-4281
DOI
https://doi.org/10.1007/s13369-021-05867-2

Other articles of this Issue 2/2022

Arabian Journal for Science and Engineering 2/2022 Go to the issue

Research Article-Computer Engineering and Computer Science

On the Application of IoT: Slope Monitoring System for Open-cast Mines Based on LoRa Wireless Communication

Research Article-Computer Engineering and Computer Science

A Highly Reliable and Cost-effective Service Model for Finite Population Clouds: Analysis and Implementation

Research Article-Computer Engineering and Computer Science

Optimization, Modeling and Implementation of Plant Water Consumption Control Using Genetic Algorithm and Artificial Neural Network in a Hybrid Structure

Research Article-Computer Engineering and Computer Science

Research on Behavior of Two New Random Entity Mobility Models in 3-D Space

Research Article-Computer Engineering and Computer Science

The Role of Vertical Elastic Namenode in Handling Big Data in Small Files

Research Article-Computer Engineering and Computer Science

To Secure the Communication in Powerful Internet of Things Using Innovative Post-Quantum Cryptographic Method

Premium Partners

    Image Credits