Abstract
Nowadays, hand gesture recognition has become an alternative for human-machine interaction. It has covered a large area of applications like 3D game technology, sign language interpreting, VR (virtual reality) environment, and robotics. But detection of the hand portion has become a challenging task in computer vision and pattern recognition communities. Deep learning algorithm like convolutional neural network (CNN) architecture has become a very popular choice for classification tasks, but CNN architectures suffer from some problems like high variance during prediction, overfitting problem and also prediction errors. To overcome these problems, an ensemble of CNN-based approaches is presented in this paper. Firstly, the gesture portion is detected by using the background separation method based on binary thresholding. After that, the contour portion is extracted, and the hand region is segmented. Then, the images have been resized and fed into three individual CNN models to train them in parallel. In the last part, the output scores of CNN models are averaged to construct an optimal ensemble model for the final prediction. Two publicly available datasets (labeled as Dataset-1 and Dataset-2) containing infrared images and one self-constructed dataset have been used to validate the proposed system. Experimental results are compared with the existing state-of-the-art approaches, and it is observed that our proposed ensemble model outperforms other existing proposed methods.
Similar content being viewed by others
References
Chen Z-h, Kim J-T, Liang J, Zhang J, Yuan Y-B (2014) Real-time hand gesture recognition using finger segmentation. The Scientific World Journal, Hindawi
Chen Zh, Kim JT, Liang J, Zhang J, Yuan YB (2014) Real-time hand gesture recognition using finger segmentation. The Scientific World Journal, vol 2014
Chuan C-H, Regina E, Guardino C (2014) American sign language recognition using leap motion sensor. In: 13th international conference on machine learning and applications, pp 541–544
Deng J, Dong W, Socher R, Li LJ et al (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255
Fang W, Ding Y, Zhang F, Sheng J (2019) Gesture recognition based on CNN and DCGAN for calculation and text output. IEEE Access 7:28230–28237
Gupta G (2011) Algorithm for image processing using improved median filter and comparison of mean, median and improved median filter. Int J Soft Comput Eng (IJSCE) 1(5):304–311
Hu B, Wang J (2020) Deep learning based hand gesture recognition and UAV flight controls. International Journal of Automation and Computing, Springer 17(1):17–29
Huang D-Y, Hub W-C, Chang S-H (2011) Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination. Expert Syst Appl 38(5):6031–6042
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv:1412.6980
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, pp 1097–1105
Li G, Tang H, Sun Y, Kong J, et al. (2019) Jiang Hand gesture recognition based on convolution neural network. Clust Comput 22(2):2719–2729
Mantecón T, del Blanco CR, Jaureguizar F, García N (2016) Hand gesture recognition using infrared imagery provided by leap motion controller. In: International conference on advanced concepts for intelligent vision systems, pp 47–57
Mantecón T, del Blanco CR, Jaureguizar F, García N (2019) A real-time gesture recognition system using near-infrared imagery. PloS One, 14(10)
Neethu PS, Suguna R, Sathish D (2020) An efficient method for human hand gesture detection and recognition using deep learning convolutional neural networks. Soft Comput, pp 1–10
Pititeeraphab Y, Choitkunnan P, Thongpance N, Kullathum K, Pintavirooj C (2016) Robot-arm control system using LEAP motion controller. In: 2016 international conference on biomedical engineering (BME-HUST), pp 109–112
Polikar R (2012) Ensemble learning. Springer, Berlin, pp 1–34
Rajaraman S, Jaeger S, Antani SK (2019) Performance evaluation of deep neural ensembles toward malaria parasite detection in thin-blood smear images. PeerJ 7:e6977
Rakibe RS, Patil BD (2013) Background subtraction algorithm based human motion detection. International Journal of Scientific and Research Publications (Citeseer) 3(5):2250–3153
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, et al. (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9
Wang H, Wang P, Song Z, Li W (2017) Large-scale multimodal gesture segmentation and recognition based on convolutional neural networks. In: Proceedings of the IEEE international conference on computer vision workshops, pp 3138–3146
Wei L, Tong Z, Chu J (2016) Dynamic hand gesture recognition with leap motion controller. IEEE Signal Process Lett 23(9):1188–1192
Xu P (2017) A real-time hand gesture recognition and human-computer interaction system. arXiv:1704.07296
Yingxin X, Jinghua L, Lichun W, Dehui K (2016) A robust hand gesture recognition method via convolutional neural network. In: 6th international conference on digital home (ICDH), pp 64–67
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sen, A., Mishra, T.K. & Dash, R. A novel hand gesture detection and recognition system based on ensemble-based convolutional neural network. Multimed Tools Appl 81, 40043–40066 (2022). https://doi.org/10.1007/s11042-022-11909-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-11909-0