nach oben

Arabian Journal for Science and Engineering

Erschienen in:

22.04.2022 | Research Article-Computer Engineering and Computer Science

Applying Hybrid Deep Neural Network for the Recognition of Sign Language Words Used by the Deaf COVID-19 Patients

verfasst von: Adithya Venugopalan, Rajesh Reghunadhan

Erschienen in: Arabian Journal for Science and Engineering | Ausgabe 2/2023

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

The rapid spread of the novel corona virus disease (COVID-19) has disrupted the traditional clinical services all over the world. Hospitals and healthcare centers have taken extreme care to minimize the risk of exposure to the virus by restricting the visitors and relatives of the patients. The dramatic changes happened in the healthcare norms have made it hard for the deaf patients to communicate and receive appropriate care. This paper reports a work on automatic sign language recognition that can mitigate the communication barrier between the deaf patients and the healthcare workers in India. Since hand gestures are the most expressive components of a sign language vocabulary, a novel dataset of dynamic hand gestures for the Indian sign language (ISL) words commonly used for emergency communication by deaf COVID-19 positive patients is proposed. A hybrid model of deep convolutional long short-term memory network has been utilized for the recognition of the proposed hand gestures and achieved an average accuracy of 83.36%. The model performance has been further validated on an alternative ISL dataset as well as a benchmarking hand gesture dataset and obtained average accuracies of \(97\%\) and \(99.34\pm 0.66\%\), respectively.

Vorheriger Artikel A Traffic Sign Detection Network Based on PosNeg-Balanced Anchors and Domain Adaptation

Nächster Artikel Expectation-Based Probabilistic Naive Approach for Forecasting Involving Optimized Parameter Estimation

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Adithya, V.; Rajesh, R.: Hand Gestures for emergency situations: a video dataset based on words from Indian sign language. Data Brief (2020). https://doi.org/10.1016/j.dib.2020.106016CrossRef

Wadhawan, A.; Kumar, P.: Sign language recognition systems: a decade systematic literature review. Arch. Comput. Methods Eng. (2017). https://doi.org/10.1007/s11831-019-09384-2CrossRef

Elakkiya, R.: Machine learning based sign language recognition: a review and its research frontier. J. Ambient Intell. Humaniz. Comput. (2020). https://doi.org/10.1007/s12652-020-02396-yCrossRef

Khan, A.; Sohail, A.; Zahoora, U.; Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artif. Intell. Rev. (2020). https://doi.org/10.1007/s10462-020-09825-6CrossRef

Sherstinsky, A.: Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Phys. D Nonlinear Phenom. (2020). https://doi.org/10.1016/j.physd.2019.132306MathSciNetCrossRefMATH

Pisharady, P.K.; Saerbeck, M.: Recent methods and databases in vision based hand gesture recognition: a review. Comput. Vis. Image Underst. 141, 152–165 (2015). https://doi.org/10.1016/j.cviu.2015.08.004CrossRef

Cheok, M.J.; Omar, Z.; Jaward, M.H.: A review of hand gesture and sign language recognition techniques. Int. J. Mach. Learn. Cybern. 10, 131–153 (2019). https://doi.org/10.1007/s13042-017-0705-5CrossRef

Matetelki, P.; Pataki, M.; Turbucz, S.; Kovacs, L.: An assistive interpreter tool using glove based hand gesture recognition. In: IEEE Canada International Humanitarian Technology Conference—(IHTC); Montreal, QC, Canada; (2014). pp.1-5. https://doi.org/10.1109/IHTC.2014.7147529

Soli, Google: Project soli, Google. https://atap.google.com/soli (2015)

10.

Jaramillo-Yánez, A.; Benalcázar, M.E.; Mena-Maldonado, E.: Real-time hand gesture recognition using surface electromyography and machine learning: a systematic literature review. Sensors 20, 2467 (2020). https://doi.org/10.3390/s20092467CrossRef

11.

Adaloglou, N.; Chatzis, T.; Papastratis, I.; Stergioulas, A.; Papadopoulos, G.T. et al.: A Comprehensive Study on Sign Language Recognition Methods. arXiv e-prints 2020; arXiv:2007.12530

12.

Rastgoo, R.; Kiani, K.; Escalera, S.: Sign language recognition: a deep survey. Expert Syst. Appl. (2021). https://doi.org/10.1016/j.eswa.2020.113794CrossRef

13.

Jiang, X.; Zhu, Z.; Zhang, M.: Recognition of chinese finger sign language via gray-level co-occurrence matrix and k-nearest neighbor algorithm. In: 3rd International Conference on Electronic Information Technology and Computer Engineering (EITCE); Xiamen, China; 2019. pp. 152-156. https://doi.org/10.1109/EITCE47263.2019.9094915

14.

Nogales, R.; Benalcazar, M.: Real-time hand gesture recognition using the leap motion controller and machine learning. In: IEEE Latin American conference on computational intelligence (LA-CCI); Guayaquil, Ecuador; (2019). pp.1-7. https://doi.org/10.1109/LA-CCI47412.2019.9037037

15.

Verma, B.; Choudhary, A.: Grassmann manifold based dynamic hand gesture recognition using depth data. Multimed. Tools Appl. 79, 2213–2237 (2020). https://doi.org/10.1007/s11042-019-08266-wCrossRef

16.

Neyra-Gutiérrez, A.; Shiguihara-Juárez, M P.: Feature extraction with video summarization of dynamic gestures for peruvian sign language recognition. In: IEEE XXVII International conference on electronics, electrical engineering and computing (INTERCON); Lima, Peru, (2020). pp.1-4. https://doi.org/10.1109/INTERCON50315.2020.9220243

17.

Li, C.; Bai, X.; Xie, X.; Tian, L.: Dynamic hand gesture recognition based on 3D skeleton. In: IEEE 5th International conference on computer and communications (ICCC); Montreal, QC, Canada, (2019). pp. 1701-1705. https://doi.org/10.1109/ICCC47050.2019.9064200

18.

Bai, X.; Li, C.; Tian, L.; Song, H.: Dynamic Hand gesture recognition based on depth information. In: International Conference on control, automation and information sciences (ICCAIS); Hangzhou, China, (2018). pp. 216-221. https://doi.org/10.1109/ICCAIS.2018.8570336

19.

Agab, S.E.; Chelali, F.Z.: Dynamic Hand Gesture Recognition based on Textural Features. In: International Conference on Advanced Electrical Engineering (ICAEE); Algiers, Algeria; 2019. pp.1-6. https://doi.org/10.1109/ICAEE47123.2019.9014683

20.

Ahmed, W.; Chanda, K.; Mitra, S.: Vision based Hand Gesture Recognition using Dynamic Time Warping for Indian Sign Language. In: IEEE international conference on information science (ICIS); Kochi, India, (2016). pp.120-125. https://doi.org/10.1109/INFOSCI.2016.7845312

21.

Rao, G.A.; Kishore, P.V.V.: Selfie Video based continuous Indian sign language recognition system. Ain Shams Eng. J. 9(4), 1929–1939 (2018). https://doi.org/10.1016/j.asej.2016.10.013CrossRef

22.

Athira, P.K.; Sruthi, C.J.; Lijiya, A.: A signer independent sign language recognition with co-articulation elimination from live videos: an Indian scenario. J. King Saud Univ. Comput. Inf. Sci. (2019). https://doi.org/10.1016/j.jksuci.2019.05.002CrossRef

23.

Ji, Y.; Zhang, H.; Jie, Z.; Ma, L.; Wu, Q.M.: Jonathan: CASNet: a cross-attention siamese network for video salient object detection. IEEE Trans. Neural Netw. Learn. Syst. 32(6), 2676–2690 (2021). https://doi.org/10.1109/TNNLS.2020.3007534CrossRef

24.

Li, X.; He, M.; Li, H.; Shen, H.: A combined loss-based multiscale fully convolutional network for high-resolution remote sensing image change detection. IEEE Geosci. Remote Sens. Lett. (2021). https://doi.org/10.1109/LGRS.2021.3098774CrossRef

25.

Al-Hammadi, M.; Muhammad, G.; Abdu, W.; Alsulaiman, M.; Bencherif, M.A.; et al.: Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation. IEEE Access 8, 192527–192542 (2020). https://doi.org/10.1109/ACCESS.2020.3032140CrossRef

26.

Aly, S.; Aly, W.: DeepArSLR: a novel signer-independent deep learning framework for isolated arabic sign language gestures recognition. IEEE Access 8, 83199–83212 (2020). https://doi.org/10.1109/ACCESS.2020.2990699CrossRef

27.

Rahim, M.A.; Shin, J.; Islam, M.R.: Dynamic Hand Gesture Based Sign Word Recognition Using Convolutional Neural Network with Feature Fusion. In: IEEE 2nd international conference on knowledge innovation and invention (ICKII); Seoul, Korea; 2019. pp. 221-224. https://doi.org/10.1109/ICKII46306.2019.9042600

28.

Lim, K.M.; Tan, A.W.C.; Lee, C.P.; Tan, S.C.: Isolated sign language recognition using convolutional neural network hand modelling and hand energy image. Multimed. Tools Appl. 78, 19917–19944 (2019). https://doi.org/10.1007/s11042-019-7263-7CrossRef

29.

Erhu, Z.; Xue, B.; Cao, F.; Duan, J.; Lin, G.; et al.: Fusion of 2D CNN and 3D DenseNet for dynamic gesture recognition. Electronics 8, 1511 (2019). https://doi.org/10.3390/electronics8121511CrossRef

30.

Hoang, N.N.; Lee, G.-S.; Kim, S.-H.; Yang, H.-J.: A real-time multimodal hand gesture recognition via 3D convolutional neural network and key frame extraction. In: International conference on machine learning and machine intelligence; Ha Noi Viet Nam; (2018). pp. 32-37. https://doi.org/10.1145/3278312.3278314

31.

Dhingra, N.; Kunz, A.: Res3ATN - Deep 3D Residual Attention Network for Hand Gesture Recognition in Videos. In: International Conference on 3D Visio; Québec City, QC, Canada; (2019). pp. 491–501. https://doi.org/10.1109/3DV.2019.00061

32.

Nunez, J.C.; Cabido, R.; Pantrigo, J.J.; Montemayor, A.S.; Velez, J.F.: Convolutional neural networks and long short-term memory for skeleton based human activity and hand gesture recognition. Pattern Recognit. 76, 80–94 (2018). https://doi.org/10.1016/j.patcog.2017.10.033CrossRef

33.

Hakim, N.K.; Shih, T.K.; Kasthuri Arachchi, S.P.; Aditya, W.; Chen, Y.C.; et al.: Dynamic hand gesture recognition using 3DCNN and LSTM with FSM context-aware model. Sensors 19(24), 5429 (2019). https://doi.org/10.3390/s19245429CrossRef

34.

Kamruzzaman, M.M.: Arabic sign language recognition and generating arabic speech using convolutional neural network. Wirel. Commun. Mobile Comput. (2020). https://doi.org/10.1155/2020/3685614CrossRef

35.

Li, G.; Tang, H.; Sun, Y.; Kong, J.; Jiang, G.; Jiang, D.; Tao, B.; Xu, S.; Liu, H.: Hand gesture recognition based on convolution neural network. Cluster Comput. 22, 2719–2729 (2019). https://doi.org/10.1007/s10586-017-1435-xCrossRef

36.

Tasmere, D.; Ahmed, B.: Hand gesture recognition for bangla sign language using deep convolution neural network. 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI) (2020); 1-5. https://doi.org/10.1109/STI50764.2020.9350484.

37.

Avola, D.; Bernardi, M.; Cinque, L.; Foresti, G.L.; Massaroni, C.: Exploiting recurrent neural networks and leap motion controller for the recognition of sign language and semaphoric hand gestures. IEEE Trans. Multimed. 21(1), 234–245 (2018). https://doi.org/10.1109/TMM.2018.2856094CrossRef

38.

Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-yMathSciNetCrossRef

39.

Karen, S.; Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014

40.

Van Houdt, G.; Mosquera, C.; Napoles, G.: A review on the long short-term memory model. Artif. Intell. Rev. 53, 5929–5955 (2020). https://doi.org/10.1007/s10462-020-09838-1CrossRef

41.

Zhang, H.; Huang, B.; Tian, G.: Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture. Pattern Recognit. Lett. 131, 128–134 (2020). https://doi.org/10.1016/j.patrec.2019.12.013CrossRef

42.

Kim, T-K.; Wong, S-F.; Cipolla, R.: Tensor canonical correlation analysis for action classification. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, MN (2007)

43.

Kurmanji, M.; Ghaderi, F.: Hand gesture recognition from RGB-D data using 2D and 3D convolutional neural networks: a comparative study. J. AI Data Min. 8(2), 177–188 (2020). https://doi.org/10.22044/jadm.2019.7903.1929CrossRef

44.

John, V.; Boyali, A.; Mita, S.; Imanishi, M.; Sanma, N.: Dep learning-based fast hand gesture recognition using representative frames. In: International Conference on Digital Image Computing: Techniques and Applications (DICTA) (2016) https://doi.org/10.1109/DICTA.2016.7797030.

45.

Lui, Y.M.: Human gesture recognition on product manifolds. J. Mach. Learn. Res. 13(1), 3297–3321 (2012)MathSciNetMATH

46.

Sanin, A.; Sanderson, C.; Harandi, M.T.; and Lovell, B.C.: Spatio-Temporal Covariance Descriptors for Action and Gesture Recognition. In: IEEE Workshop on Applications of Computer Vision (WACV) (2013), 103-110

47.

Baraldi, L.; Paci, F.; Serra, G.; Benini, L.; Cucchiara, R.: Gesture recognition in ego-centric videos using dense trajectories and hand segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops 702–707 (2014). https://doi.org/10.1109/CVPRW.2014.107

48.

Chandra, S.; Jawahar, C.V.: Partial Least Squares kernel for computing similarities between video sequences. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR) (2012); 13-516

49.

Souza, L.S.; Gatto, B.B.; Xue, J.; Fukui, K.: Enhanced Grassmann discriminant analysis with randomized time warping for motion recognition. Pattern Recognit. (2020). https://doi.org/10.1016/j.patcog.2019.107028CrossRef

50.

Zhao, Z.; Elgammal, A.: Information theoretic key frame selection for action recognition. BMVC (2008)

51.

Hoang, N.N.; Lee, G-S; Kim, S.-H.; Yang, H.-J.: effective hand gesture recognition by key frame selection and 3D neural network [Internet]. Korean Institute of Smart Media. Korean Institute of Smart Media 2020; 9: 23–29. https://doi.org/10.30693/SMJ.2020.9.1.23

52.

Tang, H.; Liu, H.; Xiao, W.; Sebe, N.: Fast and robust dynamic hand gesture recognition via key frames extraction and feature fusion. Neurocomputing 331, 424–433 (2019). https://doi.org/10.1016/j.neucom.2018.11.038CrossRef

53.

Wang, L.; Garg, H.: Algorithm for multiple attribute decision-making with interactive archimedean norm operations under pythagorean fuzzy uncertainty. Int. J. Comput. Intell. Syst. 14(1), 503–527 (2020). https://doi.org/10.2991/ijcis.d.201215.002CrossRef

Titel: Applying Hybrid Deep Neural Network for the Recognition of Sign Language Words Used by the Deaf COVID-19 Patients
verfasst von: Adithya Venugopalan
Rajesh Reghunadhan
Publikationsdatum: 22.04.2022
Verlag: Springer Berlin Heidelberg
Erschienen in: Arabian Journal for Science and Engineering / Ausgabe 2/2023
Print ISSN: 2193-567X
Elektronische ISSN: 2191-4281
DOI: https://doi.org/10.1007/s13369-022-06843-0

Premium Partner

Marktübersichten

Die im Laufe eines Jahres in der „adhäsion“ veröffentlichten Marktübersichten helfen Anwendern verschiedenster Branchen, sich einen gezielten Überblick über Lieferantenangebote zu verschaffen.

Zur Marktübersicht

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Weitere Artikel der Ausgabe 2/2023

Indoor Pedestrian Trajectory Reconstruction Using Spatial–Temporal Error Correction and Dynamic Time Warping-Based Path Matching for Fingerprints Map Creation

A Speech-Based Hybrid Decision Support System for Early Detection of Parkinson's Disease

A Hybrid Method for Big Data Analysis Using Fuzzy Clustering, Feature Selection and Adaptive Neuro-Fuzzy Inferences System Techniques: Case of Mecca and Medina Hotels in Saudi Arabia

Complexity Metric Methodology of Infrared Image Sequence for Single-Object Tracking

Research on Trenching Data Correction Method Based on Wavelet Denoising-Kalman Filtering Algorithm

Lightweight Two-Factor-Based User Authentication Protocol for IoT-Enabled Healthcare Ecosystem in Quantum Computing

Premium Partner

Marktübersichten