Skip to main content
Top
Published in: Wireless Personal Communications 1/2022

28-09-2021

Recognition of Indian Sign Language (ISL) Using Deep Learning Model

Authors: Sakshi Sharma, Sukhwinder Singh

Published in: Wireless Personal Communications | Issue 1/2022

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

An efficient sign language recognition system (SLRS) can recognize the gestures of sign language to ease the communication between the signer and non-signer community. In this paper, a computer-vision based SLRS using a deep learning technique has been proposed. This study has primary three contributions: first, a large dataset of Indian sign language (ISL) has been created using 65 different users in an uncontrolled environment. Second, the intra-class variance in dataset has been increased using augmentation to improve the generalization ability of the proposed work. Three additional copies for each training image are generated in this paper, by using three different affine transformations. Third, a novel and robust model using Convolutional Neural Network (CNN) have been proposed for the feature extraction and classification of ISL gestures. The performance of this method is evaluated on a self-collected ISL dataset and publicly available dataset of ASL. For this total of three datasets have been used and the achieved accuracy is 92.43, 88.01, and 99.52%. The efficiency of this method has been also evaluated in terms of precision, recall, f-score, and time consumed by the system. The results indicate that the proposed method shows encouraging performance compared with existing work.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
2.
go back to reference Johnson, J. E., & Johnson, R. J. (2008). Assessment of regional language varieties in indian sign language. SIL International, Dallas, Texas, vol 2008, (pp. 1–121). Johnson, J. E., & Johnson, R. J. (2008). Assessment of regional language varieties in indian sign language. SIL International, Dallas, Texas, vol 2008, (pp. 1–121).
3.
go back to reference Kumar, D. A., Sastry, A. S. C. S., Kishore, P. V. V., & Kumar, E. K. (2018). 3D sign language recognition using spatio temporal graph kernels. Journal of King Saud University-Computer and Information Sciences. Kumar, D. A., Sastry, A. S. C. S., Kishore, P. V. V., & Kumar, E. K. (2018). 3D sign language recognition using spatio temporal graph kernels. Journal of King Saud University-Computer and Information Sciences.
4.
go back to reference Sharma, S., & Singh, S. (2020). Vision-based sign language recognition system: A Comprehensive Review. In: IEEE International Conference on Inventive Computation Technologies (ICICT), (pp. 140–144). Sharma, S., & Singh, S. (2020). Vision-based sign language recognition system: A Comprehensive Review. In: IEEE International Conference on Inventive Computation Technologies (ICICT), (pp. 140–144).
5.
go back to reference Sharma, S., & Singh, S. (2021). Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Systems with Applications, 182, 115657.CrossRef Sharma, S., & Singh, S. (2021). Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Systems with Applications, 182, 115657.CrossRef
6.
go back to reference Cheok, M. J., Omar, Z., & Jaward, M. H. (2019). A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cybernetics, 10(1), 131–153.CrossRef Cheok, M. J., Omar, Z., & Jaward, M. H. (2019). A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cybernetics, 10(1), 131–153.CrossRef
7.
go back to reference Gangrade, J., & Bharti, J. (2020). Vision-based hand gesture recognition for indian sign language using convolution neural network. IETE Journal of Research, 1–10. Gangrade, J., & Bharti, J. (2020). Vision-based hand gesture recognition for indian sign language using convolution neural network. IETE Journal of Research, 1–10.
8.
go back to reference Sharma, S., & Singh, S. (2019). An analysis of reversible data hiding algorithms for encrypted domain. In: 2019 Third International conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), (pp. 644–648). IEEE. Sharma, S., & Singh, S. (2019). An analysis of reversible data hiding algorithms for encrypted domain. In: 2019 Third International conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), (pp. 644–648). IEEE.
10.
go back to reference Kakoty, N. M., & Sharma, M. D. (2018). Recognition of sign language alphabets and numbers based on hand kinematics using A data glove. Procedia Computer Science, 133, 55–62.CrossRef Kakoty, N. M., & Sharma, M. D. (2018). Recognition of sign language alphabets and numbers based on hand kinematics using A data glove. Procedia Computer Science, 133, 55–62.CrossRef
11.
go back to reference Suri, K., & Gupta, R. (2019). Convolutional neural network array for sign language recognition using wearable IMUs. In: 2019 6th International Conference on Signal Processing and Integrated Networks (SPIN), (pp. 483–488). Suri, K., & Gupta, R. (2019). Convolutional neural network array for sign language recognition using wearable IMUs. In: 2019 6th International Conference on Signal Processing and Integrated Networks (SPIN), (pp. 483–488).
12.
go back to reference Rewari, H., Dixit, V., Batra, D., & Hema, N. (2018). Automated sign language interpreter. In: Eleventh International Conference on Contemporary Computing (IC3), (pp. 1–5). Rewari, H., Dixit, V., Batra, D., & Hema, N. (2018). Automated sign language interpreter. In: Eleventh International Conference on Contemporary Computing (IC3), (pp. 1–5).
13.
go back to reference Chong, T. W., & Kim, B. J. (2020). American sign language recognition system using wearable sensors with deep learning approach. The Journal of the Korea Institute of Electronic Communication Sciences, 15(2), 291–298. Chong, T. W., & Kim, B. J. (2020). American sign language recognition system using wearable sensors with deep learning approach. The Journal of the Korea Institute of Electronic Communication Sciences, 15(2), 291–298.
14.
go back to reference Gupta, R., & Kumar, A. (2020). Indian sign language recognition using wearable sensors and multi-label classification. Computers & Electrical Engineering, 90, 106898.CrossRef Gupta, R., & Kumar, A. (2020). Indian sign language recognition using wearable sensors and multi-label classification. Computers & Electrical Engineering, 90, 106898.CrossRef
15.
go back to reference Das, S. P., Talukdar, A. K., & Sarma, K. K. (2015). Sign language recognition using facial expression. Procedia Computer Science, 58, 210–216.CrossRef Das, S. P., Talukdar, A. K., & Sarma, K. K. (2015). Sign language recognition using facial expression. Procedia Computer Science, 58, 210–216.CrossRef
16.
go back to reference Tripathi, K., & Nandi, N. B. G. (2015). Continuous Indian sign language gesture recognition and sentence formation. Procedia Computer Science, 54, 523–531.CrossRef Tripathi, K., & Nandi, N. B. G. (2015). Continuous Indian sign language gesture recognition and sentence formation. Procedia Computer Science, 54, 523–531.CrossRef
17.
go back to reference Lee, G. C., Yeh, F. H., & Hsiao, Y. H. (2016). Kinect-based Taiwanese sign-language recognition system. Multimedia Tools and Applications, 75(1), 261–279.CrossRef Lee, G. C., Yeh, F. H., & Hsiao, Y. H. (2016). Kinect-based Taiwanese sign-language recognition system. Multimedia Tools and Applications, 75(1), 261–279.CrossRef
18.
go back to reference Ansari, Z. A., & Harit, G. (2016). Nearest neighbour classification of Indian sign language gestures using kinect camera. Sadhana, 41(2), 161–182.MathSciNetCrossRef Ansari, Z. A., & Harit, G. (2016). Nearest neighbour classification of Indian sign language gestures using kinect camera. Sadhana, 41(2), 161–182.MathSciNetCrossRef
19.
go back to reference Beena, M. V., Namboodiri, M. A., & Dean, P. G. (2017). Automatic sign language finger spelling using convolution neural network: Analysis. International Journal of Pure and Applied Mathematics, 117(20), 9–15. Beena, M. V., Namboodiri, M. A., & Dean, P. G. (2017). Automatic sign language finger spelling using convolution neural network: Analysis. International Journal of Pure and Applied Mathematics, 117(20), 9–15.
20.
go back to reference Kumar, E. K., Kishore, P. V. V., Sastry, A. S. C. S., Kumar, M. T. K., & Kumar, D. A. (2018). Training CNNs for 3-D sign language recognition with color texture coded joint angular displacement maps. IEEE Signal Processing Letters, 25(5), 645–649.CrossRef Kumar, E. K., Kishore, P. V. V., Sastry, A. S. C. S., Kumar, M. T. K., & Kumar, D. A. (2018). Training CNNs for 3-D sign language recognition with color texture coded joint angular displacement maps. IEEE Signal Processing Letters, 25(5), 645–649.CrossRef
21.
go back to reference Müller, M., Röder, T., Clausen, M., Eberhardt, B., Krüger, B., & Weber, A. (2007). Documentation mocap database hdm05. Müller, M., Röder, T., Clausen, M., Eberhardt, B., Krüger, B., & Weber, A. (2007). Documentation mocap database hdm05.
22.
go back to reference Rao, G. A., & Kishore, P. V. V. (2018). Selfie video based continuous Indian sign language recognition system. Ain Shams Engineering Journal, 9(4), 1929–1939.CrossRef Rao, G. A., & Kishore, P. V. V. (2018). Selfie video based continuous Indian sign language recognition system. Ain Shams Engineering Journal, 9(4), 1929–1939.CrossRef
23.
go back to reference Xie, B., He, X., & Li, Y. (2018). RGB-D static gesture recognition based on convolutional neural network. The Journal of Engineering, 1515–1520. Xie, B., He, X., & Li, Y. (2018). RGB-D static gesture recognition based on convolutional neural network. The Journal of Engineering, 1515–1520.
24.
go back to reference Pugeault, N., & Bowden, R. (2011). Spelling it out: Real-time ASL fingerspelling recognition. In: IEEE International Conference on Computer Vision Workshops (ICCV workshops), (pp. 1114–1119). Pugeault, N., & Bowden, R. (2011). Spelling it out: Real-time ASL fingerspelling recognition. In: IEEE International Conference on Computer Vision Workshops (ICCV workshops), (pp. 1114–1119).
25.
go back to reference Elpeltagy, M., Abdelwahab, M., Hussein, M. E., Shoukry, A., Shoala, A., & Galal, M. (2018). Multi-modality-based Arabic sign language recognition. IET Computer Vision, 12(7), 1031–1039.CrossRef Elpeltagy, M., Abdelwahab, M., Hussein, M. E., Shoukry, A., Shoala, A., & Galal, M. (2018). Multi-modality-based Arabic sign language recognition. IET Computer Vision, 12(7), 1031–1039.CrossRef
26.
go back to reference Ibrahim, N. B., Selim, M. M., & Zayed, H. H. (2018). An automatic arabic sign language recognition system (ArSLRS). Journal of King Saud University-Computer and Information Sciences, 30(4), 470–477.CrossRef Ibrahim, N. B., Selim, M. M., & Zayed, H. H. (2018). An automatic arabic sign language recognition system (ArSLRS). Journal of King Saud University-Computer and Information Sciences, 30(4), 470–477.CrossRef
27.
go back to reference Kumar, P., Roy, P. P., & Dogra, D. P. (2018). Independent bayesian classifier combination based sign language recognition using facial expression. Information Sciences, 428, 30–48.MathSciNetCrossRef Kumar, P., Roy, P. P., & Dogra, D. P. (2018). Independent bayesian classifier combination based sign language recognition using facial expression. Information Sciences, 428, 30–48.MathSciNetCrossRef
28.
go back to reference Jose, H., & Julian, A. (2019). Tamil sign language translator—An assistive system for hearing-and speech-impaired people. In: Information and Communication Technology for Intelligent Systems, Springer, (pp. 249–257). Jose, H., & Julian, A. (2019). Tamil sign language translator—An assistive system for hearing-and speech-impaired people. In: Information and Communication Technology for Intelligent Systems, Springer, (pp. 249–257).
29.
go back to reference Ferreira, P. M., Cardoso, J. S., & Rebelo, A. (2019). On the role of multimodal learning in the recognition of sign language. Multimedia Tools and Applications, 78(8), 10035–10056.CrossRef Ferreira, P. M., Cardoso, J. S., & Rebelo, A. (2019). On the role of multimodal learning in the recognition of sign language. Multimedia Tools and Applications, 78(8), 10035–10056.CrossRef
30.
go back to reference Sruthi, C. J., & Lijiya, A. (2019). Signet: A deep learning based indian sign language recognition system. In: 2019 International Conference on Communication and Signal Processing (ICCSP), (pp. 596–600). Sruthi, C. J., & Lijiya, A. (2019). Signet: A deep learning based indian sign language recognition system. In: 2019 International Conference on Communication and Signal Processing (ICCSP), (pp. 596–600).
31.
go back to reference Wadhawan, A., & Kumar, P. (2020). Deep learning-based sign language recognition system for static signs. Neural Computing and Applications, 1–12. Wadhawan, A., & Kumar, P. (2020). Deep learning-based sign language recognition system for static signs. Neural Computing and Applications, 1–12.
32.
go back to reference Kumar, A., & Kumar, R. (2021). A novel approach for ISL alphabet recognition using Extreme Learning Machine. International Journal of Information Technology, 13(1), 349–357.CrossRef Kumar, A., & Kumar, R. (2021). A novel approach for ISL alphabet recognition using Extreme Learning Machine. International Journal of Information Technology, 13(1), 349–357.CrossRef
33.
go back to reference Sharma, A., Sharma, N., Saxena, Y., Singh, A., & Sadhya, D. (2020). Benchmarking deep neural network approaches for Indian Sign Language recognition. Neural Computing and Applications, 1–12. Sharma, A., Sharma, N., Saxena, Y., Singh, A., & Sadhya, D. (2020). Benchmarking deep neural network approaches for Indian Sign Language recognition. Neural Computing and Applications, 1–12.
34.
go back to reference Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., & Rabinovich, A. (2015). Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 1–9). Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., & Rabinovich, A. (2015). Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 1–9).
35.
go back to reference Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint. arXiv:1409.1556. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint. arXiv:1409.1556.
36.
go back to reference Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 2818–2826). Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 2818–2826).
37.
go back to reference Fukushima, K. (1980). A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36, 193–202.CrossRef Fukushima, K. (1980). A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36, 193–202.CrossRef
38.
go back to reference Rahman, M. M., Islam, M. S., Sassi, R., & Aktaruzzaman, M. (2019). Convolutional neural networks performance comparison for handwritten bengali numerals recognition. SN Applied Sciences, 1(12), 1–11. Rahman, M. M., Islam, M. S., Sassi, R., & Aktaruzzaman, M. (2019). Convolutional neural networks performance comparison for handwritten bengali numerals recognition. SN Applied Sciences, 1(12), 1–11.
39.
go back to reference LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.CrossRef LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.CrossRef
40.
go back to reference Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv preprint. arXiv:1712.04621. Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv preprint. arXiv:1712.04621.
41.
go back to reference Triesch, J., & Von Der Malsburg, C. (2001). A system for person-independent hand posture recognition against complex backgrounds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(12), 1449–1453.CrossRef Triesch, J., & Von Der Malsburg, C. (2001). A system for person-independent hand posture recognition against complex backgrounds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(12), 1449–1453.CrossRef
42.
go back to reference Rokade, Y. I., & Jadav, P. M. (2017). Indian sign language recognition system. International Journal of engineering and Technology, 9(3), 189–196.CrossRef Rokade, Y. I., & Jadav, P. M. (2017). Indian sign language recognition system. International Journal of engineering and Technology, 9(3), 189–196.CrossRef
43.
go back to reference Kaur, J., & Krishna, C. R. (2019). An efficient Indian sign language recognition system using sift descriptor. International Journal of Engineering and Advanced Technology (IJEAT), 8(6). Kaur, J., & Krishna, C. R. (2019). An efficient Indian sign language recognition system using sift descriptor. International Journal of Engineering and Advanced Technology (IJEAT), 8(6).
44.
go back to reference Kumar, D. A., Kishore, P. V. V., Sastry, A. S. C. S., & Swamy, P. R. G. (2016). Selfie continuous sign language recognition using neural network. In: 2016 IEEE Annual India Conference (INDICON), (pp. 1–6). Kumar, D. A., Kishore, P. V. V., Sastry, A. S. C. S., & Swamy, P. R. G. (2016). Selfie continuous sign language recognition using neural network. In: 2016 IEEE Annual India Conference (INDICON), (pp. 1–6).
45.
go back to reference Dour, G., & Sharma, S. (2016). Recognition of alphabets of indian sign language by Sugeno type fuzzy neural network. Pattern Recognit Lett, 30, 737–742. Dour, G., & Sharma, S. (2016). Recognition of alphabets of indian sign language by Sugeno type fuzzy neural network. Pattern Recognit Lett, 30, 737–742.
46.
go back to reference Athira, P. K., Sruthi, C. J., & Lijiya, A. (2019). A signer independent sign language recognition with co-articulation elimination from live videos: An indian scenario. Journal of King Saud University-Computer and Information Sciences. Athira, P. K., Sruthi, C. J., & Lijiya, A. (2019). A signer independent sign language recognition with co-articulation elimination from live videos: An indian scenario. Journal of King Saud University-Computer and Information Sciences.
47.
go back to reference Just, A., Rodriguez, Y., & Marcel, S. (2006). Hand posture classification and recognition using the modified census transform. In: 7th International Conference on Automatic Face and Gesture Recognition (FGR06), (pp. 351–356). Just, A., Rodriguez, Y., & Marcel, S. (2006). Hand posture classification and recognition using the modified census transform. In: 7th International Conference on Automatic Face and Gesture Recognition (FGR06), (pp. 351–356).
48.
go back to reference Kelly, D., McDonald, J., & Markham, C. (2010). A person independent system for recognition of hand postures used in sign language. Pattern Recognition Letters, 31(11), 1359–1368.CrossRef Kelly, D., McDonald, J., & Markham, C. (2010). A person independent system for recognition of hand postures used in sign language. Pattern Recognition Letters, 31(11), 1359–1368.CrossRef
49.
go back to reference Dahmani, D., & Larabi, S. (2014). User-independent system for sign language finger spelling recognition. Journal of Visual Communication and Image Representation, 25(5), 1240–1250.CrossRef Dahmani, D., & Larabi, S. (2014). User-independent system for sign language finger spelling recognition. Journal of Visual Communication and Image Representation, 25(5), 1240–1250.CrossRef
50.
go back to reference Kaur, B., Joshi, G., & Vig, R. (2017). Identification of ISL alphabets using discrete orthogonal moments. Wireless Personal Communications, 95(4), 4823–4845.CrossRef Kaur, B., Joshi, G., & Vig, R. (2017). Identification of ISL alphabets using discrete orthogonal moments. Wireless Personal Communications, 95(4), 4823–4845.CrossRef
51.
go back to reference Sahoo, J. P., Ari, S., & Ghosh, D. K. (2018). Hand gesture recognition using DWT and F-ratio based feature descriptor. IET Image Processing, 12(10), 1780–1787.CrossRef Sahoo, J. P., Ari, S., & Ghosh, D. K. (2018). Hand gesture recognition using DWT and F-ratio based feature descriptor. IET Image Processing, 12(10), 1780–1787.CrossRef
52.
go back to reference Joshi, G., Vig, R., & Singh, S. (2018). DCA-based unimodal feature-level fusion of orthogonal moments for Indian sign language dataset. IET Computer Vision, 12(5), 570–577.CrossRef Joshi, G., Vig, R., & Singh, S. (2018). DCA-based unimodal feature-level fusion of orthogonal moments for Indian sign language dataset. IET Computer Vision, 12(5), 570–577.CrossRef
Metadata
Title
Recognition of Indian Sign Language (ISL) Using Deep Learning Model
Authors
Sakshi Sharma
Sukhwinder Singh
Publication date
28-09-2021
Publisher
Springer US
Published in
Wireless Personal Communications / Issue 1/2022
Print ISSN: 0929-6212
Electronic ISSN: 1572-834X
DOI
https://doi.org/10.1007/s11277-021-09152-1

Other articles of this Issue 1/2022

Wireless Personal Communications 1/2022 Go to the issue