Top

Universal Access in the Information Society

Published in:

03-11-2016 | Long Paper

Signing avatars: making education more inclusive

Authors: José Mario De Martino, Ivani Rodrigues Silva, Carmen Zink Bolognini, Paula Dornhofer Paro Costa, Kate Mamhy Oliveira Kumada, Luis Cláudius Coradine, Patrick Henrique da Silva Brito, Wanessa Machado do Amaral, Ângelo Brandão Benetti, Enzo Telles Poeta, Leandro Martin Guertzenstein Angare, Carolina Monteiro Ferreira, Davi Faria De Conti

Published in: Universal Access in the Information Society | Issue 3/2017

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In Brazil, there are approximately 9.7 million inhabitants who are deaf or hard of hearing. Moreover, about 30% of the Brazilian deaf community is illiterate in Brazilian Portuguese due to difficulties to offer deaf children an inclusive environment based on bilingual education. Currently, the prevailing teaching practice depends heavily on verbal language and on written material, making the inclusion of the deaf a challenging task. This paper presents the author’s approach for tackling this problem and improving deaf students’ accessibility to written material in order to help them master Brazilian Portuguese as a second language. We describe an ongoing project aimed at developing an automatic Brazilian Portuguese-to-Libras translation system that presents the translated content via an animated virtual human, or avatar. The paper describes the methodology adopted to compile a source language corpus having the deaf student needs in central focus. It also describes the construction of a parallel Brazilian Portuguese/Brazilian Sign Language (Libras) corpus based on motion capture technology. The envisioned translation architecture includes the definition of an Intermediate Language to drive the signing avatar. The results of a preliminary assessment of signs intelligibility highlight the application potential.

previous article Development of a web-based insulin decision aid for the elderly: usability barriers and guidelines

next article Designing video games for the blind: results of an empirical study

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Almohimeed, A., Wald, M,. Damper, R.I.: Arabic text to arabic sign language translation system for the deaf and hearing-impaired community. In: Proceedings of the Second Workshop on Speech and Language Processing for Assistive Technologies, Association for Computational Linguistics, Stroudsburg, PA, USA, SLPAT’11, pp. 101–109 (2011)

Amaral, W.M.: Sistema de transcrição da língua brasileira de sinais voltado à produção de conteúdo sinalizado por avatares 3D. PhD thesis, Faculdade de Engenharia Elétrica e de Computação—Universidade Estadual de Campinas (2012)

Amaral, W.M., De Martino, J.M.: Towards a transcription system of sign language for 3D virtual agents. In: Tarek, Sobh (ed.) Innovations in Computing Sciences and Software Engineering, pp. 85–90. Springer, Netherlands (2010)CrossRef

Amaral, W.M., Angare, L., De Martino, J.M.: Sign Language 3D Virtual Agent. In: Proceedings of the 5th International Multi-Conference on Society, Cybernetics and Informatics (IMSCI 2011), International Institute of Informatics and Systemics, Orlando, Florida, vol. I, pp. 93–97 (2011)

Baker, M.: Corpus linguistics and translation studies: implications and applications. In: Baker, M., Francis, G., Tognini-Bonelli, E. (eds.) Text and Technology: In Honour of John Sinclair, pp. 233–248. John Benjamins, Amsterdam (1993)CrossRef

Brasil: Lei no. 9.394. In: Diário Oficial União, Imprensa Oficial República Federativa do Brasil, Brasília, DF, pp. 1–9 (1996)

Brasil: Lei no. 10.436. In: Diário Oficial União, Imprensa Oficial República Federativa do Brasil, Brasília, DF, p. 23 (2002)

Brasil: Decreto no. 5.626. In: Diário Oficial União, Imprensa Oficial República Federativa do Brasil, Brasília, DF, pp. 28–30 (2005)

Breda, W.L., Caon, D.R.S., Brito, P.H.S., Tavares, O.L., Meneses, C., Coradine, L.C., Albuquerque, F.C.: SOTAC: a software for knowledge-based automatic translation. In: Proceedings of the 9th IFIP World Conference on Computers in Education, Springer, Bento Gonçalves, RS, Brazil, vol. 1, pp. 1–10 (2009)

10.

Brito, P.H.S., Franco, N., Coradine, L.C.: Falibras: Uma Ferramenta Flexível para Promover Acessibilidade de Pessoas Surdas. In: Anais del XVII Congreso Internacional de Informática Educativa (TISE 2012), Universidad de Chile, Santiago, Chile, pp. 87–96 (2012)

11.

Bungeroth, J., Ney, H.: Statistical sign language translation. In: Workshop on Representation and Processing of Sign Languages, LREC vol. 4, 105–108 (2004)

12.

Capovilla, F.C., Raphael, W.D.: Dicionário Enciclopédico Ilustrado Trilíngue da Língua de Sinais Brasileira \(Libras\), 3rd edn. EDUSP, São Paulo, SP, Brazil (2008)

13.

Capovilla, F.C., Raphael, W.D., Mauricio, A.C.: Novo Deit-libras: Dicionário Enciclopédico Ilustrado Trilingue da Língua de Sinais Brasileira \(Libras\). EDUSP, São Paulo, SP (2009)

14.

Coradine, L.C., Albuquerque, F.C., Brito, P.H.S., Silva, R.L., Silva, T.F.L.: Sistema Falibras: interpretação animada, em Libras, de palavras e expressões em português. In: Anais do III Congresso Ibero-Americano de Informática na Educação Especial (CIIEE 2002), Fortaleza/CE, Brazil, vol. 1 (2002)

15.

Coradine, L.C., Tavares, O.L., Albuquerque, F.C., Ribeiro, M.N., Breda, W.L., Calado, I.A.A.R., Smarsaro, A.D.: Sistema Falibras: Um intérprete virtual como ferramenta de apoio pedagógico à educação de surdos. In: Anais do VII Congresso Ibero-Americano de Informática Educativa Especial (CIIEE 2007), Mar del Plata-Argentina, vol. 1, pp. 1–10 (2007)

16.

Côrrea, Y., Vieira, M.C., Santarosa, L.M.C., Biasuz, M.C.V.: Aplicativos de tradução automática como facilitadores da comunicaçã entre surdos e ouvintes. In: Santarosa, L.M.C., Conforto, D., Vieira, M.C. (eds.) Tecnologia e Acessibilidade: Passos em direção à inclusão escolar e sociodigital. Evangraf, Porto Alegre (2014)

17.

Cox, S., Lincoln, M., Tryggvason, J., Nakisa, M., Wells, M., Tutt, M., Abbott, S.: Tessa, a system to aid communication with deaf people. In: Proceedings of the Fifth International ACM Conference on Assistive Technologies, ACM, pp. 205–212 (2002)

18.

Darwiche, A.: Bayesian networks. Commun. ACM 53(12), 80–90 (2010)CrossRefMATH

19.

De Martino, J.M., Costa, P.D.P., Benetti, A., Rosa, L.A., Kumada, K.M.O., Silva, I.R.: Building a brazilian portuguese-brazilian sign language parallel corpus using motion capture data. In: Proceedings of Workshop of Corpora and Tools for Processing Corpora, pp. 56-63 (2016)

20.

Editora Moderna (Org): Projeto Pitanguá, 1st edn. Moderna, São Paulo (2005)

21.

Elliott, R., Glauert, J.R., Kennaway, J., Marshall, I., Safar, E.: Linguistic modelling and language-processing technologies for Avatar-based sign language presentation. Univers. Access Inf. Soc. 6(4), 375–391 (2008)CrossRef

22.

Felipe, T.A.: Introdução à Gramática de Libras. Ministério da Educação e do Desporto. Secretaria de Educação Especial. Educação especial: Deficiência Auditiva, Brasília, DF (1997)

23.

Forster, J., Schmidt, C., Hoyoux, T., Koller, O., Zelle, U., Piater, J.H., Ney, H.: RWTH-PHOENIX-weather: a large vocabulary sign language recognition and translation corpus. In: LREC, pp. 3785–3789 (2012)

24.

Gibet, S., Courty, N., Duarte, K., Naour, T.L.: The SignCom system for data-driven animation of interactive virtual signers: methodology and evaluation. ACM Trans. Interact. Intell. Syst. 1(1), 6:1–6:23 (2011)CrossRef

25.

Goés, M.C.R.d., Tartuci, D.: Alunos surdos e experiências de letramento. In: Fernades E, Dorzial A, Lodi ACB (eds.) Letramento, Bilinguismo e Educação de Surdos, 2nd edn. Mediação Editora, Porto Alegre, pp. 289–302 (2015)

26.

Gottlieb, G.L., Song, Q., Almeida, G.L., Hong, D.A., Corcos, D.: Directional control of planar human arm movement. J. Neurophysiol 78, 2985–2998 (1997)

27.

Hirashima, M., Kudo, K., Watarai, K., Ohtsuki, T.: Control of 3D Limb Dynamics in unconstrained overarm throws of different speeds performed by skilled baseball players. J. Neurophysiol. 97(1), 680–691 (2007)CrossRef

28.

Huenerfauth, M.: A multi-path architecture for machine translation of English text into American sign language animation. In: Proceedings of the Student Research Workshop at HLT-NAACL 2004, Association for Computational Linguistics, Stroudsburg, PA, USA, HLT-SRWS ’04, pp. 25–30 (2004)

29.

IBGE: Censo Demográfico 2010—Características gerais da população, religião e pessoas com deficiência. Instituto Brasileiro de Geografia e Estatística, Ministério do Planejamento, Orçamento e Gestão e Instituto Brasileiro de Geografia e Estatística, Rio de Janeiro, Brasil (2010)

30.

Imamura, K., Okuma, H., Watanabe, T., Sumita, E.: Example-based machine translation based on syntactic transfer with statistical models. In: Proceedings of COLING, pp. 99–105 (2004)

31.

Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of hand configuration: the fingers. Sign Lang. Stud. 12(1), 5–45 (2011)CrossRef

32.

Johnson, R.E., Liddell, S.K.: A segmental framework for representing signs phonetically. Sign Lang. Stud. 11(3), 408–463 (2011)CrossRef

33.

Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of signs: sequentiality and contrast. Sign Lang. Stud. 11(2), 241–274 (2011)CrossRef

34.

Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of hand configuration: the thumb. Sign Lang. Stud. 12(2), 316–333 (2012)CrossRef

35.

Liddell, S.K., Johnson, R.E.: American sign language: the phonological base. Sign Lang. Stud. 64, 195–278 (1989)CrossRef

36.

Linde-Usiekniewicz, J., Czajkowska-Kisil, M., Łacheta, J., Rutkowski, P.: A corpus-based dictionary of polish sign language (PJM). In: Proceedings of the XVI EURALEX International Congress: The User in Focus (2014)

37.

Lira, G.A., de Souza, T.A.F.: Dicionário da língua brasileira de sinais. http://www.acessobrasil.org.br/libras/ (2008). Accessed August 2016 2008

38.

Lu, P., Huenerfauth, M.: Collecting a motion-capture corpus of American sign language for data-driven generation research. In: Proceedings of the NAACL HLT 2010 Workshop on Speech and Language Processing for Assistive Technologies, Association for Computational Linguistics, Stroudsburg, PA, USA, SLPAT ’10, pp. 89–97 (2010)

39.

Marotta, J.J., Medendorp, W.P., Crawford, J.D.: Kinematic rules for upper and lower arm contributions to grasp orientation. J. Neurophysiol. 90(6), 3816–3827 (2003)CrossRef

40.

Martin, P.J.M., Belhe, S., Mudliar, S., Kulkarni, M., Sahasrabudhe, S.: An Indian Sign Language (ISL) corpus of the domain disaster message using Avatar. In: Proceedings of the Third International Symposium in Sign Language Translations and Technology (SLTAT-2013), pp. 1–4 (2013)

41.

Massó, G., Badia, T.: Dealing with sign language morphemes in statistical machine translation. In: 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, Valletta, Malta, pp. 154–157 (2010)

42.

McCleary, L., Viotti, E.: Transcrição de dados de uma língua sinalizada: um estudo piloto da transcrição de narrativas na língua de sinais brasileira (LSB). In: Salles, H. (ed.) Bilinguismo e surdez: Questões linguísticas e educacionais, pp. 73–96. Cânone Editorial, Goiania (2007)

43.

McCracken, D.D., Reilly, E.D.: Backus-Naur form (BNF). In: Ralston, A., Reilly, E.D., Hemmendinger, D. (eds.) Encyclopedia of Computer Science, 4th edn, pp. 129–131. Wiley, Chichester (2003)

44.

Meir, I., Sandler, W., Padden, C., Aronoff, M.: Emerging Sign Languages, vol. 2. Oxford University Press, Oxford, pp. 267–280 (Chapter 18) (2010)

45.

Morrissey, S., Way, A.: An example-based approach to translating sign language. In: Second Workshop on Example-Based Machine Translation, Dublin City University, vol. 16, pp. 109–116 (2005)

46.

New London Group: A pedagogy of multiliteracies: designing social futures. In: Cope, B., Kalantzis, M. (eds.) Multiliteracies: Literacy Learning and the Design of Social Futures, pp. 9–38. Macmillan, South Yarra (2000)

47.

Offutt, J.: Quality attributes of web software applications. IEEE Softw. 19(2), 25–32 (2002)CrossRef

48.

Othman, A., Jemni, M.: Statistical sign language machine translation: from English written text to American sign language gloss. Int. J. Comput. Sci. Issues 8(3), 65–73 (2011)

49.

Pereira, M.C.d.C., Choi, D., Vieira, M.I., Gaspar, P., Nakasato, R.: Libras: conhecimento além dos sinais, 1st edn. Pearson Prentice Hall, São Paulo (2011)

50.

Porta, J., López-Colino, F., Tejedor, J., Colás, J.: A rule-based translation from written Spanish to Spanish Sign Language glosses. Comput. Speech Lang. 28(3), 788–811 (2014)CrossRef

51.

Probst, K., Levin, L., Peterson, E., Lavie, A., Carbonell, J.: MT for minority languages using elicitation-based learning of syntactic transfer rules. Mach. Transl. 17(4), 245–270 (2002)CrossRef

52.

Reily, L.H.: As imagens: o lúdico e o absurdo no ensino de arte para pré-escolares surdos. In: Silva, I.R., Kauchakje, S., Gesueli, Z.M. (eds.) Cidadania, surdez e linguagem: desafios e realidades, pp. 161–192. Plexus Editora, Sã Paulo (2003)

53.

Rojo, R.: Gêneros discursivos do Círculo de Bakhtin e multiletramentos. In: Rojo, R.H.R. (ed) Escol@ conectada: os multiletramentos e as TICs, Parábola Editorial, São Paulo, pp. 13–36 (2013)

54.

San-Segundo, R., Montero, J.M., Córdoba, R., Sama, V., Fernández, F., D’Haro, L.F., López-Ludeña, V., Sánchez, D., García, A.: Design, development and field evaluation of a Spanish into sign language translation system. Pattern Anal. Appl. 15(2), 203–224 (2012)MathSciNetCrossRef

55.

Schermer, T., Koolhof, C.: The first national Dutch Sign Language (NGT) Dictionary in book form: Van Dale Basiswoordenboek Nederlandse Gebarentaal. In: Dykstra, A., Schoonheim, T. (eds.) Proceedings of the 14th EURALEX International Congress, pp. 1555–1564. Fryske Akademy, Leeuwarden/Ljouwert, The Netherlands (2010)

56.

Schnieder, L., Tschare, G.: Passenger information using a sign language avatar–individual travel assistance for passengers with special needs in public transport. Int. Transp. 67(1), 8–10 (2015)

57.

Silva, I.R., Kumada, K.M.O.: Representações sobre o contexto multilíngue da surdez. Interdisciplinar–Revista de Estudos em Língua e Literatura 19, 99–114 (2013)

58.

Silva, I.R., Nogueira, A.S., Hildebrand, H.R., Kumada, K.M.O.: O uso de jogos eletrônicos no processo ensino-aprendizagem de surdos. In: Valle, L.E.L.R., Matos, M.J.V.M., Costa, J.W. (eds.) Educação digital: a tecnologia a favor da inclusão, pp. 213–238. Penso, Porto Alegre (2013)

59.

Siple, P.: Attentional resources and working memory: a new framework for the study of the impact of deafness on cognition. In: Emmorey, K., Lane, H. (eds.) The Signs of Language Revisited: An Anthology to Honor Ursula Bellugi and Edward Klima, pp. 115–134. Lawrence Erlbaum Associates, London (2000)

60.

Soechting, J.F., Buneo, C.A., Herrmann, U., Flanders, M.: Moving effortlessly in three dimensions: does Donders’ law apply to arm movement? J. Neurosci. 15, 6271–6280 (1995)

61.

Somers, H., Diaz, G.F.: Translation memory vs. example-based MT: what is the difference? Int. J. Trans. 16, 5–33 (2004)

62.

Stein, D., Schmidt, C., Ney, H.: Analysis, preparation, and optimization of statistical sign language machine translation. Mach.Transl. 26(4), 325–357 (2012)CrossRef

63.

Stokoe Jr, W.C.: Sign language structure: an outline of the visual communication system of the American deaf. Studies in Linguistics: Occasional Papers 8, pp. 1–78 (Revised 1978) (1960)

64.

Sumita, E., Iida, H., Kohyama, H.: Translating with examples: a new approach to machine translation. In: Third International Conference on Theoretical and Methodological Issues in Machine Translation of Natural Language. vol. 3, 203–212 (1990)

65.

Suszczańska, N., Szmal, P., Francik, J.: Translating polish texts into sign language in the TGT system. In: 20th IASTED International Multi–Conference Applied Informatics AI, Innsbruck, Austria, pp. 282–287 (2002)

66.

Sze, F., Woodward, J., Tang, G., Lee, J., Cheng, K., Mak, J.: Sign language documentation in the Asia–Pacific region: a deaf-centered approach. In: Proceedings of the 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon, Language Resources and Evaluation Conference (LREC) 2012, pp. 155–158 (2012)

67.

Umeda, S., Azuma, M., Uchida, T., Miyazaki, T., Kato, N., Inoue, S., Hiruma, N.: Automatic translation system to JSL (Japanese Sign Language) about weather information. In: Proceedings of the Sign Language Translation and Avatar Technology (SLTAT 2015) (2015)

68.

Uno, Y., Kawato, M., Suzuki, R.: Formation and control of optimal trajectory in human multijoint arm movement. Biol. Cybern. 61, 89–101 (1989)CrossRef

69.

Van Zijl, L., Baker, D.: South African sign language machine translation system. In: Proceedings of the 2nd International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa, Association for Computing Machinery-ACM, pp. 49–52 (2003)

70.

Veale, T., Conway, A.: Cross modal comprehension in ZARDOZ an English to sign-language translation system. In: Proceedings of the Seventh International Workshop on Natural Language Generation, Association for Computational Linguistics, pp. 249–252 (1994)

71.

Verlinden, M., Tijsseling, C., Frowein, H.: A signing Avatar on the WWW. In: Wachsmuth, I., Sowa, T. (eds.) Gesture and Sign Language in Human–Computer Interaction. Lecture Notes in Computer Science, vol. 2298, pp. 169–172. Springer, Berlin (2002)

72.

Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., Sloetjes, H.: Elan: a professional framework for multimodality research. In: Proceedings of LREC, Fifth International Conference on Language Resources and Evaluation, pp. 1556–1559 (2006)

73.

Xu, H.Y., Li, D., Tang, S.: Kinematic Inbetweening along implicit curves for motion animations. In: Proceedings of the 16th International Conference on Artificial Reality and Telexistence–Workshops, IEEE Computer Society, Washington, DC, USA, ICAT ’06, pp. 1–4 (2006)

74.

Zhao, L., Kipper, K., Schuler, W., Vogler, C., Badler, N.I., Palmer, M.: A machine translation system from English to American sign language. In: Proceedings of the 4th Conference of the Association for Machine Translation in the Americas on Envisioning Machine Translation in the Information Future, Springer-Verlag, London, UK, AMTA ’00, pp. 54–67 (2000)

Title: Signing avatars: making education more inclusive
Authors: José Mario De Martino
Ivani Rodrigues Silva
Carmen Zink Bolognini
Paula Dornhofer Paro Costa
Kate Mamhy Oliveira Kumada
Luis Cláudius Coradine
Patrick Henrique da Silva Brito
Wanessa Machado do Amaral
Ângelo Brandão Benetti
Enzo Telles Poeta
Leandro Martin Guertzenstein Angare
Carolina Monteiro Ferreira
Davi Faria De Conti
Publication date: 03-11-2016
Publisher: Springer Berlin Heidelberg
Published in: Universal Access in the Information Society / Issue 3/2017
Print ISSN: 1615-5289
Electronic ISSN: 1615-5297
DOI: https://doi.org/10.1007/s10209-016-0504-x

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3/2017

Evaluation of in-air hand gestures interaction for older people

Technology learning and use among older adults with late-life vision impairments

Usability evaluation of navigation tasks by people with intellectual disabilities: a Google and SAPO comparative study regarding different interaction modalities

A mobile instructional pervasive game method for language learning

Externalizing cognitive maps via map reconstruction and verbal description

Promoting interactive television (iTV) accessibility: an adapted service for users with visual impairments

Premium Partner