Matt Huenerfauth
Elaine Gale
Abstract
This research investigates how to best present video-based feedback information to students learning American Sign Language (ASL); these results are relevant not only for the design of a software tool for providing automatic feedback to students but also in the context of how ASL instructors could convey feedback on students’ submitted work. It is known that deaf children benefit from early exposure to language, and higher levels of written language literacy have been measured in deaf adults who were raised in homes using ASL. In addition, prior work has established that new parents of deaf children benefit from technologies to support learning ASL. As part of a long-term project to design a tool to automatically analyze a video of a students’ signing and provide immediate feedback about fluent and non-fluent aspects of their movements, we conducted a study to compare multiple methods of conveying feedback to ASL students, using videos of their signing. Through two user studies, with a Wizard-of-Oz design, we compared multiple types of feedback in regard to users’ subjective judgments of system quality and the degree students’ signing improved (as judged by an ASL instructor who analyzed recordings of students’ signing before and after they viewed each type of feedback). The initial study revealed that displaying videos to students of their signing, augmented with feedback messages about their errors or correct ASL usage, yielded higher subjective scores and greater signing improvement. Students gave higher subjective scores to a version in which time-synchronized pop-up messages appeared overlaid on the student's video to indicate errors or correct ASL usage. In a subsequent study, we found that providing images of correct ASL face and hand movements when providing feedback yielded even higher subjective evaluation scores from ASL students using the system.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Evaluation of Language Feedback Methods for Student Videos of American Sign Language
- Ben Bahan. 2016. The wolf who cried sheep. Online video. Retrieved from https://youtu.be/7Y44OUbwthQ.Google Scholar
- Helen Cooper, Brian Holt, and Richard Bowden. 2011. Sign language recognition. In Visual Analysis of Humans. Springer, 539--562. Google ScholarCross Ref
- Gallaudet Research Institute. 2011. Regional and National Summary Report of Data from the 2009--10 Annual Survey of Deaf and Hard of Hearing Children and Youth.Google Scholar
- David Goldberg, Dennis Looney, and Natalia Lusin. 2015. Enrollments in Languages other than English in US Institutions of Higher Education, Fall 2013. Modern Language Association.Google Scholar
- Susan Goldin-Meadow and Rachel I. Mayberry. 2001. How do profoundly deaf children learn to read? Learn Disabil Pract Res 16, 4, 222--229. Google ScholarCross Ref
- Foad Hamidi and Melanie Baljko. 2013. Automatic speech recognition: A shifted role in early speech intervention? In Proceedings of the SLPAT Workshop 2013. 55--61.Google Scholar
- Valerie Henderson, Seungyon Lee, Helene Brashear, Harley Hamilton, Thad Starner, and Steve Hamilton. 2005. Development of an ASL game for deaf children.In Proceedings of the 2005 Conference on Interaction Design and Children (Boulder, CO). Google ScholarDigital Library
- Matt Huenerfauth, Elaine Gale, Brian Penly, Mackenzie Willard, and Dhananjai Hariharan. 2015. Comparing methods of displaying language feedback for student videos of american sign language. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS’15). ACM, New York, 139--146. DOI=http://dx.doi.org/10.1145/2700648.2809859 Google ScholarDigital Library
- Pengfei Lu and Matt Huenerfauth. 2014. Collecting and evaluating the CUNY ASL corpus for research on American Sign Language animation. Comput. Speech Lang. 28, 3 (May 2014). 812--831. Elsevier. doi:10.1016/j.csl.2013.10.004 Google ScholarDigital Library
- Dawn MacLaughlin, Carol Neidle, and David Greenfield. 2000. SignStreamTM User's Guide, Version 2.0. American Sign Language Linguistic Research Project, Report Number 9, Boston University, Boston, MA. Retrieved from http://www.bu.edu/asllrp/reports.html#RPT9.Google Scholar
- Mark Marschark, Patricia Sapere, Carol M. Convertino, Connie Mayer, Loes Wauters, and Thomastine Sarchet. 2009. Are deaf students’ reading challenges really about reading? Am. Ann. Deaf 154, 4, 357--370. Google ScholarCross Ref
- Rachel I. Mayberry and Ellen B. Eichen. 1991. The long-lasting advantage of learning sign language in childhood: Another look at the critical period for language acquisition. J Mem. Lang. 30:486--498. Google ScholarCross Ref
- Ross E. Mitchell and Michael A. Karchmer. 2004. Chasing the mythical ten percent: Parental hearing status of deaf and hard of hearing students in the United States. Sign Lang Studies 4, 2, 138--163. Google ScholarCross Ref
- Ross E. Mitchell, Travas A. Young, Bellamie Bachleda, and Michael A. Karchmer. 2006. How many people use ASL in the United States? Why estimates need updating. Sign Lang Studies 6, 3, 306--335. Google ScholarCross Ref
- Caio D. D. Monteiro, Ricardo Gutierrez-Osuna, and Frank M. Shipman. 2012. Design and evaluation of classifier for identifying sign language videos in video sharing sites. In Proceedings of ASSETS’12. 191--198. DOI=10.1145/2384916.2384950 Google ScholarDigital Library
- Frank R. Lin, John K. Niparko, and Luigi Ferrucci. 2011. Hearing loss prevalence in the US. Arch. Intern. Med. 17, 1, 20, 1851--1852.Google Scholar
- Carol Neidle, Judy Kegl, Dawn MacLaughlin, Ben Bahan, and Robert G. Lee. 2000. The Syntax of ASL: Functional Categories and Hierarchical Structure. Cambridge: MIT Press.Google Scholar
- Russel S. Rosen. 2004. Beginning L2 production errors in ASL lexical phonology: A cognitive phonology model. Sign Language 8 Linguistics 7, 1, 31--61.Google Scholar
- Jenny L. Singleton and Elissa L. Newport. 2004. When learners surpass their models: The acquisition of American Sign Language from inconsistent input. Cognit. Psych. 49, 4 (Dec. 2004), 370--407. Google ScholarCross Ref
- Patricia Spencer and R. Lederberg. 1997. Different modes, different models: Communication and language of young deaf children and their mothers. In Communication and Language: Discoveries from Atypical Development. M. Romski (Ed.). Harvard University Press. 203--230.Google Scholar
- Michael Strong and Phillip M. Prinz. 1997. A study of the relationship between American Sign Language and English literacy. J. Deaf Stud. Deaf Educ. 2, 1, 37--46. Google ScholarCross Ref
- Carol Bloomquist Traxler. 2000. The Stanford achievement test, 9th edition: National norming and performance standards for deaf and hard-of-hearing students. J. Deaf Stud. Deaf Educ. 5, 4, 337--348.Google ScholarCross Ref
- Clayton Valli, Ceil Lucas, Kristin J. Mulrooney, and Miako Villanueva. 2011. Linguistics of American Sign Language: An Introduction. Gallaudet University Press.Google Scholar
- Haijing Wang, Alexandra Stefan, S. Moradi, Vassilis Athitsos, Carol Neidle, and F. Kamangar. 2010. A system for large vocabulary sign search. In Proceedings of the Workshop on Sign, Gesture and Activity.Google Scholar
- Kimberly A. Weaver, Thad Starner, and Harley Hamilton. 2010. An evaluation of video intelligibility for novice ASL learners on a mobile device. In Proceedings of ASSETS’10. 107--114. DOI=10.1145/1878803.1878824 Google ScholarDigital Library
- Kimberly A. Weaver and Thad Starner. 2011. We need to communicate: Helping hearing parents of deaf children learn American Sign Language. In Proceedings of ASSETS’11. 91--98. DOI=10.1145/2049536.2049554Google Scholar
- Ronnie B. Wilbur. 2000. The use of ASL to support the development of English and literacy. J. Deaf Stud. Deaf Educ. 5, 1, 81--104. Google ScholarCross Ref
Index Terms
- Evaluation of Language Feedback Methods for Student Videos of American Sign Language
Recommendations
Comparing Methods of Displaying Language Feedback for Student Videos of American Sign Language
ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & AccessibilityDeaf children benefit from early exposure to language, and higher levels of written language literacy have been measured in deaf adults who were raised in homes using American Sign Language (ASL). Prior work has established that new parents of deaf ...
Comparing native signers' perception of American Sign Language animations and videos via eye tracking
ASSETS '13: Proceedings of the 15th International ACM SIGACCESS Conference on Computers and AccessibilityAnimations of American Sign Language (ASL) have accessibility benefits for signers with lower written-language literacy. Our lab has conducted prior evaluations of synthesized ASL animations: asking native signers to watch different versions of ...
Effect of Displaying Human Videos During an Evaluation Study of American Sign Language Animation
Many researchers internationally are studying how to synthesize computer animations of sign language; such animations have accessibility benefits for people who are deaf and have lower literacy in written languages. The field has not yet formed a ...
Comments