Alex Olwal
Kevin Balke
ABSTRACT
Mobile solutions can help transform speech and sound into visual representations for people who are deaf or hard-of-hearing (DHH). However, where handheld phones present challenges, head-worn displays (HWDs) could further communication through privately transcribed text, hands-free use, improved mobility, and socially acceptable interactions.
Wearable Subtitles is a lightweight 3D-printed proof-of-concept HWD that explores augmenting communication through sound transcription for a full workday. Using a low-power microcontroller architecture, we enable up to 15 hours of continuous use. We describe a large survey (n=501) and three user studies with 24 deaf/hard-of-hearing participants which inform our development and help us refine our prototypes. Our studies and prior research identify critical challenges for the adoption of HWDs which we address through extended battery life, lightweight and balanced mechanical design (54 g), fitting options, and form factors that are compatible with current social norms.
Supplemental Material
Available for Download
srt
ufp1929pvc.srt (738 B)
Preview video captions
srt
ufp1929vfc.srt (3.7 KB)
Video figure captions
- Matthias Böhmer, T. Scott Saponas, and Jaime Teevan. 2013. Smartphone use does not have to be rude. Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services (MobileHCI '13), 342--351. https://doi.org/10.1145/2493190.2493237Google Scholar
Digital Library
- Ozan Cakmakci, Oscar Martinez, and Jerry Carollo. 2019. Optical design of a thin curved lightguide and manufacturing using ophthalmic approaches. Digital Optical Technologies 2019. https://doi.org/10.1117/12.2530526Google Scholar
- Leah Findlater, Bonnie Chinh, Dhruv Jain, Jon Froehlich, Raja Kushalnagar, and Angela Carey Lin. 2019. Deaf and Hard-of-hearing Individuals? Preferences for Wearable and Mobile Sound Awareness Technologies. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19), 1--13. https://doi.org/10.1145/3290605.3300276Google ScholarDigital Library
- Michele Fiorentino, Saverio Debernardis, Antonio E. Uva, and Giuseppe Monno. 2013. Augmented Reality Text Style Readability with See-Through Head-Mounted Displays in Industrial Context. Presence: Teleoperators and Virtual Environments 22, 171--190. https://doi.org/10.1162/pres_a_00146Google ScholarDigital Library
- Joseph L. Gabbard, J. Edward Swan, and Deborah Hix. 2006. The Effects of Text Drawing Styles, Background Textures, and Natural Lighting on Text Legibility in Outdoor Augmented Reality. Presence: Teleoperators and Virtual Environments 15, 16--32. https://doi.org/10.1162/pres.2006.15.1.16Google ScholarDigital Library
- Georgia Institute of Technology. 2014. Captioning on Glass: Enable Everyday Communication for the Hard-of-Hearing. Georgia Institute of Technology. Retrieved August 17, 2020 from http://cog.gatech.edu/Google Scholar
- Google. 2020. Analytics Solutions | Surveys. Google Surveys. Retrieved August 17, 2020 from https://surveys.withgoogle.com/Google Scholar
- Google. 2020. Opinion Rewards - It Pays to Share Your Opinion. Google Surveys. Retrieved August 17, 2020 from https://surveys.google.com/google-opinion-rewards/Google Scholar
- Google. 2020. Live Transcribe: Speech to Text App. Android. Retrieved August 17, 2020 from https://www.android.com/accessibility/live-transcribe/Google Scholar
- Google. 2020. Protocol Buffers - Google's data interchange format. Github. Retrieved August 17, 2020 from https://github.com/protocolbuffers/protobufGoogle Scholar
- Greta App. 2020. Barrier-free cinema and inclusion. Greta & Starks. Retrieved August 17, 2020 from http://www.gretaundstarks.de/greta/GretaAndStarksGoogle Scholar
- Scharon Harding. 2019. North Focals Review: Stealthy, Stylish Smart Glasses. Tom's Hardware. Retrieved August 17, 2020 from https://www.tomshardware.com/reviews/north-focals-smart-glasses-ar,5968.htmlGoogle Scholar
- James D. House. 2016. Telecommunications for the Deaf and Hard of Hearing, Inc., Formation of. The SAGE Deaf Studies Encyclopedia, 960--961. https://doi.org/10.4135/9781483346489.n310Google Scholar
- Dhruv Jain, Bonnie Chinh, Leah Findlater, Raja Kushalnagar, and Jon Froehlich. 2018. Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study. In Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems (DIS '18 Companion), 7--11. https://doi.org/10.1145/3197391.3205404Google ScholarDigital Library
- Dhruv Jain, Leah Findlater, Jamie Gilkeson, Benjamin Holland, Ramani Duraiswami, Dmitry Zotkin, Christian Vogler, and Jon E. Froehlich. 2015. Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 241-- 250. https://doi.org/10.1145/2702123.2702393Google ScholarDigital Library
- Dhruv Jain, Rachel Franz, Leah Findlater, Jackson Cannon, Raja Kushalnagar, and Jon Froehlich. 2018. Towards Accessible Conversations in a Mobile Context for People who are Deaf and Hard of Hearing. Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '18), 81--92. https://doi.org/10.1145/3234695.3236362Google ScholarDigital Library
- Dhruv Jain, Angela Lin, Rose Guttman, Marcus Amalachandran, Aileen Zeng, Leah Findlater, and Jon Froehlich. 2019. Exploring Sound Awareness in the Home for People who are Deaf or Hard of Hearing. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19), 1--13. https://doi.org/10.1145/3290605.3300324Google ScholarDigital Library
- Dhruv Jain, Kelly Mack, Akli Amrous, Matt Wright, Steven Goodman, Leah Findlater, and Jon E. Froehlich. 2020. HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing Users. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20), 1--12. https://doi.org/10.1145/3313831.3376758Google Scholar
- Michael Jones, M. Jeannette Lawler, Eric Hintz, Nathan Bench, Fred Mangrubang, and Mallory Trullender. 2014. Head mounted displays and deaf children. Proceedings of the 2014 conference on Interaction design and children (IDC '14) 317--320. https://doi.org/10.1145/2593968.2610481Google ScholarDigital Library
- Ichiro Kasai, Yasushi Tanijiri, Takeshi Endo and Hiroaki Ueda. A forgettable near eye display. Digest of Papers. Fourth International Symposium on Wearable Computers, 115--118. https://doi.org/10.1109/iswc.2000.888472Google ScholarCross Ref
- Elisa Maria Klose, Nils Adrian Mack, Jens Hegenberg, and Ludger Schmidt. 2019. Text Presentation for Augmented Reality Applications in Dual-Task Situations. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR '19), 636--644. https://doi.org/10.1109/vr.2019.8797992Google Scholar
- Marion Koelle, Swamy Ananthanarayan, and Susanne Boll. 2020. Social Acceptability in HCI: A Survey of Methods, Measures, and Design Strategies. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '19), 1--19. https://doi.org/10.1145/3313831.3376162Google ScholarDigital Library
- Catherine Kudlick. 2011. Black Bike, White Cane: Nonstandard Deviations of a Special Self. Disability Studies Quarterly, 31(1). https://doi.org/10.18061/dsq.v31i1.1373Google Scholar
- Walter S. Lasecki, Raja Kushalnagar, and Jeffrey P. Bigham. 2014. Helping students keep up with real-time captions by pausing and highlighting. Proceedings of the 11th Web for All Conference on (W4A '14), 1--8. https://doi.org/10.1145/2596695.2596701Google Scholar
- Tara Matthews, Scott Carter, Carol Pai, Janette Fong, and Jennifer Mankoff. 2006. Scribe4Me: evaluating a mobile sound transcription tool for the deaf. In Proceedings of the 8th international conference on Ubiquitous Computing (UbiComp'06), 159--176. https://doi.org/10.1007/11853565_10Google ScholarDigital Library
- Abby McCormack and Heather Fortnum. 2013. Why do people fitted with hearing aids not wear them?International Journal of Audiology 52, 360--368. https://doi.org/10.3109/14992027.2013.769066Google Scholar
- MediaTek. MT2523D for Wearables: Highly integrated and efficient wearable-focused chip with power management unit and dual-mode Bluetooth. MediaTek. Retrieved August 17, 2020 from http://www.mediatek.com/products/wearables/mt2523dGoogle Scholar
- Microsoft. 2020. Microsoft Translator for Consumers. Microsoft. Retrieved August 17, 2020 from https://www.microsoft.com/en-us/translator/Google Scholar
- Ashley Miller, Joan Malasig, Brenda Castro, Vicki L. Hanson, Hugo Nicolau, and Alessandra Brandão. 2017. The Use of Smart Glasses for Lecture Comprehension by Deaf and Hard of Hearing Students. Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17), 1909--1915. https://doi.org/10.1145/3027063.3053117Google ScholarDigital Library
- Shalini Misra, Lulu Cheng, Jamie Genevie, and Miao Yuan. 2016. The iPhone Effect: The Quality of In-Person Social Interactions in the Presence of Mobile Devices. Environment and behavior 48, 2: 275--298. https://doi.org/10.1177/0013916514539755Google Scholar
- National Association of the Deaf. 2020. CART: Communication Access Realtime Translation. National Association of the Deaf. Retrieved August 17, 2020 from https://www.nad.org/resources/technology/captioning-for-access/communication-access-realtime-translation/Google Scholar
- Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu, Tzu-Chuan Chen, Te-Yen Wu, Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen. 2018. SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-of-Hearing People in Group Conversations. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 1--10. https://doi.org/10.1145/3173574.3173867Google ScholarDigital Library
- Halley Profita, Reem Albaghli, Leah Findlater, Paul Jaeger, and Shaun K. Kane. 2016. The AT Effect. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 4884--4895. https://doi.org/10.1145/2858036.2858130Google Scholar
- Rachel Rood. 2013. New Closed-Captioning Glasses Help Deaf Go Out To The Movies. NPR.org. Retrieved August 17, 2020 from https://www.npr.org/sections/alltechconsidered/2013/0 5/12/183218751/new-closed-captioning-glasses-help-deaf-go-out-to-the-moviesGoogle Scholar
- Chris Schipper and Bo Brinkman. 2017. Caption Placement on an Augmented Reality Head Worn Device. Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17), 365--366. https://doi.org/10.1145/3132525.3134786Google ScholarDigital Library
- Kristen Shinohara and Jacob O. Wobbrock. 2011. In the shadow of misperception. Proceedings of the 2011 annual conference on Human factors in computing systems (CHI '11), 705--714. https://doi.org/10.1145/1978942.1979044Google Scholar
- Mark B. Spitzer, Noa M. Rensing, Robert McClelland, and Paul Aquilino. Eyeglass-based systems for wearable computing. Digest of Papers. First International Symposium on Wearable Computers, 48-- 51. https://doi.org/10.1109/iswc.1997.629918Google ScholarCross Ref
- Mark Spitzer. Personal Communication. 2011.Google Scholar
- Texas Instruments. 2020. BQ24230: USB Friendly 0.5A Li-ion charger. Texas Instruments. Retrieved August 17, 2020 from https://www.ti.com/product/BQ24230Google Scholar
- Aaron Toney, Barrie Mulley, Bruce H. Thomas, and Wayne Piekarski. 2003. Social weight: designing to minimise the social consequences arising from technology use by the mobile professional. Personal and Ubiquitous Computing 7, 309--320. https://doi.org/10.1007/s00779-003-0245--8Google ScholarDigital Library
- Hubert W. Upton. 1968. Wearable eyeglass speechreading aid. American annals of the deaf 113, 2: 222--229.Google Scholar
- Mordekhai Velger. 1998. Helmet-mounted Displays and Sights. Artech House Publishers. ISBN-13: 978-0890068229.Google Scholar
- Stefan Werrlich, Phuc-Anh Nguyen, and Gunther Notni. 018. Evaluating the training transfer of Head-Mounted Display based training for assembly tasks. In Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference (PETRA '18), 297--302. https://doi.org/10.1145/3197768.3201564Google Scholar
- What is CART and FAQ?s -- Find a CART Captioning Provider. 2012. Captioning Activism and Community. Retrieved August 17, 2020 from http://ccacaptioning.org/faqs-cart/Google Scholar
- Winbond. 2020. W25N01GV: (1G-bit) Serial SLC NAND Flash Memory. Winbond. Retrieved August 17, 2020 from https://www.winbond.com/resource-files/w25n01gv%20revg%20032116.pdfGoogle Scholar
Index Terms
- Wearable Subtitles: Augmenting Spoken Communication with Lightweight Eyewear for All-day Captioning
Recommendations
Implementing Assistive Technology on Wearable Computers
The article presents the author's work in developing assistive technology for disabled users. He makes a case for wearable computers and focuses on design strategies that accommodate the users' changing needs. He points out that what might seem ...
Tactile Board: A Multimodal Augmentative and Alternative Communication Device for Individuals with Deafblindness
MUM '20: Proceedings of the 19th International Conference on Mobile and Ubiquitous MultimediaDeafblindness, also known as dual sensory loss, is the combination of sight and hearing impairments of such extent that it becomes difficult for one sense to compensate for the other. Communication issues are a key concern for the Deafblind community. ...
Real-time Activity-sensitive Wearable Ankle Edema Monitoring System For Elderly and Visually Impaired Heart Failure Patients
ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and AccessibilityHeart failure (HF) is a leading cause of hospital admissions and readmissions, and carries significant morbidity and mortality in the elderly patient population. Worsening ankle edema is an early sign of an acute HF exacerbation and needs to be ...
Comments