Francis Quek
Skip Abstract Section
Abstract
Human discourse is an embodied activity emerging from the embodied imagery and construction of our talk. Gesture and speech are coexpressive, conveying this imagery and meaning simultaneously. Mathematics instruction and discourse typically involve two modes of communication: speech and graphical presentation. Our goal is to assist Individuals who are Blind or Severely Visually Impaired (IBSVI) to access such instruction/communication. We employ a haptic glove interface to furnish the IBSVI with awareness of the deictic gestures performed by the instructor over the graphic in conjunction with speech. We present a series of studies spanning two years where we show how our Haptic Deictic System (HDS) can support learning in inclusive classrooms where IBSVI receive instruction alongside sighted students. We discuss how the introduction of the HDS was advantageous to all parties: IBSVI, instructor, and sighted students. The HDS created more learning opportunities, increasing mutual understanding and promoting greater engagement.
- Alibali, M. W. 2005. Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cogn. Comput. Interdiscipl. J. 5, 4, 307--331.Google Scholar
Cross Ref
- Alibali, M. W. and Nathan, M. J. 2007. Teachers' gestures as a means of scaffolding students' understanding: Evidence from an early algebra lesson. In Video Research in the Learning Sciences, R. Goldman, B. B. R. Pea, and S. J. Derry, Eds., Erlbaum, Mahwah, NJ.Google Scholar
- Archambault, D., Ossmann, R., Gaudy, T., and Miesenberger, K. 2007. Computer games and visually impaired people. Upgrade 8, 2, 43--53.Google Scholar
- Bach-Yrita, P., Collins, C., Saunders, F., White, B., and Scadden, L. 1969. Vision substitution by tactile image projection. Nature 221, 5184, 963--964.Google Scholar
- Baear, R., Flexer, R., and Mcmahan, R. K. 2005. Transition models and promising practices. In Transition Planning for Secondary Students with Disabilities. 53--82.Google Scholar
- Baker, E., Wang, M., and Walberg, H. 1994. The effects of inclusion on learning. Educ. Leader. 52, 4, 32--35.Google Scholar
- Beattie, G. 2003. Visible Thought: The New Psychology of Body Language. Routledge, London.Google Scholar
- Brophy, J. and Good, T. 1986. Teacher behavior and student achievement. In Handbook of Research on Teaching 3rd Ed. 328--375.Google Scholar
- Cancelli, A. 1993. Type of instruction and the relationship of classroom behavior to achievement among learning disabled children. J. Classroom Interact. 28, 1, 13--21.Google Scholar
- Chen, J. 2007. Flow in games (and everything else). Comm. ACM 50, 4, 31--34. Google ScholarDigital Library
- Christman, R. J. 1979. Sensory Experience. Harper and Row Publishers.Google Scholar
- Clark, H. and Marshall, C. 2002. Definite reference and mutual knowledge. In Psycholinguistics: Critical Concepts in Psychology. 414.Google Scholar
- Clark, H. and Wilkes-Gibbs, D. 1990. Referring as a collaborative process. In Intentions in Communication, MIT Press, 463--493.Google Scholar
- Clark, H. H. 1992. Arenas of Language Use. University of Chicago Press.Google Scholar
- Clark, H. H. 1996. Using Language. Cambridge University Press, Cambridge, UK.Google Scholar
- Clark, H. H. and Brennan, S. E. 1991. Grounding in communication. In Perspectives on Socially Shared Cognition, L. B. Resnick, J. M. Levine, and S. D. Teasley, Eds., American Phycological Association, Washignton, DC.Google Scholar
- Clark, H. H. and Krych, M. 2003. Speaking while monitoring addressees for understanding. J. Mem. Lang. 50, 62--85.Google ScholarCross Ref
- Csikszentmihalyi, M. 1975. Play and intrinsic rewards. J. Humanist. Psychol. 15, 3, 41--63.Google ScholarCross Ref
- Department of Health. 2007. Certificate of vision impairment: Explanatory notes for consultant ophthalmologists and hospital eye clinic staff. Tech. rep., Social Care Policy & Innovation--Person Involvement and Independent Living, U.S. Department of Health.Google Scholar
- Dick, T. and Evelyn, K. 1997. Issues and aids for teaching mathematics to the blind. Math. Teacher 90, 5, 344--349.Google Scholar
- Dourish, P. 2001. Where the Action Is: The Foundations of Embodied Interaction. MIT Press. Google ScholarDigital Library
- Fang, B., Oliveira, F., and Quek, F. 2010. Using vision based tracking to support real-time graphical instruction for student who have visual impairments. In Proceedings of the 3rd Workshop on Computer Vision Applications for the Visually Impaired (CVAVI'10). 9--14.Google Scholar
- Fisk, A. and Schneider, A. 1981. Controlled and automatic processing during tasks requiring sustained attention. Hum. Factors 23, 737--750.Google ScholarCross Ref
- Fredrick, W. 1977. The use of classroom time in high schools above or below the median reading score. Urban Educ. 11, 4, 459.Google ScholarCross Ref
- Gallace, A., Tan, H., and Spence, C. 2007. The body surface as a communication system: The state of the art after 50 years. PRESENCE: Teleop. Virt. Environ. 16, 6, 655--676. Google ScholarDigital Library
- Gilliland, K. and Schlegel, R. 1994. Tactile stimulation of the human head for information display. Hum. Factors: J. Hum. Factors Ergon. Society 36, 4, 700--717.Google ScholarCross Ref
- Godthelp, J. and Schumann, J. 1993. Intelligent accelerator: An element of driver support. In Driving Future Vehicles, 265--275.Google Scholar
- Goldin-Meadow, S. 1999. The role of gesture in communication and thinking. Trends Cogn. Sci. 3, 11, 419--429.Google ScholarCross Ref
- Goldin-Meadow, S. 2003. Hearing Gesture: How Our Hands Help Us Think. Belknap Press.Google Scholar
- Harrison, S., Tatar, D., and Sengers, P. 2007. The three paradigms of hci. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.Google Scholar
- Hinrichs, E. and Polanyi, L. 1986. Pointing the way: A unified treatment of referential gesture in interactive discourse. In Papers from the Parasession on Pragmatics and Grammatical Theory, Chicago Linguistics Society 22nd Meeting. 71--78.Google Scholar
- Ho, C., Tan, H., and Spence, C. 2005. Using spatial vibrotactile cues to direct a driver's visual attention. Transport. Res. F-Traffic Psychol. Behav. 8, 397--412.Google ScholarCross Ref
- Hodzic, A., Veit, R., Karim, A., Erb, M., and Godde, B. 2004. Improvement and decline in tactile discrimination behavior after cortical plasticity induced by passive tactile coactivation. J. Neurosci. 24, 2, 442.Google ScholarCross Ref
- Iglesias, R., Casado, S., Gutierrez, T., Barbero, J., Avizzano, C., Marcheschi, S., Bergamasco, M., Labein, F., and Derio, S. 2004. Computer graphics access for blind people through a haptic and audio virtual environment. In Proceedings of the the 3rd IEEE International Workshop on Haptic, Audio and Visual Environments and Their Applications (HAVE'04). 13--18.Google Scholar
- Inman, D., Loge, K., and Cram, A. 2000. Teaching orientation and mobility skills to blind children using computer generated 3-d sound environments. In Proceedings of the International Community for Auditory Display.Google Scholar
- Inman, D., Peaks, J., Loge, K., Chen, V., and Ferrington, G. 1994. Virtual reality training program for motorized wheelchair operation. Tech. rep., California State University, Northridge Center on Disabilities, Los Angeles.Google Scholar
- International Council of Ophthalmology. 2002. International standards: Visual standards—Aspects and ranges of vision loss with emphasis on population surveys. Tech. rep., 29th International Congress of Ophthalmology.Google Scholar
- Iverson, J. and Goldin-Meadow, S. 1998. Why people gesture as they speak. Nature 396, 228.Google ScholarCross Ref
- Janssen, W. and Nilsson, L. 1993. Behavioural effects of driver support. In Driving Future Vehicles, 147--155.Google Scholar
- Johansson, A. and Linde, J. 1999. Using simple force feedback mechanisms as haptic visualizationtools. In Proceedings of the 16th IEEE Instrumentation and Measurement Technology Conference (IMTC'99). Vol. 2.Google Scholar
- Johnson, D. and Wiles, J. 2003. Effective affective user interface design in games. Ergon. 46, 13, 1332--1345.Google ScholarCross Ref
- Jones, L., Lockyer, B., and Piateski, E. 2006. Tactile display and vibrotactile pattern recognition on the torso. Adv. Robotics 20, 12, 1359--1374.Google ScholarCross Ref
- Kaczmarek, K. A. and Webster, J. 1991. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Trans. Biomed. Engin. 38, 1--16.Google ScholarCross Ref
- Kendon, A. 2004. Gesture: Visible Action as Utterance. Cambridge University Press, Cambridge, U.K.Google ScholarCross Ref
- Kraut, R., Gergle, D., and Fussell, S. 2002. The use of visual information in shared visual spaces: Informing the development of virtual co-presence. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. ACM Press, New York, 31--40. Google ScholarDigital Library
- Landau, B., Spelke, E., and Gleitman, H. 1984. Spatial knowledge in a young blind child. Cogn. 16, 225--260.Google ScholarCross Ref
- Lecuyer, A., Moubuchon, P., Megard, C., Perret, J., Andriot, C., and Colinot, J. 2003. Homere: A multimodal system for visually impaired people to explore virtual environments. In Proceedings of the IEEE Annual International Symposium on Virtual Reality. 251--258. Google ScholarDigital Library
- Lee, J., Hoffman, J., and Hayes, E. 2004. Collision warning design to mitigate driver distraction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, New York, 65--72. Google ScholarDigital Library
- Lloyd, D., Merat, N., Mcglone, F., and Spence, C. 2003. Crossmodal links between audition and touch in covert endogenous spatial attention. Percept. Psychophys. 65, 6, 901--924.Google ScholarCross Ref
- Lohse, G. L. 1997. Models of graphical perception. In Handbook of Human-Computer Interaction.Google Scholar
- Manshad, M. and Manshad, A. 2008. Multimodal vision glove for touchscreens. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility. ACM New York, 251--252. Google ScholarDigital Library
- Mcgehee, D., Raby, M., Lee, J., and Nourse, G. 2001. Final design and operating characteristics of a snowplow lane awareness system. In Proceedings of the 80th Annual Meeting of the Transportation Research Board. 7--11.Google Scholar
- Mcgookin, D. and Brewster, S. 2007. An initial investigation into non-visual computer supported collaboration. In CHI'07 Extended Abstracts on Human Factors in Computing Systems. ACM Press, New York, 2573--2578. Google ScholarDigital Library
- Mcneill, D. 1992. Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago, IL.Google Scholar
- Mcneill, D. 2000. Language and Gesture. Cambridge University Press.Google Scholar
- Mcneill, D. 2005. Gesture and Thought. University of Chicago Press.Google Scholar
- Mcneill, D. 2006. Gesture, gaze, and ground. In Machine Learning for Multimodal Interaction. Lecture Notes in Computer Science, vol. 3869, Springer, 1--14. Google ScholarDigital Library
- Millar, S. 1985. Movement cues and body orientation in recall of locations by blind and sighted children. Quart. J. Psychol. 257--279.Google Scholar
- Monk, A. 2003. Common ground in electronically mediated communication. In HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science, J. Carroll, Ed. Morgan Kaufman, San Francisco, 265--286.Google Scholar
- Montgomery, K. J., Isenberg, N., and Haxby, J. V. 2007. Communicative hand gestures and object-directed hand movements activated the mirror neuron system. Soc. Cogn. Affect. Neurosci. 2, 2, 114--122.Google ScholarCross Ref
- Mynatt, E. and Wber, G. 1994. Nonvisual presentation of graphical user interfaces. In Proceedings of the ACM Human Factors in Computing Systems Conference (CHI'94). ACM Press, New York. Google ScholarDigital Library
- Oberti V. Clementon. 1993. Court decision. 995 F.2D 1204, (3rd Cir. 1993).Google Scholar
- Oliveira, F. 2010. Enabling the blind to see gestures. Ph.D. thesis, Virginia Polytechnic Institute and State University.Google Scholar
- Oliveira, F., Cowan, H., Fang, B., and Quek, F. 2010. Fun to develop embodied skill: How games help the blind to understand pointing. In Proceedings of the 3rd International Conference on Pervasive Technologies Related to Assistive Environments (PETRA). ACM Press, New York, 1--8. Google ScholarDigital Library
- Oliveira, F. and Quek, F. 2008. A multimodal communication with a haptic glove: On the fusion of speech and deictic over a raised line drawing. In 1st International Conference on Pervasive Technologies Related to Assistive Environments (PETRA). Google ScholarDigital Library
- Oliveira, F., Quek, F., Cowan, H., and Fang, B. 2011. The haptic deictic system - hds: Bringing blind students to mainstream classrooms. IEEE Trans. Haptics PP, 99, 1--1. Google ScholarDigital Library
- Penrose, R. 1989. The Emperor's New Mind. Oxford University Press, New York.Google Scholar
- Poizner, H., Klima, E., and Bellugi, U. 2000. What the Hands Reveal About the Brain. MIT Press.Google Scholar
- Prablanc, C., Echallier, J., Komilis, E., and Jeannerod, M. 1979. Optimal response of eye and hand motor systems in pointing at a visual target. Spatio-Temporal characteristics of eye and hand movements and their relationships when varying the amount of visual information. Biol. Cybern. 35, 2, 113--124.Google ScholarDigital Library
- Quek, F. 2004. The catchment feature model: A device for multimodal fusion and a bridge between signal and sense. EURASIP J. Appl. Signal Process. 1, 1619--1636. Google ScholarDigital Library
- Quek, F., Mcneill, D., Ansari, R., Ma, X., Bryll, R., Duncan, S., and Mccullough, K.-E. 2002. Multimodal human discourse: Gesture and speech. ACM Trans. Comput.-Hum. Interact. 9, 3, 171--193. Google ScholarDigital Library
- Raisamo, R., Patomaki, S., Hasu, M., and Pasto, V. 2007. Design and evaluation of a tactile memory game for visually impaired children. Interact. Comput. 19, 2, 196--205. Google ScholarDigital Library
- Rizzolatti, G., Fadiga, L., Gallese, V., and Fogassi, L. 1996. Premotor cortex and the recognition of motor actions. Cogn. Brain Res. 3, 2, 131--141.Google ScholarCross Ref
- Rose, R., Quek, F., and Shi, Y. 2004. MacVisSTA: A system for multimodal analysis. In Proceedings of the 6th International Conference on Multimodal Interfaces. ACM Press, New York, 259--264. Google ScholarDigital Library
- Roth, P., Petrucci, L., Pun, T., and Assimacopoulos, A. 1999. Auditory browser for blind and visually impaired users. In CHI EA'99: CHI'99 Extended Abstracts on Human Factors in Computing Systems. ACM Press, New York, 218--219. Google ScholarDigital Library
- Sallnas, E., Bjerstedt-Blom, K., Winberg, F., and Eklundh, K. 2006. Navigation and control in haptic applications shared by blind and sighted users. In Proceedings of the 1st International Conference on Haptic and Audio Interaction Design (HAID'06). 68--80. Google ScholarDigital Library
- Sepchat, A., Monmarche, N., Slimane, M., and Archambault, D. 2006. Semi automatic generator of tactile video games for visually impaired children. In Proceedings of the 10th International Conference on Computers Helping People with Special Needs. Lecture Notes in Computer Science, vol. 4061, Springer, 372--379. Google ScholarDigital Library
- Sherrick, C. 1985. Touch as a communicative sense: Introduction. J. Acoust. Soc. Amer. 77, 218.Google ScholarCross Ref
- Sjostrom, C. 2001. Designing haptic computer interfaces for blind people. In Proceedings of the 6th International Symposium on Signal Processing and its Applications (ISSPA'01). 1--4.Google ScholarCross Ref
- Spence, C., Nicholls, M., and Driver, J. 2001. The cost of expecting events in the wrong sensory modality. Percept. Psychophys. 63, 2, 330--336.Google ScholarCross Ref
- Spence, C., Shore, D., and Klein, R. 2001. Multisensory prior entry. J. Exp. Psychol. Gen. 130, 4, 799--832.Google ScholarCross Ref
- Splindler, R. 2005. Teaching mathematics to a student who is blind. Teach. Math. Appl. 25, 3.Google Scholar
- Staub, D. and Peck, C. 1994. What are the outcomes for nondisabled students? Educ. Leader. 52, 4, 36--40.Google Scholar
- Supalo, C. 2005. Techniques to enhance instructors' teaching effectiveness with chemistry students who are blind or visually impaired. J. Chem. Educ. 82, 10, 1513.Google ScholarCross Ref
- Sweetser, P. and Wyeth, P. 2005. Gameflow: A model for evaluating player enjoyment in games. Comput. Entertain. 3, 3--3. Google ScholarDigital Library
- Tatar, D. G., Foster, G., and Bobrow, D. 1991. Designing for conversation: Lessons from cognoter. Int. J. Man-Mach. Stud. 34, 185--209. Google ScholarDigital Library
- Universal Low Vision Aids Inc. 2009a. Cost of 8.5 × 11 braille paper. http://www.ulva.com/Online-Store/Braille-Paper/tractor8x11.htm.Google Scholar
- Universal Low Vision Aids Inc. 2009b. Emprint. http://www.ulva.com/Online-Store/Braille-Embossers/emprint.htm.Google Scholar
- U.S.C. 1997. Individuals with disabilities education acts amendments of 1997. United States Congress.Google Scholar
- U.S.C. 2002. No child left behind act of 2001. United States Congress.Google Scholar
- Vitense, H., Jacko, J., and Emery, V. 2003. Multimodal feedback: An assessment of performance and mental workload. Ergon. 46, 1--3, 68--87.Google ScholarCross Ref
- Wall, S. and Brewster, S. 2006. Sensory substitution using tactile pin arrays: Human factors, technology and applications. Signal Process. 86, 12, 3674--3695. Google ScholarDigital Library
- Wang, Q. and Hayward, V. 2006. Compact, portable, modular, high-performance, distributed tactile transducer device based on lateral skin deformation. In Proceedings of the 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 67--72. Google ScholarDigital Library
- Wells, L. and Landau, S. 2003. Merging of tactile sensory input and audio data by means of the talking tactile tablet. In Proceedings of the Eurohaptics Conference. 414--418.Google Scholar
- WHO. 2007. Blindness and low vision. Tech. rep. H54, World Health Organization.Google Scholar
- WHO. 2010. Fact sheet visual impairment and blindness. Tech. rep. N282, World Health Organization.Google Scholar
- Wickens, C. and Hollands, J. 2001. Engineering Psychology and Human Performance. Prentice Hall.Google Scholar
- Williams, J. M. 2002. Nationwide shortage of teachers for blind students must be corrected. In National Federation of the Blind: Advocates for Equality, Canadian Blind Monitor.Google Scholar
- Winberg, F. and Bowers, J. 2004. Assembling the senses: Towards the design of cooperative interfaces for visually impaired users. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW). ACM Press, New York, 332--341. Google ScholarDigital Library
Index Terms
- Enabling the blind to see gestures
Recommendations
Usable gestures for blind people: understanding preference and performance
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsDespite growing awareness of the accessibility issues surrounding touch screen use by blind people, designers still face challenges when creating accessible touch screen interfaces. One major stumbling block is a lack of understanding about how blind ...
The Haptic Deictic System—HDS: Bringing Blind Students to Mainstream Classrooms
Mathematics instruction and discourse typically involve two modes of communication: speech and graphical presentation. For the communication to remain situated, dynamic synchrony must be maintained between the speech and dynamic focus in the graphics. ...
Fun to develop embodied skill: how games help the blind to understand pointing
PETRA '10: Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive EnvironmentsWe discuss how gaming can be used as a training strategy for students who are blind or visually impaired (SBVI) to develop embodied skill in use of haptic assistive technology. The technology takes the form of a haptic glove that is designed to give ...
Comments