Philip R. Cohen
Michael Johnston
Sharon Oviatt
ABSTRACT
This paper presents a novel multimodal system applied to the setup and control of distributed interactive simulations. We have developed the QuickSet prototype, a pen/voice system running on a hand-held PC, communicating through a distributed agent architecture to NRaD's1 LeatherNet system, a distributed interactive training simulator built for the US Marine Corps (USMC). The paper briefly describes the system and illustrates its use in multimodal simulation setup.
- Calder, J. Typed unification for natural language processing. In E. Klein and J. van Benthem (Eds.), Categories, Polymorphisms, and Unification. Centre for Cognitive Science, University of Edinburgh, Edinburgh, 1987, 65--72.Google Scholar
- Brison, E. and N. Vigouroux. (unpublished ms.). Multimodal references: A generic fusion process. URIT-URA CNRS. Université Paul Sabatier, Toulouse, France.Google Scholar
- Carpenter, R. The logic of typed feature structures. Cambridge University Press, Cambridge, 1992. Google ScholarDigital Library
- Cheyer, A., and L. Julia. Multimodal maps: An agent-based approach. International Conference on Cooperative Multimodal Communication (CMC/95), May 1995: Eindhoven, The Netherlands, 1995, 24--26. Google ScholarDigital Library
- Clarkson, J. D., and Yi., J., LeatherNet: A synthetic forces tactical training system for the USMC commander. Proceedings of the Sixth Conference on Computer Generated Forces and Behavioral Representation. Institute for simulation and training. Technical Report IST-TR-96--18, 1996, 275--281.Google Scholar
- Cohen, P. R. Integrated Interfaces for Decision Support with Simulation, Proceedings of the Winter Simulation Conference, Nelson, B. and Kelton, W. D. and Clark, G. M., (eds.), ACM, New York, December, 1991, 1066--1072. Google ScholarDigital Library
- Cohen, P. R. The Role of Natural Language in a Multimodal Interface. Proceedings of UIST'92, ACM Press, New York, 1992, 143--149. Google ScholarDigital Library
- Cohen, P. R., Cheyer, A., Wang, M., and Baeg, S. C. An Open Agent Architecture. Working notes of the AAAI Spring Symposium Series on Software Agents Stanford Univ., CA, March, 1994, 1--8.Google Scholar
- Cohen. P. R., Dalrymple, M., Moran, D. B., Pereira, F. C. N., Sullivan, J. W., Gargan, R. A., Schlossberg, J. L., and Tyler, S. W. Synergistic Use of Direct Manipulation and Natural Language, Human Factors in Computing Systems: CHI'89 Conference Proceedings, ACM, Addison Wesley Publishing Co New York, 227--234, 1989. Google ScholarDigital Library
- Courtemanche, A. J. and Ceranowicz, A. ModSAF Development Status. Proceedings of the Fifth Conference on Computer Generated Forces and Behavioral Representation, Univ. Central Florida, Orlando, 1995, 3--13.Google Scholar
- Cruz-Neira, C. D. J. Sandin, T. A. DeFanti, "Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE," Computer Graphics (Proceedings of SIGGRAH'93), ACM SIGGRAPH, August 1993, 135--142. Google ScholarDigital Library
- Johnston, M., Cohen, P. R., McGee, D., Oviatt, S. L., Pittman, J., and Smith, I., Unification-based multimodal integration, in submission. Google ScholarDigital Library
- Koons, D. B., C. J. Sparrell and K. R. Thorisson. 1993. Integrating simultaneous input from speech, gaze, and hand gestures. In Mark T. Maybury (ed.) Intelligent Multimedia Interfaces. AAAI Press/ MIT Press, Cambridge, MA, 257--276. Google ScholarDigital Library
- Moore, R., Dowding, J. Bratt, H. Gawron, J. M., and Cheyer, A., CommandTalk: A Spoken-Language Interface for Battlefield Simulations, 1997, (this volume).Google Scholar
- Neal, J. G. and Shapiro, S. C. Intelligent multi-media interface technology. In J. W. Sullivan and S. W. Tyler, editors, Intelligent User Interfaces, chapter 3, pages 45--68. ACM Press Frontier Series, Addison Wesley Publishing Co., New York, New York, 1991. Google ScholarDigital Library
- Oviatt, S. L., Pen/Voice: Complementary multimodal communication. Proceedings of SpeechTech'92, New York, February, 1992, 238--241.Google Scholar
- Oviatt, S. L. Multimodal interfaces for dynamic interactive maps. Proceedings of CHI'96 Human Factors in Computing Systems (April 13--18, Vancouver, Canada), ACM Press, NY, 1996, 95--102. Google ScholarDigital Library
- Oviatt, S. L., Multimodal interactive maps: Designing for human performance, Human-Computer Interaction, in press. Google ScholarDigital Library
- Oviatt, S. L. A. DeAngeli, and K. Kuhn. In press. Integration and synchronization of input modes during multimodal human-computer interaction. Proceedings of the Conference on Human Factors in Computing Systems (CHI '97), ACM Press, New York. Google ScholarDigital Library
- Oviatt, S. L., Cohen, P. R, Fong, M. W. and Frank, M. P., A rapid semi-automatic simulation technique for interactive speech and handwriting, Proceedings of the 1992 International Conference Spoken Language Processing, vol. 2, University of Alberta, J. Ohala (ed.), October, 1992, 1351--1354.Google Scholar
- Oviatt, S. L., Cohen, P. R., Wang, M. Q., Toward interface design for human language technology: Modality and structure as determinants of linguistic complexity, Speech Communication, 15 (3--4), 1994. Google ScholarDigital Library
- Pittman, J. A., Smith, I. A., Cohen, P. R., Oviatt, S. L., and Yang, T. C. QuickSet: A Multimodal Interface for Military Simulation. in Proceedings of the Sixth Conference on Computer-Generated Forces and Behavioral Representation, Orlando, Florida, 1996.Google Scholar
- Thorpe, J. A., The new technology of large scale simulator networking: Implications for mastering the art of warfighting. Proceedings of the 9th Interservice/industry Training Systems Conference, Orlando, Florida, December, 1987, 492--501.Google ScholarCross Ref
- Vo, M. T. and C. Wood. Building an application framework for speech and pen input integration in multimodal learning interfaces. International Conference on Acoustics, Speech, and Signal Processing, Atlanta, GA, 1996.Google Scholar
- Wauchope, K. Eucalyptus: Integrating natural language input with a graphical user interface. Naval Research Laboratory, Report NRL/FR/5510--94--9711, 1994.Google Scholar
- Zyda, M. J., Pratt, D. R., Monahan, J. G., and Wilson, K. P., NPSNET: Constructing a 3-D virtual world, Proceedings of the 1992 Symposium on Interactive 3-D Graphics, March, 1992. Google ScholarDigital Library
- QuickSet: multimodal interaction for simulation set-up and control
Recommendations
QuickSet: multimodal interaction for distributed applications
MULTIMEDIA '97: Proceedings of the fifth ACM international conference on MultimediaMulitmodal interaction for distributed interactive simulation
AAAI'97/IAAI'97: Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligenceThis paper presents an emerging application of Artificial Intelligence research to distributed interactive simulations, with the goal of reducing exercise generation time and effort, yet maximizing training effectiveness. We have developed the QuickSet ...
Multimodal interaction for distributed interactive simulation
Readings in intelligent user interfaces
Comments