A Formal Description of Low Level Interaction and its Application to Multimodal Interactive Systems

The lack of formal models for describing low-level interaction restricts programmers to interactors provided by toolkits. It impedes the construction of highly interactive systems and the design of new interaction styles, such as multimodal interaction. This article reports on our experience with formalising low-level graphical interaction. We propose primitives for event specification and handling that can be used along with Petri nets to model such interactions. We then show how multimodal interactions can be built from monomodal ones by combining those models. This is exemplified by an experimental two-handed graphical editor that has been built using the proposed model.