SketchSPORE: A Sketch Based Domain Separation and Recognition System for Interactive Interfaces

Multimodal interfaces are used to interact with devices and automata using different channels of communication. In this context, the sketch modality plays a key role since it allows users to convey concepts and/or commands using freehand drawing (graphical domain) and/or handwriting (textual domain). The acquisition of the sketch modality can be performed using touch (e.g., touchscreen) or touchless (e.g., RGB-D camera) tools supporting the development of versatile and powerful interactive interfaces. Domain separation and sketch recognition are two fundamental issues of these interfaces. This paper presents SketchSPORE a novel framework designed both to automatically distinguish graphical from textual elements within the same sketch and to recognize freehand drawing as well as handwriting. The recognition processes support both on-line and off-line modes, moreover their processing can be suitably stored within an XML file to provide a means to maintain the compatibility between the framework and service and/or application targets. Extensive experiments showing the effectiveness of the proposed method are reported and discussed.