Multimodal Analysis of Joint Attention in Remote Co-creative Design Process

We investigated how empathy for co-creative communication is achieved in the remote co-creative design process. We analyzed interactions occurring during a remote co-creative design process involving the KJ method and Miro, which is a remote working tool. In particular, we used multimodal conversation analysis to evaluate the interactions, focusing on each participant’s mouse cursor movements and how they achieved joint attention to the same object on the screen. We observed two methods of using a mouse cursor to jointly attend to something: using the cursor to produce a pointing gesture, thereby drawing others’ attention, and using it to indicate that the person is attending to others’ pointing. A participant who wants to achieve joint attention with others to a particular object in the Miro environment draws their attention by hovering the mouse cursor over the object. Other participants who are instructed to pay attention demonstrate that they are doing so by moving their mouse cursors closer to the object. This transition to joint attention is visualized on the screen in real-time, which makes it clear whether or not joint attention has been achieved. The results suggest that teleworking environments, such as Miro, provide a unique means of establishing common ground, and that these environments may create a sense of “co-presence” that is different from face-to-face conversations.