Skip to main content
Top
Published in: Journal on Multimodal User Interfaces 4/2020

03-10-2020 | Editorial

Multimodal interfaces and communication cues for remote collaboration

Authors: Seungwon Kim, Mark Billinghurst, Kangsoo Kim

Published in: Journal on Multimodal User Interfaces | Issue 4/2020

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Remote collaboration has been studied for more than two decades and now there is the possibilities for new types of collaboration with the recent advances in immersive technologies such as Virtual, Augmented, Mixed Reality (VR/AR/MR). However, despite the increasing research interest in remote collaboration study with VR/AR/MR technologies, there is still a lack of academic venues specifically focusing on VR/AR/MR remote collaboration research. This special issue provides high-quality papers on the topic of remote collaboration research and increases visibility of this timely interesting and important research area. We particularly focus on three research aspects in remote collaboration: (1) use of multimodal communication cues, (2) awareness of the task space, and (3) human factors understanding. In this editorial, we first describe five essential factors for remote collaboration: task, local user, remote user, communication, and tool/interface, and then summarize a brief history of the research areas. We also cover the feature papers accepted in this issue, which introduce novel multimodal interfaces for remote collaboration and the effects on task performance and perceptual factors. Finally, we discuss some potential future research directions while concluding the editorial.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Alem L, Tecchia F, Huang W (2011) Remote tele-assistance system for maintenance operators in mines. In: 11th Underground coal operators’ conference. University of Wollongong Alem L, Tecchia F, Huang W (2011) Remote tele-assistance system for maintenance operators in mines. In: 11th Underground coal operators’ conference. University of Wollongong
2.
go back to reference Baecker RM (1994) Readings in groupware and computer-supported cooperative work: assisting human–human collaboration, 1st edn. Morgan Kaufmann, San Francisco Baecker RM (1994) Readings in groupware and computer-supported cooperative work: assisting human–human collaboration, 1st edn. Morgan Kaufmann, San Francisco
3.
go back to reference Choudhary Z, Kim K, Schubert R, Bruder G, Welch GF (2020) Virtual big heads: analysis of human perception and comfort of head scales in social virtual reality. In: IEEE conference on virtual reality and 3D user interfaces, pp 425–433 Choudhary Z, Kim K, Schubert R, Bruder G, Welch GF (2020) Virtual big heads: analysis of human perception and comfort of head scales in social virtual reality. In: IEEE conference on virtual reality and 3D user interfaces, pp 425–433
4.
go back to reference DeVincenzi A, Yao L, Ishii H, Raskar R (2011) Kinected conference: augmenting video imaging with calibrated depth and audio. In: Proceedings of the ACM 2011 conference on computer supported cooperative work, pp 621-624 DeVincenzi A, Yao L, Ishii H, Raskar R (2011) Kinected conference: augmenting video imaging with calibrated depth and audio. In: Proceedings of the ACM 2011 conference on computer supported cooperative work, pp 621-624
5.
go back to reference Dey A, Piumsomboon T, Lee Y, Billinghurst M (2017) Effects of sharing physiological states of players in a collaborative virtual reality gameplay. In: Proceedings of the 2017 CHI conference on human factors in computing systems, pp 4045–4056 Dey A, Piumsomboon T, Lee Y, Billinghurst M (2017) Effects of sharing physiological states of players in a collaborative virtual reality gameplay. In: Proceedings of the 2017 CHI conference on human factors in computing systems, pp 4045–4056
6.
go back to reference Erickson A, Norouzi N, Kim K, LaViola JJ, Bruder G, Welch GF (2020) Effects of depth information on visual target identification task performance in shared gaze environments. IEEE Trans Visu Comput Graph 26(5):1934–1944CrossRef Erickson A, Norouzi N, Kim K, LaViola JJ, Bruder G, Welch GF (2020) Effects of depth information on visual target identification task performance in shared gaze environments. IEEE Trans Visu Comput Graph 26(5):1934–1944CrossRef
8.
go back to reference Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Stabilized annotations for mobile remote assistance. In: Proceedings of the 2016 CHI conference on human factors in computing systems, pp 1548–1560 Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Stabilized annotations for mobile remote assistance. In: Proceedings of the 2016 CHI conference on human factors in computing systems, pp 1548–1560
9.
go back to reference Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Gestures over video streams to support remote collaboration on physical tasks. Hum–Comput Interact 19(3):273–309CrossRef Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Gestures over video streams to support remote collaboration on physical tasks. Hum–Comput Interact 19(3):273–309CrossRef
10.
go back to reference Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) World-stabilized annotations and virtual scene navigation for remote collaboration. In: Proceedings of the 27th annual ACM symposium on user interface software and technology, pp 449-459 Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) World-stabilized annotations and virtual scene navigation for remote collaboration. In: Proceedings of the 27th annual ACM symposium on user interface software and technology, pp 449-459
11.
go back to reference Gupta K, Lee GA, Billinghurst M (2016) Do you see what i see? The effect of gaze tracking on task space remote collaboration. IEEE Trans Vis Comput Graph 22(11):2413–2422CrossRef Gupta K, Lee GA, Billinghurst M (2016) Do you see what i see? The effect of gaze tracking on task space remote collaboration. IEEE Trans Vis Comput Graph 22(11):2413–2422CrossRef
12.
go back to reference Gutwin C, Greenberg S (1998) Design for individuals, design for groups: tradeoffs between power and workspace awareness. In: Proceedings of the ACM conference on computer supported cooperative work, pp 207–216 Gutwin C, Greenberg S (1998) Design for individuals, design for groups: tradeoffs between power and workspace awareness. In: Proceedings of the ACM conference on computer supported cooperative work, pp 207–216
13.
go back to reference Higuch K, Yonetani R, Sato Y (2016) Can eye help you? Effects of visualizing eye fixations on remote collaboration scenarios for physical tasks. In: Proceedings of the 2016 CHI conference on human factors in computing systems, pp 5180–5190 Higuch K, Yonetani R, Sato Y (2016) Can eye help you? Effects of visualizing eye fixations on remote collaboration scenarios for physical tasks. In: Proceedings of the 2016 CHI conference on human factors in computing systems, pp 5180–5190
14.
go back to reference Huang W, Alem L, Tecchia F, Duh HBL (2018) Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration. J Multimodal User Interfaces 12(2):77–89CrossRef Huang W, Alem L, Tecchia F, Duh HBL (2018) Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration. J Multimodal User Interfaces 12(2):77–89CrossRef
15.
go back to reference Huang W, Kim S, Billinghurst M, Alem L (2019) Sharing hand gesture and sketch cues in remote collaboration. J Vis Commun Image Represent 58:428–438CrossRef Huang W, Kim S, Billinghurst M, Alem L (2019) Sharing hand gesture and sketch cues in remote collaboration. J Vis Commun Image Represent 58:428–438CrossRef
16.
go back to reference Irlitti A, Piumsomboon T, Jackson D, Thomas BH (2019) Conveying spatial awareness cues in xR collaborations. IEEE Trans Visu Comput Graph 25(11):3178–3189CrossRef Irlitti A, Piumsomboon T, Jackson D, Thomas BH (2019) Conveying spatial awareness cues in xR collaborations. IEEE Trans Visu Comput Graph 25(11):3178–3189CrossRef
17.
go back to reference Kasahara S, Rekimoto J (2014) JackIn: integrating first-person view with out-of-body vision generation for human–human augmentation. In: Proceedings of the 5th augmented human international conference, pp 46:1–46:8 Kasahara S, Rekimoto J (2014) JackIn: integrating first-person view with out-of-body vision generation for human–human augmentation. In: Proceedings of the 5th augmented human international conference, pp 46:1–46:8
18.
go back to reference Kato H, Billinghurst M (1999) Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In: Proceedings 2nd IEEE and ACM international workshop on augmented reality (IWAR’99), pp 85–94 Kato H, Billinghurst M (1999) Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In: Proceedings 2nd IEEE and ACM international workshop on augmented reality (IWAR’99), pp 85–94
19.
go back to reference Kim K, Billinghurst M, Bruder G, Duh HBL, Welch GF (2018) Revisiting trends in augmented reality research: a review of the 2nd decade of ISMAR (2008–2017). IEEE Trans Vis Comput Graph 24(11):2947–2962CrossRef Kim K, Billinghurst M, Bruder G, Duh HBL, Welch GF (2018) Revisiting trends in augmented reality research: a review of the 2nd decade of ISMAR (2008–2017). IEEE Trans Vis Comput Graph 24(11):2947–2962CrossRef
20.
go back to reference Kim K, Boelling L, Haesler S, Bailenson JN, Bruder G, Welch GF (2018) Does a digital assistant need a body? The influence of visual embodiment and social behavior on the perception of intelligent virtual agents in AR. In: IEEE international symposium on mixed and augmented reality, pp 105–114 Kim K, Boelling L, Haesler S, Bailenson JN, Bruder G, Welch GF (2018) Does a digital assistant need a body? The influence of visual embodiment and social behavior on the perception of intelligent virtual agents in AR. In: IEEE international symposium on mixed and augmented reality, pp 105–114
21.
go back to reference Kim K, de Melo C, Norouzi N, Bruder G, Welch G (2020) Reducing task load with an embodied intelligent virtual assistant for improved performance in collaborative decision making. In: Proceedings of the IEEE conference on virtual reality and 3D user interfaces, pp 529–538 Kim K, de Melo C, Norouzi N, Bruder G, Welch G (2020) Reducing task load with an embodied intelligent virtual assistant for improved performance in collaborative decision making. In: Proceedings of the IEEE conference on virtual reality and 3D user interfaces, pp 529–538
22.
go back to reference Kim K, Norouzi N, Losekamp T, Bruder G, Anderson M, Welch G (2019) Effects of patient care assistant embodiment and computer mediation on user experience. In: Proceedings of IEEE international conference on artificial intelligence and virtual reality, pp 17–24 Kim K, Norouzi N, Losekamp T, Bruder G, Anderson M, Welch G (2019) Effects of patient care assistant embodiment and computer mediation on user experience. In: Proceedings of IEEE international conference on artificial intelligence and virtual reality, pp 17–24
23.
go back to reference Kim K, Schubert R, Hochreiter J, Bruder G, Welch G (2019) Blowing in the wind: increasing social presence with a virtual human via environmental airflow interaction in mixed reality. Comput Graph 83:23–32CrossRef Kim K, Schubert R, Hochreiter J, Bruder G, Welch G (2019) Blowing in the wind: increasing social presence with a virtual human via environmental airflow interaction in mixed reality. Comput Graph 83:23–32CrossRef
24.
go back to reference Kim S, Billinghurst M, Lee C, Lee G (2018) Using freeze frame and visual notifications in an annotation drawing interface for remote collaboration. KSII Trans Internet Inf Syst 12(12):6034–6056 Kim S, Billinghurst M, Lee C, Lee G (2018) Using freeze frame and visual notifications in an annotation drawing interface for remote collaboration. KSII Trans Internet Inf Syst 12(12):6034–6056
25.
go back to reference Kim S, Billinghurst M, Lee G (2018) The effect of collaboration styles and view independence on video-mediated remote collaboration. Comput Support Cooper Work 27(3):569–607CrossRef Kim S, Billinghurst M, Lee G (2018) The effect of collaboration styles and view independence on video-mediated remote collaboration. Comput Support Cooper Work 27(3):569–607CrossRef
27.
go back to reference Kim S, Lee G, Huang W, Kim H, Woo W, Billinghurst M (2019) Evaluating the combination of visual communication cues for HMD-based mixed reality remote collaboration. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–13 Kim S, Lee G, Huang W, Kim H, Woo W, Billinghurst M (2019) Evaluating the combination of visual communication cues for HMD-based mixed reality remote collaboration. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–13
28.
go back to reference Kim S, Lee G, Sakata N, Billinghurst M (2014) Improving co-presence with augmented visual communication cues for sharing experience through video conference. In: Proceedings of the IEEE international symposium on mixed and augmented reality, pp 83–92 Kim S, Lee G, Sakata N, Billinghurst M (2014) Improving co-presence with augmented visual communication cues for sharing experience through video conference. In: Proceedings of the IEEE international symposium on mixed and augmented reality, pp 83–92
29.
go back to reference Kim S, Lee GA, Ha S, Sakata N, Billinghurst M (2015) Automatically freezing live video for annotation during remote collaboration. In: Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems, pp 1669–1674 Kim S, Lee GA, Ha S, Sakata N, Billinghurst M (2015) Automatically freezing live video for annotation during remote collaboration. In: Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems, pp 1669–1674
30.
go back to reference Kim S, Lee GA, Sakata N (2013) Comparing pointing and drawing for remote collaboration. In: Proceedings of the IEEE international symposium on mixed and augmented reality, pp 1–6 Kim S, Lee GA, Sakata N (2013) Comparing pointing and drawing for remote collaboration. In: Proceedings of the IEEE international symposium on mixed and augmented reality, pp 1–6
31.
go back to reference Kirk D, Stanton Fraser D (2006) Comparing remote gesture technologies for supporting collaborative physical tasks. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1191–1200 Kirk D, Stanton Fraser D (2006) Comparing remote gesture technologies for supporting collaborative physical tasks. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1191–1200
32.
go back to reference Kraut RE, Gergle D, Fussell SR (2002) The use of visual information in shared visual spaces: informing the development of virtual co-presence. In: Proceedings of the 2002 ACM conference on computer supported cooperative work, pp 31–40 Kraut RE, Gergle D, Fussell SR (2002) The use of visual information in shared visual spaces: informing the development of virtual co-presence. In: Proceedings of the 2002 ACM conference on computer supported cooperative work, pp 31–40
33.
go back to reference Lecuyer A, Lotte F, Reilly R, Leeb R, Hirose M, Slater M (2008) Brain–computer interfaces, virtual reality, and videogames. Computer 41(10):66–72CrossRef Lecuyer A, Lotte F, Reilly R, Leeb R, Hirose M, Slater M (2008) Brain–computer interfaces, virtual reality, and videogames. Computer 41(10):66–72CrossRef
34.
go back to reference Lee G, Kang HY, Lee JM, Han JH (2020) A user study on view-sharing techniques for one-to-many mixed reality collaborations. In: Proceedings of the IEEE conference on virtual reality and 3D user interfaces, pp 343–352 Lee G, Kang HY, Lee JM, Han JH (2020) A user study on view-sharing techniques for one-to-many mixed reality collaborations. In: Proceedings of the IEEE conference on virtual reality and 3D user interfaces, pp 343–352
35.
go back to reference Lee G, Kim S, Lee Y, Dey A, Piumsomboon T, Norman M, Billinghurst M (2017) Mutually shared gaze in augmented video conference. In: Adjunct proceedings of the 2017 IEEE international symposium on mixed and augmented reality-adjunct, pp 79–80 Lee G, Kim S, Lee Y, Dey A, Piumsomboon T, Norman M, Billinghurst M (2017) Mutually shared gaze in augmented video conference. In: Adjunct proceedings of the 2017 IEEE international symposium on mixed and augmented reality-adjunct, pp 79–80
36.
go back to reference Lee G, Teo THL, Kim S, Billinghurst M (2018) A user study on mr remote collaboration using live 360 video. In: Proceedings of the IEEE international symposium for mixed and augmented reality, pp 153–164 Lee G, Teo THL, Kim S, Billinghurst M (2018) A user study on mr remote collaboration using live 360 video. In: Proceedings of the IEEE international symposium for mixed and augmented reality, pp 153–164
37.
go back to reference Lukosch S, Billinghurst M, Alem L, Kiyokawa K (2015) Collaboration in augmented reality. Comput Support Cooper Work 24(6):515–525CrossRef Lukosch S, Billinghurst M, Alem L, Kiyokawa K (2015) Collaboration in augmented reality. Comput Support Cooper Work 24(6):515–525CrossRef
38.
go back to reference Norouzi N, Erickson A, Kim K, Schubert R, Laviola Jr, JJ, Bruder G, Welch GF (2019) Effects of shared gaze parameters on visual target identification task performance in augmented reality. In: Proceedings of ACM symposium on spatial user interaction, pp 12:1–12:11 Norouzi N, Erickson A, Kim K, Schubert R, Laviola Jr, JJ, Bruder G, Welch GF (2019) Effects of shared gaze parameters on visual target identification task performance in augmented reality. In: Proceedings of ACM symposium on spatial user interaction, pp 12:1–12:11
39.
go back to reference Norouzi N, Kim K, Hochreiter J, Lee M, Daher S, Bruder G, Welch G (2018) A systematic survey of 15 years of user studies published in the intelligent virtual agents conference. In: Proceedings of the ACM international conference on intelligent virtual agents, pp 17–22 Norouzi N, Kim K, Hochreiter J, Lee M, Daher S, Bruder G, Welch G (2018) A systematic survey of 15 years of user studies published in the intelligent virtual agents conference. In: Proceedings of the ACM international conference on intelligent virtual agents, pp 17–22
40.
go back to reference Norouzi N, Kim K, Lee M, Schubert R, Erickson A, Bailenson J, Bruder G, Welch G (2019) walking your virtual dog: analysis of awareness and proxemics with simulated support animals in augmented reality. In: Proceedings of IEEE international symposium on mixed and augmented reality, pp 253–264 Norouzi N, Kim K, Lee M, Schubert R, Erickson A, Bailenson J, Bruder G, Welch G (2019) walking your virtual dog: analysis of awareness and proxemics with simulated support animals in augmented reality. In: Proceedings of IEEE international symposium on mixed and augmented reality, pp 253–264
41.
go back to reference Paro JA, Nazareli R, Gurjala A, Berger A, Lee GK (2015) Video-based self-review. Ann Plast Surg 74:S71–S74CrossRef Paro JA, Nazareli R, Gurjala A, Berger A, Lee GK (2015) Video-based self-review. Ann Plast Surg 74:S71–S74CrossRef
42.
go back to reference Piumsomboon T, Lee GA, Hart JD, Ens B, Lindeman RW, Thomas BH, Billinghurst M (2018) Mini-Me: an adaptive avatar for mixed reality remote collaboration. In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp 1–13 Piumsomboon T, Lee GA, Hart JD, Ens B, Lindeman RW, Thomas BH, Billinghurst M (2018) Mini-Me: an adaptive avatar for mixed reality remote collaboration. In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp 1–13
43.
go back to reference Piumsomboon T, Lee GA, Irlitti A, Ens B, Thomas BH, Billinghurst M (2019) On the shoulder of the giant: a multi-scale mixed reality collaboration with 360 video sharing and tangible interaction. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–17 Piumsomboon T, Lee GA, Irlitti A, Ens B, Thomas BH, Billinghurst M (2019) On the shoulder of the giant: a multi-scale mixed reality collaboration with 360 video sharing and tangible interaction. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–17
44.
go back to reference Teo T, Lawrence L, Lee GA, Billinghurst M, Adcock M (2019) Mixed reality remote collaboration combining 360 video and 3D reconstruction. In: Proceedings of the ACM CHI conference on human factors in computing systems, vol 201. ACM Press, New York, pp 1–14 Teo T, Lawrence L, Lee GA, Billinghurst M, Adcock M (2019) Mixed reality remote collaboration combining 360 video and 3D reconstruction. In: Proceedings of the ACM CHI conference on human factors in computing systems, vol 201. ACM Press, New York, pp 1–14
Metadata
Title
Multimodal interfaces and communication cues for remote collaboration
Authors
Seungwon Kim
Mark Billinghurst
Kangsoo Kim
Publication date
03-10-2020
Publisher
Springer International Publishing
Published in
Journal on Multimodal User Interfaces / Issue 4/2020
Print ISSN: 1783-7677
Electronic ISSN: 1783-8738
DOI
https://doi.org/10.1007/s12193-020-00346-8

Other articles of this Issue 4/2020

Journal on Multimodal User Interfaces 4/2020 Go to the issue

Premium Partner