Top

Journal on Multimodal User Interfaces

Published in:

15-07-2020 | Original Paper

The combination of visual communication cues in mixed reality remote collaboration

Authors: Seungwon Kim, Gun Lee, Mark Billinghurst, Weidong Huang

Published in: Journal on Multimodal User Interfaces | Issue 4/2020

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Many researchers have studied various visual communication cues (e.g. pointer, sketch, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, within the two user studies when a dependent view and independent view are supported respectively. In the first user study with the dependent view, the results showed that the hand gesture cue was the main visual communication cue and adding sketch cue to the hand gesture cue helped participants complete the task faster. In the second study with the independent view, the results showed that the hand gesture had an issue of local worker understanding remote expert’s hand gesture cue and the main visual communication cue was the pointer cue with fast completion time and high level of co-presence.

previous article Multimodal interfaces and communication cues for remote collaboration

next article The effects of spatial auditory and visual cues on mixed reality remote collaboration

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Milgram P, Kishino F (1994) IEICE Trans Inf Syst 77(12):1321

Kim S, Lee G, Sakata N, Billinghurst M (2014) 2014 IEEE international symposium on mixed and augmented reality (ISMAR) pp. 83–92. https://doi.org/10.1109/ISMAR.2014.6948412

Kim S, Lee G, Huang W, Kim H, Woo W, Billinghurst M (2019) Proceedings of the 2019 CHI conference on human factors in computing systems (2019), pp 1–13

Lee G, Kim S, Lee Y, Dey A, Piumsomboon T, Norman M, Billinghurst M (2017) Adjunct proceedings of the 2017 IEEE international symposium on mixed and augmented reality, ISMAR-adjunct 2017 (Institute of Electrical and Electronics Engineers Inc., 2017), pp 79–80

Kim S, Lee GA, Sakata N (2013) 2013 IEEE international symposium on mixed and augmented reality (ISMAR), pp. 1–6. https://doi.org/10.1109/ISMAR.2013.6671833

Kim S, Lee GA, Ha S, Sakata N, Billinghurst M (2015) Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems (ACM, New York, NY, USA, 2015), CHI EA ’15, pp 1669–1674. 10.1145/2702613.2732838

Kim S, Billinghurst M, Lee G (2018) Comput Support Cooper Work (CSCW) 27(3):569. https://doi.org/10.1007/s10606-018-9324-2CrossRef

Kasahara S, Rekimoto J (2014) Proceedings of the 5th augmented human international conference (ACM, New York, NY, USA, 2014), AH ’14, pp 46:1–46:8. http://doi.org/10.1145/2582051.2582097

Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) Proceedings of the 27th annual ACM symposium on user interface software and technology (ACM, New York, NY, USA, 2014), UIST ’14, pp 449–459. http://doi.org/10.1145/2642918.2647372

10.

Huang W, Alem L, Tecchia F, Duh HBL (2018) J Multimod User Interfaces 12(2):77. https://doi.org/10.1007/s12193-017-0250-2CrossRef

11.

Kuzuoka H, Ishimoda G, Nishimura Y, Suzuki R, Kondo K (1995) Proceedings of the fourth european conference on computer-supported cooperative work ECSCW’95 (Springer, 1995), pp 181–196

12.

Sakata N, Kurata T, Kato T, Kourogi M, Kuzuoka H (2003) ISWC, vol 2003 (Citeseer, 2003), vol 2003, p 7th

13.

Gupta K, Lee GA, Billinghurst M (2016) IEEE Trans Vis Comput Graph 22(11):2413. https://doi.org/10.1109/TVCG.2016.2593778CrossRef

14.

Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Hum-Comput Interaction 19(3):273. https://doi.org/10.1207/s15327051hci1903_3CrossRef

15.

Kim S, Lee GA, Sakata N, Daijnser A, Vartiainen E, Billinghurst, M (2013) IEEE international symposium on mixed and augmented reality (ISMAR) (2013), pp 261–262. http://doi.org/10.1109/ISMAR.2013.6671795

16.

Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) Proceedings of the 20th ACM symposium on virtual reality software and technology (2014), pp 197–205

17.

Huang W, Kim S, Billinghurst M, Alem L (2019) J Vis Commun Image Represent 58:428CrossRef

18.

Higuch K, Yonetani R, Sato Y (2016) Proceedings of the 2016 CHI conference on human factors in computing systems (ACM, New York, NY, USA, 2016), CHI ’16, pp 5180–5190. http://doi.org/10.1145/2858036.2858438

19.

Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Proceedings of the 2016 CHI conference on human factors in computing systems (ACM, New York, NY, USA, 2016), CHI ’16, pp 1548–1560. http://doi.org/10.1145/2858036.2858171

20.

Tang JC, Minneman SL (1990) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 1990), CHI ’90, pp 313–320. http://doi.org/10.1145/97243.97302

21.

Kato H, Billinghurst M (1999) Proceedings 2nd IEEE and ACM international workshop on augmented reality (IWAR’99) pp. 85–94. https://doi.org/10.1109/IWAR.1999.803809

22.

Kim S, Billinghurst M, Lee C, Lee G (2018) KSII Trans Internet Inf Syst 12(12):6034. https://doi.org/10.3837/tiis.2018.12.023CrossRef

23.

Alem L, Tecchia F, Huang W (2011) (University of Wollongong, 2011), 11th underground coal operators’ conference

24.

Adcock M, Anderson S, Thomas B (2013) Proceedings of the 12th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry (2013), pp 235–242

25.

Kirk D, Stanton Fraser D (2006) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 2006), CHI ’06, pp 1191–1200. http://doi.org/10.1145/1124772.1124951

26.

Kirk D, Rodden T, Fraser DS (2007) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 2007), CHI ’07, pp 1039–1048. http://doi.org/10.1145/1240624.1240782

27.

Friedman H (1979) Skill in nonverbal communication pp 2–27

28.

Goldin-Meadow S (1999) Trends Cogni Sci 3(11):419. https://doi.org/10.1016/S1364-6613(99)01397-2CrossRef

29.

Teo T, Lee GA, Billinghurst M, Adcock M (2018) Proceedings of the 30th australian conference on computer-human interaction (2018), pp 406–410

30.

Chen S, Chen M, Kunz A, Yantaç AE, Bergmark M, Sundin A, Fjeld M (2013) Proceedings of the 4th augmented human international conference (ACM, New York, NY, USA, 2013), AH ’13, pp 69–76. http://doi.org/10.1145/2459236.2459249

31.

Stefik M, Bobrow DG, Foster G, Lanning S, Tatar D (1987) ACM Trans Inf Syst (TOIS) 5(2):147CrossRef

32.

Gutwin C, Greenberg S (1998) Proceedings of the 1998 ACM conference on computer supported cooperative work (1998), pp 207–216

33.

Young J, Langlotz T, Cook M, Mills S, Regenbrecht H (2019) IEEE Trans Vis Comput Graph 25(5):1908CrossRef

34.

Teo T, Lawrence L, Lee GA, Billinghurst M, Adcock M (2019) Proceedings of the 2019 CHI conference on human factors in computing systems (2019), pp 1–14

35.

Lee G, Teo THL, Kim S, Billinghurst M (2018) Proceedings of the IEEE international symposium for mixed and augmented reality, pp 153–164. http://doi.org/10.1109/ISMAR.2018.00051

36.

Lee GA, Teo T, Kim S, Billinghurst M (2017) SIGGRAPH Asia 2017 Mobile graphics & interactive applications, pp 1–4

37.

Piumsomboon T, Lee GA, Hart JD, Ens B, Lindeman RW, Thomas BH, Billinghurst M (2018) Proceedings of the 2018 CHI conference on human factors in computing systems (Association for Computing Machinery, Montreal QC, Canada, 2018), CHI ’18, pp. 1–13. https://doi.org/10.1145/3173574.3173620

38.

Piumsomboon T, Day A, Ens B, Lee Y, Lee GA, Billinghurst M (2017) SA ’17. http://doi.org/10.1145/3132787.3139200

39.

Gao L, Bai H, Piumsomboon T, Lee G, Lindeman RW, Billinghurst M (2017)

40.

Meta2 website. https://www.metavision.com/

41.

Durrant-Whyte H, Bailey T (2006) IEEE Robot Autom Mag 13(2):99CrossRef

42.

Fove head mounted display website. https://www.getfove.com/

43.

Leap motion hand tracking website. https://www.leapmotion.com/

44.

Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Sensors 13(5):6380CrossRef

45.

Leap motion pinch example. https://gallery.leapmotion.com/pinch-draw/

46.

Zijlstra FRH (1993) Efficiency in work behavior. A design approach for modern tools (Delft University of Technology, Delft, The Netherlands, 1993). Doctoral dissertation from Delft University of Technology

47.

Zijlstra FRH, Doorn LV (1985) Delft, Netherlands p 43

48.

Harms PC, Biocca PF (2004)

49.

Wobbrock JO, Findlater L, Gergle D, Higgins JJ (2011) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA), CHI ’11, pp 143–146. http://doi.org/10.1145/1978942.1978963

50.

Meta2 unity. https://docs.metavision.com/external/doc/latest/unity_sdk_release_notes_260.html

51.

Ricoh theta v. https://theta360.com/en/about/theta/v.html

52.

Htc vive. https://www.vive.com/au/

53.

T. Teo, A. F. Hayati, G. A. Lee, M. Billinghurst, M. Adcock, in 25th ACM Symposium on Virtual Reality Software and Technology (2019), pp. 1–11

54.

Di Stasi LL, McCamy MB, Catena A, Macknik SL, Canas JJ, Martinez-Conde S (2013) European Journal of Neuroscience 38(3):2389CrossRef

55.

Siegenthaler E, Costela FM, McCamy MB, Di Stasi LL, Otero-Millan J, Sonderegger A, Groner R, Macknik S, Martinez-Conde S (2014) European Journal of Neuroscience 39(2):287CrossRef

Title: The combination of visual communication cues in mixed reality remote collaboration
Authors: Seungwon Kim
Gun Lee
Mark Billinghurst
Weidong Huang
Publication date: 15-07-2020
Publisher: Springer International Publishing
Published in: Journal on Multimodal User Interfaces / Issue 4/2020
Print ISSN: 1783-7677
Electronic ISSN: 1783-8738
DOI: https://doi.org/10.1007/s12193-020-00335-x

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 4/2020

Exploring interaction techniques for 360 panoramas inside a 3D reconstructed scene for mixed reality remote collaboration

The effects of spatial auditory and visual cues on mixed reality remote collaboration

Multimodal interfaces and communication cues for remote collaboration

Sharing gaze rays for visual target identification tasks in collaborative augmented reality

Effects of personality traits on user trust in human–machine collaborations

Premium Partner