Daniel Leithinger
Sean Follmer
Alex Olwal
Hiroshi Ishii
ABSTRACT
Recent research in 3D user interfaces pushes towards immersive graphics and actuated shape displays. Our work explores the hybrid of these directions, and we introduce sublimation and deposition, as metaphors for the transitions between physical and virtual states. We discuss how digital models, handles and controls can be interacted with as virtual 3D graphics or dynamic physical shapes, and how user interfaces can rapidly and fluidly switch between those representations. To explore this space, we developed two systems that integrate actuated shape displays and augmented reality (AR) for co-located physical shapes and 3D graphics. Our spatial optical see-through display provides a single user with head-tracked stereoscopic augmentation, whereas our handheld devices enable multi-user interaction through video seethrough AR. We describe interaction techniques and applications that explore 3D interaction for these new modalities. We conclude by discussing the results from a user study that show how freehand interaction with physical shape displays and co-located graphics can outperform wand-based interaction with virtual 3D graphics.
Supplemental Material
- Aksak, B., Bhat, P. S., Campbell, J., DeRosa, M., Funiak, S., Gibbons, P. B., Goldstein, S. C., Guestrin, C., Gupta, A., Helfrich, C., Hoburg, J., Kirby, B., Kuffner, J., Lee, P., Mowry, T. C., Pillai, P. S., Ravichandran, R., Rister, B. D., Seshan, S., Sitti, M., and Yu, H. Claytronics: highly scalable communications, sensing, and actuation networks. In SenSys '05 (2005), 299--299. Google Scholar
Digital Library
- Anabuki, M., and Ishii, H. Ar-jig: A handheld tangible user interface for modification of 3d digital form via 2d physical curve. In ISMAR '07 (2007), 1--10. Google ScholarDigital Library
- Benko, H., Jota, R., and Wilson, A. Miragetable: freehand interaction on a projected augmented reality tabletop. In CHI '12 (2012), 199--208. Google ScholarDigital Library
- Bimber, O., Frohlich, B., Schmalsteig, D., and Encarnacao, L. The virtual showcase. Computer Graphics and Applications, IEEE 21, 6 (nov/dec 2001), 48--55. Google ScholarDigital Library
- Bimber, O., and Raskar, R. Spatial augmented reality: merging real and virtual worlds. A. K. Peters, 2005. Google ScholarDigital Library
- Butler, A., Hilliges, O., Izadi, S., Hodges, S., Molyneaux, D., Kim, D., and Kong, D. Vermeer: direct interaction with a 360 deg viewable 3d display. In UIST '11 (2011), 569--576. Google ScholarDigital Library
- Hachet, M., Bossavit, B., Cohé, A., and de la Rivière, J.-B. Toucheo: multitouch and stereo combined in a seamless workspace. In UIST '11 (2011), 587--592. Google ScholarDigital Library
- Harrison, C., Benko, H., and Wilson, A. D. Omnitouch: wearable multitouch interaction everywhere. In UIST '11 (2011), 441--450. Google ScholarDigital Library
- Hart, S. G., and Staveland, L. E. Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Human Mental Workload, P. A. Hancock and N. Meshkati, Eds., vol. 52 of Advances in Psychology. North-Holland, 1988, 139--183.Google Scholar
- Hashimoto, S., Ishida, A., Inami, M., and Igarashi, T. Touchme: An augmented reality based remote robot manipulation. In ICAT2011 (2011).Google Scholar
- Henderson, S. J., and Feiner, S. Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality. In VRST '08 (2008), 211--218. Google ScholarDigital Library
- Hilliges, O., Kim, D., Izadi, S., Weiss, M., and Wilson, A. Holodesk: direct 3d interactions with a situated see-through display. In CHI '12 (2012), 2421--2430. Google ScholarDigital Library
- Hoshi, T., Takahashi, M., Nakatsuma, K., and Shinoda, H. Touchable holography. In SIGGRAPH '09 (2009), 23:1--23:1. Google ScholarDigital Library
- Ishii, H., Lakatos, D., Bonanni, L., and Labrune, J.-B. Radical atoms: beyond tangible bits, toward transformable materials. interactions 19, 1 (Jan. 2012), 38--51. Google ScholarDigital Library
- Ishii, K., Zhao, S., Inami, M., Igarashi, T., and Imai, M. Designing laser gesture interface for robot control. In INTERACT '09 (2009), 479--492. Google ScholarDigital Library
- Iwata, H., Yano, H., Nakaizumi, F., and Kawamura, R. Project feelex: adding haptic surface to graphics. In SIGGRAPH '01 (2001), 469--476. Google ScholarDigital Library
- Leithinger, D., Lakatos, D., DeVincenzi, A., Blackshaw, M., and Ishii, H. Direct and gestural interaction with relief: a 2.5d shape display. In UIST '11 (2011), 541--548. Google ScholarDigital Library
- Massie, T., and Salisbury, K. The phantom haptic interface: A device for probing virtual objects. In ASME Winter Annual Meeting (1994).Google Scholar
- Olwal, A., Lindfors, C., Gustafsson, J., Kjellberg, T., and Mattsson, L. Astor: An autostereoscopic optical see-through augmented reality system. In ISMAR '05 (2005), 24--27. Google ScholarDigital Library
- Piper, B., Ratti, C., and Ishii, H. Illuminating clay: a 3-d tangible interface for landscape analysis. In CHI '02 (2002), 355--362. Google ScholarDigital Library
- Plesniak, W., Pappu, R., and Benton, S. Haptic holography: a primitive computational plastic. Proceedings of the IEEE 91, 9 (sept. 2003), 1443--1456.Google ScholarCross Ref
- Poupyrev, I., Nashida, T., and Okabe, M. Actuation and tangible user interfaces: the vaucanson duck, robots, and shape displays. In TEI '07 (2007), 205--212. Google ScholarDigital Library
- Raskar, R., Welch, G., Low, K.-L., and Bandyopadhyay, D. Shader lamps: Animating real objects with image-based illumination. In Eurographics, Springer-Verlag (London, UK, 2001), 89--102. Google ScholarDigital Library
- Rasmussen, M. K., Pedersen, E. W., Petersen, M. G., and Hornbaek, K. Shape-changing interfaces: a review of the design space and open research questions. In CHI '12 (2012), 735--744. Google ScholarDigital Library
- Ratti, C., Wang, Y., Piper, B., Ishii, H., and Biderman, A. Phoxel-space: an interface for exploring volumetric data with physical voxels. In DIS '04 (2004), 289--296. Google ScholarDigital Library
- Reed, M. Prototyping digital clay as an active material. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction, TEI '09 (2009), 339--342. Google ScholarDigital Library
- Scharver, C., Evenhouse, R., Johnson, A., and Leigh, J. Designing cranial implants in a haptic augmented reality environment. Commun. ACM 47, 8 (Aug. 2004), 32--38. Google ScholarDigital Library
- Schmandt, C. Spatial input/display correspondence in a stereoscopic computer graphic work station. In SIGGRAPH '83 (1983), 253--261. Google ScholarDigital Library
- Stevenson, D. R., Smith, K. A., McLaughlin, J. P., Gunn, C. J., Veldkamp, J. P., and Dixon, M. J. Haptic workbench: a multisensory virtual environment. SPIE 3639, Stereoscopic Displays and Virtual Reality Systems VI (1999), 356--366.Google ScholarCross Ref
- Sutherland, I. The ultimate display. In In International Federation of Information Processing (1965).Google Scholar
- Tani, M., Yamaashi, K., Tanikoshi, K., Futakawa, M., and Tanifuji, S. Object-oriented video: interaction with real-world objects through live video. In CHI '92 (1992), 593--598. Google ScholarDigital Library
- Walairacht, S., Yamada, K., Hasegawa, S., Koike, Y., and Sato, M. 4 + 4 fingers manipulating virtual objects in mixed-reality environment. Presence: Teleoper. Virtual Environ. 11, 2 (Apr. 2002), 134--143. Google ScholarDigital Library
- Yoshida, T., Kamuro, S., Minamizawa, K., Nii, H., and Tachi, S. Repro3d: full-parallax 3d display using retro-reflective projection technology. In SIGGRAPH '10 (2010), 20:1--20:1. Google ScholarDigital Library
Index Terms
- Sublimate: state-changing virtual and physical rendering to augment interaction with shape displays
Recommendations
shapeShift: 2D Spatial Manipulation and Self-Actuation of Tabletop Shape Displays for Tangible and Haptic Interaction
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsWe explore interactions enabled by 2D spatial manipulation and self-actuation of a tabletop shape display. To explore these interactions, we developed shapeShift, a compact, high-resolution (7 mm pitch), mobile tabletop shape display. shapeShift can be ...
Mediate: A Spatial Tangible Interface for Mixed Reality
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing SystemsRecent Virtual Reality (VR) systems render highly immersive visual experiences, yet currently lack tactile feedback for feeling virtual objects with our hands and bodies. Shape Displays offer solid tangible interaction but have not been integrated with ...
To touch or not to touch?: comparing 2D touch and 3D mid-air interaction on stereoscopic tabletop surfaces
SUI '13: Proceedings of the 1st symposium on Spatial user interactionRecent developments in touch and display technologies have laid the groundwork to combine touch-sensitive display systems with stereoscopic three-dimensional (3D) display. Although this combination provides a compelling user experience, interaction with ...
Comments