Skip to main content
SpringerLink
Log in

Projection-based visualization of tangential deformation of nonrigid surface by deformation estimation using infrared texture

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

In this paper, we propose a projection-based mixed reality system that visualizes the tangential deformation of a nonrigid surface by superimposing graphics directly onto the surface by projected imagery. The superimposed graphics are deformed according to the surface deformation. To achieve this goal, we develop a computer vision technique that estimates the tangential deformation by measuring the frame-by-frame movement of an infrared (IR) texture on the surface. IR ink, which can be captured by an IR camera under IR light, but is invisible to the human eye, is used to provide the surface texture. Consequently, the texture does not degrade the image quality of the augmented graphics. The proposed technique measures individually the surface motion between two successive frames. Therefore, it does not suffer from occlusions caused by interactions and allows touching, pushing, pulling, and pinching, etc. The moving least squares technique interpolates the measured result to estimate denser surface deformation. The proposed method relies only on the apparent motion measurement; thus, it is not limited to a specific deformation characteristic, but is flexible for multiple deformable materials, such as viscoelastic and elastic materials. Experiments confirm that, with the proposed method, we can visualize the surface deformation of various materials by projected illumination, even when the user’s hand occludes the surface from the camera.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. See http://en.wikipedia.org/wiki/Flubber_(material) for more information.

References

  • Alvarez L, Weickert J, Snchez J (2000) Reliable estimation of dense optical flow fields with large displacements. Int J Comput Vis 39(1):41–56

    Article  MATH  Google Scholar 

  • Bandyopadhyay D, Raskar R, Fuchs H (2001) Dynamic shader lamps: painting on movable objects. In: Proceedings of the IEEE/ACM international symposium on augmented reality, pp 207–216

  • Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. A. K. Peters, Ltd., USA

    Google Scholar 

  • Bimber O, Iwai D (2008) Superimposing dynamic range. ACM Trans Graph 27(5):15:1–15:8

    Article  Google Scholar 

  • Bluteau J, Kitahara I, Kameda Y, Noma H, Kogure K, Ohta Y (2005) Visual support for medical communication by using projector-based augmented reality and thermal markers. In: Proceedings of the international conference on artificial reality and telexistence, pp 98–105

  • Chang RC, Tseng FC (2010) Automatic detection and correction for glossy reflections in digital photograph. In: 3rd IEEE international conference on Ubi-media computing (U-Media), pp 44–49

  • Follmer S, Johnson M, Adelson E, Ishii H (2011) deform: An interactive malleable surface for capturing 2.5d arbitrary objects, tools and touch. In: Proceedings of the 24th annual ACM symposium on user interface software and technology. ACM, New York, UIST’11, pp 527–536

  • Fujimoto Y, Smith R, Taketomi T, Yamamoto G, Miyazaki J, Kato H, Thomas B (2014) Geometrically-correct projection-based texture mapping onto a deformable object. IEEE Trans Vis Comput Graph 20(4):540–549

    Article  Google Scholar 

  • Haouchine N, Dequidt J, Kerrien E, Berger MO, Cotin S (2012) Physics-based augmented reality for 3d deformable object. In: Workshop on virtual reality interaction and physical simulation

  • Heo S, Lee G (2013) Indirect shear force estimation for multi-point shear force operations. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, CHI’13, pp 281–284

  • Hisada M, Yamamoto K, Kanaya I, Sato K (2006) Free-form shape design system using stereoscopic projector—hyperreal 2.0. In: SICE-ICASE international joint conference, pp 4832–4835

  • Ito Y, Kim Y, Obinata G (2014) Acquisition of contact force and slippage using a vision-based tactile sensor with a fluid-type touchpad for the dexterous handling of robots. Adv Robot Autom 3(116)

  • Iwai D, Sato K (2010) Document search support by making physical documents transparent in projection-based mixed reality. Virtual Real 15(2–3):147–160

    Google Scholar 

  • Johnson MK, Cole F, Raj A, Adelson EH (2011) Microgeometry capture using an elastomeric sensor. ACM Trans Graph 30(4):46:1–46:8

    Article  Google Scholar 

  • Jones BR, Benko H, Ofek E, Wilson AD (2013) Illumiroom: peripheral projected illusions for interactive experiences. In: Proceedings of the ACM annual conference on human factors in computing systems, pp 869–878

  • Kamiyama K, Vlack K, Mizota T, Kajimoto H, Kawakami N, Tachi S (2005) Vision-based sensor for real-time measuring of surface traction fields. IEEE Comput Graph Appl 25(1):68–75

    Article  Google Scholar 

  • Kocev B, Ritter F, Linsen L (2013) Projector-based surgeon–computer interaction on deformable surfaces. Int J Comput Assist Radiol Surg 8(6):1015–1025

    Article  Google Scholar 

  • Matoba Y, Sato T, Takahashi N, Koike H (2012) Claytricsurface: an interactive surface with dynamic softness control capability. In: ACM SIGGRAPH emerging technologies, p 6:1

  • Mine M, van Baar J, Grundhofer A, Rose D, Yang B (2012) Projection-based augmented reality in disney theme parks. IEEE Comput 45(7):32–40

    Article  Google Scholar 

  • Ni T, Karlson AK, Wigdor D (2011) AnatOnMe: facilitating doctor-patient communication using a projection-based handheld device. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 3333–3342

  • Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76(2):109–122

    Article  Google Scholar 

  • Piper B, Ratti C, Ishii H (2002) Illuminating clay: a 3-d tangible interface for landscape analysis. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 355–362

  • Raskar R, Welch G, Low KL, Bandyopadhyay D (2001) Shader lamps: animating real objects with image-based illumination. In: Proceedings of the eurographics workshop on rendering, pp 89–102

  • Raskar R, Ziegler R, Willwacher T (2006) Cartoon dioramas in motion. In: Proceedings of the international symposium on non-photorealistic animation and rendering, pp 7–12

  • Rivers A, Adams A, Durand F (2012) Sculpting by numbers. ACM Trans Graph 31(6):157:1–157:7

    Article  Google Scholar 

  • Saakes D, Chiu K, Hutchison T, Buczyk BM, Koizumi N, Inami M, Raskar R (2010) Slow display. In: ACM SIGGRAPH emerging technologies, p 22

  • Sato T, Mamiya H, Koike H, Fukuchi K (2009) PhotoelasticTouch: transparent rubbery tangible interface using an LCD and photoelasticity. In: Proceedings of the 22nd annual ACM symposium on User interface software and technology—UIST ’09. ACM Press, New York, pp 43–50

  • Schaefer S, McPhail T, Warren J (2006) Image deformation using moving least squares. ACM Trans Graph 25(3):533–540

    Article  Google Scholar 

  • Shi J, Tomasi C (1994) Good features to track. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 593–600

  • Shimazu S, Iwai D, Sato K (2011) 3d high dynamic range display system. In: Proceedings of the 10th IEEE/ACM international symposium on mixed and augmented reality, pp 235–236

  • Shimizu N, Yoshida T, Hayashi T, de Sorbier F, Saito H (2013) Non-rigid surface tracking for virtual fitting system. In: International conference on computer vision theory and applications, pp 1–7

  • Steimle J, Jordt A, Maes P (2013) Flexpad: highly flexible bending interactions for projected handheld displays. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 237–246

  • Uchiyama H, Marchand E (2011) Deformable random dot markers. In: Proceedings of IEEE international symposium on mixed and augmented reality, pp 237–238

  • Uchiyama H, Saito H (2011) Random dot markers. In: 2014 IEEE virtual reality (VR), pp 35–38

  • Vlack K, Mizota T, Kawakami N, Kamiyama K, Kajimoto H, Tachi S (2005) GelForce: a vision-based traction field computer interface. In: CHI ’05 extended abstracts on human factors in computing systems. ACM, New York, pp 1154–1155

  • Wu HY, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W (2012) Eulerian video magnification for revealing subtle changes in the world. ACM Trans Graph 31(4):65–72

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Osaka University, 1-3 Machikaneyama, Toyonaka, Osaka, 560-8531, Japan

    Parinya Punpongsanon, Daisuke Iwai & Kosuke Sato

Authors
  1. Parinya Punpongsanon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daisuke Iwai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kosuke Sato
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Parinya Punpongsanon.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Punpongsanon, P., Iwai, D. & Sato, K. Projection-based visualization of tangential deformation of nonrigid surface by deformation estimation using infrared texture. Virtual Reality 19, 45–56 (2015). https://doi.org/10.1007/s10055-014-0256-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-014-0256-y

Keywords

Search

Navigation