Interacting with Real Objects in Virtual Worlds

In a virtual reality (VR) experience, the manner in which a user interacts with the virtual environment and computer generated objects in the scene greatly effects the feeling of immersion. Traditionally, VR systems use controllers as a means of facilitating interaction, with a sequence of button presses corresponding to a particular action. However, controllers do not accurately model the intuitive way to interact with a real-world object and offer limited tactile feedback. Alternatively, a physical object can be tracked and used to regulate the behaviour of a virtual object. In this chapter, we review a range of approaches which use the tracked behaviour of a physical object to control elements of the virtual environment. These virtual props have the potential to be used as a more immersive alternative to the traditional controllers. We discuss how motion capture systems and external sensors can be used to track rigid and non-rigid objects, in order to drive the motion of computer generated 3D models. We then consider two neural network based tracking solutions and explain how these can be used for transporting real objects into virtual environments.