Virtual and Augmented Reality Applications in Manufacturing

In the current highly competitive business and manufacturing environment, manufacturing industry is facing the constant challenge of producing innovative products at reduced time-to-market. The increasing trend of globalized manufacturing environments requires real-time information exchanges between the various nodes in a product development life cycle, e.g., design, setup planning, production scheduling, machining, assembly, etc., as well as seamless task collaboration among these nodes. In addition, with increased environmental awareness and legislation, more constraints have been placed on product disposal, hence promoting product recycling, servicing and repairing activities. Product development processes are becoming increasingly more complex as products become more versatile and intricate, and inherently complicated, and as product variations multiply with the trend of mass customization. Thus, manufacturing processes have to be more systematic in order to be efficient and economically competitive. An innovative and effective solution to overcome these problems is the application of virtual reality (VR) and augmented reality (AR) technologies to simulate and improve these manufacturing processes before they are carried out. This would ensure that activities such as design, planning, machining, etc., are done right-the-first-time without the need for subsequent rework and modifications.

With today’s virtual reality (VR) systems, it is difficult to directly and precisely create and modify objects in a VR environment. This chapter presents an approach for solid modelling in a VR environment. Solid modelling in the VR environment is performed precisely in an intuitive manner through constraint-based manipulations. A hierarchically structured and constraint-based data model is developed to support solid modelling in the VR environment. The data model integrates a high-level constraint-based model for precise object definition, a mid-level constructive solid geometry/boundary representation (CSG/BRep) hybrid solid model for hierarchical geometry abstractions and object creation, and a low-level polygon model for real-time visualization and interaction in the VR environment. Constraints are embedded in the solid model and are organized at different levels to reflect the modelling process from features to parts. Constraint-based manipulations are accompanied with automatic constraint recognition and precise constraint satisfaction to establish the hierarchically structured constraint-based data model and are realized by allowable motions for precise 3D interactions in the VR environment. The allowable motions are represented as a mathematical matrix for conveniently deriving allowable motions from constraints. A procedure-based degree-of-freedom (DOF) combination approach for 3D constraint solving is presented for deriving the allowable motions.

This chapter presents the development of a Virtual Sculpting system and addresses the issues of interactive solid modelling with a haptic interface. A virtual reality (VR) approach is used to make the sculpting process more intuitive and interactive, by providing stereo viewing and force feedback capabilities in carving a solid block into a 3D free-form object. The geometric modelling in the system is based on the sweep differential equation method to compute the tool swept volume, and uses the ray-casting method to perform Boolean operations between the tool swept volume and the virtual stock in dexel data. The PHANToM manipulator is used as an input device to provide the position and orientation of the sculpting tool, and also as an output device to provide force feedback during the sculpting process. Multithreading is used to help address the different update rates required in the graphic rendering and haptic rendering.

In this chapter, an efficient technique for distributing the data in collaborative virtual reality simulations is presented. The technique described incorporates the culling and level of detail concepts in virtual reality (VR) to obtain cell-based bounding volumes in each virtual environment. Defined bounding volumes are utilized in filtering the data that is transferred between different virtual environments in the simulation system. Depending on the nature of the data and the current relationships of the virtual cells and collaborator’s bounding volume, the selection of one of the data-sharing options (real-time or delayed data transferring) by utilizing the supervisory control system is described.

This chapter presents an approach to integrate a virtual reality (VR) environment with manufacturing control methodologies leading to enhanced visualization and control of manufacturing processes. A Petri Net-based model is used to represent the control architecture of a manufacturing system. A Decision Support System (DSS) is developed here that integrates the process control methodology designed using Petri nets with a VR-based 3D immersive environment. Process plans provide the inputs to the system in terms of operations to be performed. A web-based implementation of the DSS is shown to illustrate the interfacing of the Petri Net execution with the VR system. The real-time visualization of a process flow leads to an easy analysis of bottleneck situations and could lead to significant cost savings during process optimization.

This chapter discusses the concept and the first pilot implementation of a hybrid approach to the realistic representation of human performance in virtual manufacturing environments. The approach employs simulation principles of both virtual reality (VR) and digital mannequin (DM) technologies. The software tool developed, called VRSP, enables the immersive virtual performance of an industrial task, in an adequately realistic and user-friendly way, and the control of a mannequin within the virtual environment, based upon the spatially tracked user interactions of the immersed human. Design and implementation aspects of VRSP are described. An industrial test case is presented to demonstrate the capability of VRSP to deal with real process tasks. There follows a discussion, focused on the results and the future perspectives of this research work.

The purpose of this research is to develop new a virtual assembly (VA) and disassembly system that uses natural interaction and control of objects and machinery. This VA system deals with parts, tools, machines and workplaces positioned in their usual places and working in the expected ways. We developed a system optimized for both performance and ease of use. For example, tools and machines work are controlled by a user with his/her hands, and he/she responds, moves, and works as expected. Instead of the traditional interactions with a keyboard/mouse, hand gestures, body movements, touching, holding, and carrying of the parts/tools/sub-assemblies, pushing buttons for machines and equipment controls are performed in a natural way. In addition, helpful information to enhance the immersiveness to substitute for missing natural factors like weight, inertia, and force are introduced. Depending on the desired effect, this may be the sound of a working machine, a rotating gear, a snapping sound between two parts, or the dropping a part on the floor or table. Further, a warning sound is used for: signalling of impossible operations, changing colour of object is used when hand/part collision detection occurs to show when a certain part can be grasped or assembled/disassembled. Finally, models and simulation optimization allow a high rendering frame rate and real-time interaction using moderate performance personal computers with affordable peripheral devices.

Augmented reality (AR) systems can be effectively used to enhance manufacturing and industrial processes. However, not all the existing prototypes of AR systems can be used in an industrial environment due to heavy constraints such as low robustness or cumbersome equipment. Our AR system relies purely on passive techniques to solve the real-time registration problem, and it can run on a portable computer. We combined a powerful virtual reality (VR) component-based simulation framework with computer vision techniques, turning it into an AR system. The resulting system allows us to produce complex rendering and animation of avatars, and blend them into the real world. The system tracks the 3D camera position by means of a natural features tracker, which, given a rough CAD model, can deal with complex 3D objects. The tracking method is robust and can handle large camera displacements and aspect changes. The target applications of our AR system are industrial maintenance, repair and training. The tracking robustness makes the AR system able to work in real environments, such as industrial facilities, and not only in the laboratory.

An ongoing research problem in augmented reality (AR) is to improve tracking and display technology in order to minimize registration errors. However, registration is not always necessary for users to understand the intent of an augmentation, especially in industrial applications where the user and the system have extensive semantic knowledge of the environment. In this chapter, we review the ideas of communicative intent developed for desktop graphical explanation systems by Seligmann and Feiner, and discuss how these approaches are the basis for our hypothesis that semantic knowledge of a scene can be used to ameliorate the effects of registration errors. We describe a set of AR visualization techniques for augmentations that adapt to changing registration errors. We first define a set of strategies that use semantic knowledge of the augmentation to enhance the augmentations with additional contextual cues. These context cues help users understand the intent of the augmentation in the presence of registration error. We then introduce algorithms that use features and feature points on objects to control these strategies in the presence of changing registration errors. Finally, these algorithms and techniques are demonstrated in four maintenance situations that challenge a user’s ability to interpret the semantics of a scene.

Industrial enterprises are working in a difficult business environment: high dynamics, short innovation cycles and rising product complexity. This requires a fast and error-free planning of the manufacturing process. Subsequent necessary modifications of the manufacturing system and environment (e.g., movement of a column, redirection of a pipe system, exchange of machines) in a late phase are very time and cost intensive. Therefore, the planning of manufacturing systems is becoming an increasingly continuous task, with time and correctness being a critical success factor.

Components and potential manufacturing applications of the teleportal augmented reality (AR) system are described. This teleportal system is designed to support applications such as distributed 3D design and work-team collaboration. The optomechanical design of an emerging type of AR head-mounted display (HMD), referred to as the Teleportal Head-Mounted Projection Display (T-HMPD) is detailed. A feature of HMPDs is the invariance of the optics size and weight across a significant increase in field of view (FOV). Results are shown for 52° and 70° FOVs projection optics. Research on associated technologies and methods that provide the basis for an integrated distributed manufacturing AR system is introduced, which includes calibration and registration of virtual and physical objects, the creation of AR tool spaces around the body of a mobile user, a face-to-face collaboration tool, and finally an integration of the teleportal AR technologies within the Artificial Reality Centre (ARC) Work Room.

Over the last few years, virtual reality (VR) has become part of the mainstream product design and development processes of many companies in the manufacturing industry. Augmented reality (AR) is a technology that promises to enhance these processes further by augmenting the real world with judiciously chosen digital information that facilitates decision making and evaluation of product variants. At DaimlerChrysler Research, we have developed a range of AR applications that support several stages of the product lifecycle.

Service applications for machine tools are currently under substantial development. One technology playing a major part in this area is augmented reality (AR), which has been recognized as being capable of assisting a service technician in complex maintenance and repair situations. However, for an efficient use, this new technology needs to integrate with the company’s informational infrastructure. This includes access to process data and information retrieval from enterprise systems. Moreover, authoring tools, which can establish and manipulate AR-based documents, are still in an early stage of development. Furthermore, difficulties are caused by insufficient interaction technologies. Normally, the technician needs both hands to operate technical devices. Speech control and the use of head-mounted-displays (HMDs) are two ways of interacting independent of hands, eyes and working location.

An augmented reality (AR) environment was developed to validate the 3D dynamic simulation of a parts feeding system by augmenting virtual objects with real images of an experimental background. By means of Taguchi’s orthogonal arrays, the possible contributing factors for the collision simulation were analyzed to align virtual objects with real systems and reduce the position mismatches made by the simulator. In case studies, simulation of a parts feeding system was conducted using a commercial simulator and was subsequently visualized in the AR environment to compare with images from the physical experiments. This technology also sets a procedure to calibrate and fine-tune augmented systems with precision requirements.

We consider the problem of scene augmentation in the context of a human engaged in assembling an object from its components. In order to exploit the potential of augmented reality (AR) in this context, two main problems need to be considered: designing an effective augmentation scheme for information presentation/control, and providing accurate and fast sensing to determine the state of the assembly. We utilized concepts from robot assembly planning to develop a systematic framework for presenting augmentation stimuli for the assembly domain. An interactive augmentation design and control engine called AUDIT is described. To provide sensing, we utilized computer vision methods for assembly object recognition without special markers. Even though fiducials currently constitute the only feasible vision-based solution, occlusion by the manipulator as well as other assembly parts make the use of more general computer vision techniques desirable. Here, we investigate computer vision techniques with the goal of eventually substituting markers. Constraints from the domain of assembly, as well as transformation space search-based algorithms make the problem tractable.

Although there has been great speculation about the potential of augmented reality (AR) in manufacturing applications, there have been very few empirical studies that assess the effectiveness of the technology. This chapter describes an experiment that tested the relative effectiveness of AR instructions in computer-assisted assembly. The AR solution displayed task information in the user’s field of view (FOV) as 3D objects registered with the workspace. The presentation demonstrates the exact execution of a procedural step. Three traditional instructional media approaches were compared with the AR system: a printed manual, computer-assisted instruction (CAI) using a monitor-based display, and CAI utilizing a head-mounted display (HMD). Results indicate that overlaying 3D instructions on the actual work pieces reduced the error rate for an assembly task by 82%, particularly diminishing cumulative errors, errors due to previous assembly mistakes. Measurement of the mental effort indicated decreased mental effort in the AR condition, suggesting some of the mental calculation of the assembly task is offloaded to the system. The results indicate that an AR system for computer-assisted assembly can improve worker performance.

This chapter presents the prototypical design and implementation of an Intelligent Welding Gun to help welders in the automotive industry shoot studs with high precision into experimental vehicles. A presentation of the stud welding scenario and the system requirements identified is followed by a thorough exploration of the design space of the potential system setups, analyzing the feasibility of different options to place sensors, displays and landmarks in the work area. The setup yielding the highest precision for stud welding purposes is the Intelligent Welding Gun, which is a regular welding gun with a display attachment, a few buttons for user interactions, and reflective markers to track the gun position from stationary cameras. While welders operate and move the gun, the display shows 3D stud locations on the car frame relative to the current gun position. Navigational metaphors, such as notch and bead and a compass, are used to help welders place the gun at the planned stud positions with the required precision. The setup has been tested by a number of welders. It shows significant time improvements over the traditional stud welding process. It is currently in the process of being modified and installed for production use.

An augmented reality (AR) system integrated in a newly developed welding helmet improves the conditions for manual welding. The scene is acquired by a stereoscopic high dynamic range CMOS (HDRC) camera system that allows the observance of the welding arc and the environment simultaneously. Using image processing, the scene will be enhanced and displayed on a video see-through head mounted display (HMD). Besides a better view of the welding scene, the welder can operate an “on-screen” remote control of the welding machine to change machine parameters without lifting the helmet or changing the working position. During the welding process, the inclusion of additional on-line process data provides more information about the quality of the seam and erroneous states like sputter or sub-optimal welding gun inclination. Electrical data and synchronized acquired images are recorded on-line and may be used for off-line documentation and quality inspection. With an AR helmet, the welder will be able to produce seams of higher quality and the inspection cost will decrease. The chapter presents the development and research of the helmet in detail.