Holorailway: an augmented reality system to support assembly operations in the railway industry

The IAR field was growing at the end of the 1990s with initiatives such as the ARVIKA [24] and ARTESAS projects [10], though the term “industrial augmented reality” was not introduced until 2011 by Georgel [18] when trying to define AR’s use in industrial processes, such as assembly [25], and manufacturing [26], product design [27], maintenance [28], among others. Moreover, this technology provides powerful tools that assist operators in their assigned tasks, specially, when they are complex [29]. In the following years, AR’s presence increased, to where its importance has been recognized and classified by the European Union as a core technology that will lead to the development of smart factories [30]. However, its implementation in the industry is still scarce compared to other technologies like robotics and automation [13, 17, 26, 31].

There are some industrial fields where AR is applied successfully, such as maintenance and repair [32], design [33], and training [10]. These applications and AR’s improvements to these fields are briefly explained in addition to some proposed approaches.

AR is applied in the maintenance and repair field to make tasks efficient and error-free by providing workers with augmented instructions and/or virtual content like a DT [34] aligned with the real scene or object of interest used to carry out the machinery maintenance or repair task. For example, welding tasks [32] can be improved by aligning the DT with the real scene to highlight spot-weld locations to improve accuracy by 52%. Fiorentino et al. [35] introduced a method that uses AR for maintenance tasks and achieved a significant reduction in errors (92%) and time (79%). During the last few years, remote assistance has been explored [36] in machine monitoring or repair. Using IAR could improve communication between workers and facilitate access to documentation (manuals, 3D models, etc.). Additionally, these applications allow an expert to visualize any problems, and any directions they give can be visualized by a worker on-site [28].

Another important field where AR is applied is training, where it can shorten the time required to train new employees and, increase information retention, cost savings, safety, realism, scalability, and flexibility [10]. Augmented reality training can be more efficient, effective, interactive, and engaging than traditional training methods and can be accessed remotely and on demand. In addition, AR can also be used to simulate dangerous environments, allowing employees to practice safety procedures without risking injury. For example, Hořejší [37] proposed virtual training using an AR system that added virtual 3D model instructions into a real scene for workers who used machinery. They concluded that using AR, reduced the skill requirements and training time. Moreover, Füchter et al. [38] presented an approach for creating a proto-type app for smart-phones to train new pilots, enhancing safety, and accelerating the training process.

Within the product development field, product design is a crucial task that can be enhanced through AR technology [33]. This allows for realistic simulation of the final product while making design changes in real-time, allowing for faster iterations and more accurate designs. For instance, in the naval industry, Oh et al. [27] proposed a method based on mobile AR to visualize 3D models linked to design drawings. In the automotive industry, Fründ et al. [39] introduced an AR method that could evaluate different car interiors during initial development phases by overlaying 3D models on a real car.

Assembly is a vital aspect that is considered throughout the design and manufacturing process, but in some sectors, it is still done using traditional techniques. In the traditional assembly process, different components or parts are manually put together to form a finished product. This process usually involves the use of hand tools, and manual labor. The assembly process may include tasks such as fitting and fastening parts in complex systems with many components, and testing the finished product. Typically, these tasks require a heavy mental workload [40] and an experienced worker who understands the technical information describing the required tasks [41]. This process can be time-consuming and requires skilled workers who have been trained in specific assembly techniques. It can also be prone to errors and inconsistencies, as each worker may take a slightly different approach to the task.

The primary objective of using AR in assembly tasks is to guide the operators performing the tasks. Simplifying assembly instructions by using 3D models aligned with the real-world scene [42], videos, etc., makes assembly safer, more accurate, and more efficient. It reduces the cognitive workload, which is closely linked to task performance [43]. Industrial augmented reality assembly methods were also developed in different fields such as the naval industry [10], aerospace [42], automotive [9], and agricultural machinery [44], among others.

Within the automotive sector, Lima et al. [25] suggested a system to guide the worker to the 3D location in different assembly processes. This sector also uses assembly simulation and planning methods. For instance, Woll et al. [45] developed an AR-based simulation system for assembling a car power generator.

Another industrial sector where AR is applied to assembly tasks is the naval sector. For instance, Fraga-Lamas et al. [10] reviewed different use cases where IAR assembly systems could be applied in a shipyard.

The aerospace industry also applied AR in assembly tasks. Mizell [46] evaluated the effectiveness of using a HMD in wire assembly tasks for Boeing airplanes. Liu et al. [47] introduced an IAR assembly system to increase task efficiency and guide the worker by creating a DT that is enhanced with 3D information.

In addition, AR is applied to assembly tasks in manufacturing sensors or machinery. For instance, Lai et al. [48] introduced an AR assembly system to guide workers in producing CNC carving machines. Richardson et al. [49] experimented with three different methods to perform the same assembly task for sensors in the military sector. It was concluded that the AR method obtained better performance in terms of time and efficiency.

Although AR applications in assembly tasks are becoming mature, some technical and functional limitations still need to be solved [15].

First, no agreement exists on the tracking method to be used. Multiple studies propose fiducial marker-based tracking methods [16] like ArUco [50] due to their simplicity and low computational cost. However, this could be too sensitive to occlusions and non-controlled light conditions. In addition, in some cases, it is not possible to place the markers in the scene. Alternatively, marker-less tracking methods, such as SLAM, were proposed [25, 47] in the industrial field to align the DT with the real scene. However, the proposed SLAM tracking approaches accumulate errors over time and they are generally not recommended for dynamic scenes, where the environment should be highly textured. As previously stated, each tracking method has its own set of benefits and limitations. It is crucial to choose and adapt the appropriate tracking approach based on the specific needs and characteristics of the industrial environment and application [48].

In terms of hardware devices, a clear division could be established between mobile devices and optical see-through glasses. On the one hand, many assembly tasks using AR can be performed using mobile devices such as smartphones or tablets. However, in industrial environments, many tasks are done by hand and, it is required to be hands-free, so holding a device is unfeasible [51]. On the other hand, optical see-through devices have experienced a significant evolution. Traditionally, devices like HMDs were too heavy, and had short battery lives, limited fields of view, and too many wires [17]. In the last few years, HMDs have progressed rapidly, with advancements in technology leading to improved resolution, field of view, and overall user experience [52]. Microsoft HoloLens 2 and Magic Leap 2 represent the most recent evolution in this type of device, with many examples of successful industrial applications [20, 52‐54].

Many AR systems are used as independent solutions for specific tasks, rather than as integrated components of a company’s workflow and IT infrastructure [36]. One of the main reasons for this specificity is the need to translate the CAD files describing the DT to the proper format to develop an AR application. Usually, this step requires manual adjustments. The development of AR resources is not simple, scalable or flexible and programming knowledge is required [55, 56].

Although there are some examples that test the IAR system in a production environment (e.g., the Volkswagen factory) [25], most of them are validated only under laboratory-controlled situations [45, 49]. Specifically, they tend to simulate the production conditions [48].

Despite the strategic importance of the railway sector in the industry, there are few AR applications in this field, and these are primarily focused on maintenance. For instance, Azpiazu [57] presented a remote support application, where a worker communicated with a specialist through glasses, and the specialist could add elements such as arrows or signs to the worker’s visualization to improve maintenance times and costs.

Furthermore, Kwon et al. [58] implemented an AR system for maintenance and training on the Axle-mounted Disc Brake System using a hand-held device (mobile or tablet). The results showed an improvement in efficiency of 34% and a reduction in errors on this task.

AR has been applied to inspect freight cars’ routes with out-of-gauge cargo [59], using AR-based interference detection to detect possible collisions between the train and the environment and select the optimal route. This system showed lower cost and high precision when compared with other methods.

After reviewing the literature, it could be concluded that, to the best of our knowledge, no research papers show the application of AR in the manufacturing and production of railway vehicles mainly it is a consequence of the huge amount of challenges in the implementation of AR in this sector detailed in Sect. 1. Thus, it could be beneficial [10] to develop a powerful IAR system such as the one proposed in the following sections. First, it should avoid discontinuities and gaps in the process because they can affect functionality. Second, the system should be smooth and easy to use for those users who are unfamiliar with AR technology. For example, the interaction should be user-friendly by avoiding functional inconsistencies and integrating the end users in the development loop. All these aspects have been taken into account for the proposed Holorailway system, explained in detail in the following sections.

The device used to implement this system is Microsoft’s HoloLens 2, as depicted in Fig. 4, an autonomous HMD with see-through holographic lenses and dual wave-guides configuration. This device has two layers, one for red and green and the other one for green and blue.

The main reason for using HoloLens 2 is its proven effectiveness in implementing IAR applications in the industry [54, 60, 61]. This efficacy can be attributed to the device’s autonomy, computing capacity, ease of interaction, and reduced weight, which improves ergonomics. It also allows users to view instructions hands-free while performing a task. Additionally, compared to its predecessor, HoloLens 2 features some meaningful upgrades, with enhancements in the field of view, which is larger; a more ergonomic system which adds new and more natural gestures, including the ability to touch the holograms; an added eye-tracking method; voice instruction in various languages; and improvements on the hardware, including a co-processor designed expressly for AI. Moreover, HoloLens 2 presents a high level of precision required to carry out the assembly task. In this sense, to our knowledge, no study related to precision in locating components in the industrial sector using this device exists. However, in the medical field, several studies exist. For instance, Unger et al. [62] established a tolerance of 2 cm that was outperformed by HoloLens 2.

The device features 2k resolution, a time-of-flight depth sensor, and an IMU sensor with an accelerometer, gyroscope, and magnetometer. HoloLens 2 has four visible light cameras for head tracking and two IR cameras for eye tracking, which automatically calibrate according to the user’s inter-pupillary distance. Consequently, it reduces eye strain, and can work as a retinal scanner to identify the user. Its total weight is 566 g and has an autonomy of up to 3 h of active use, which can be increased to the duration of an entire work shift using an external battery.

Regarding its understanding of the environment, the HoloLens 2 is equipped with a 6DoF tracking system that allows the user to move around the virtual space, a spatial mapping system that reproduces a real-time environment mesh, which is essential for placing objects on real surfaces, and mixed reality capture, which provides photos and video of a mixed holographic and physical environment.

Furthermore, the device presents human understanding capabilities, such as two-handled, fully articulated hand tracking, real-time eye tracking and voice commands, and on-device controls.

The process described above (see Sect. 3) is subject to improvement. With that aim, an IAR system, Holorailway, is proposed to enhance the assembly process with the use of a DT and augmented instructions. Figure 5 depicts the steps that the operator would follow when using the proposed AR system.

The first step is represented in Fig. 5 - initial alignment and it is explained in depth in Sect. 4.3.2. After that, the other four steps are repeated. However, some significant changes to the traditional approach were implemented to make the task more efficient. Mainly, a QR code is used to identify the components to avoid using the alpha-numeric label. Concretely, a QR code is scanned with HoloLens 2 (see Fig. 5 - second step), and quickly and automatically identifies the part in the database. Therefore, the operator should get the component with the QR code (see Fig. 5 - first step), which would be scanned and detected with HoloLens 2. Then, the ID of the QR code is compared to the ones in the database to determine if it is a unique code or a repeated one. The database is a list with the ID of the components that define the JT file; these IDs are the same as the QR codes. Our AR-based solution incorporates a register system that stores the placed components and their respective 3D positions, thus, in the case of repeated QRs (i.e., identical components), the system will guide the operator to the closest available location with respect to the operator’s position having into account the information of that register. To compute the closest available position it is used Eq. (1),where p_track is the 3D position of the operator, obtained by the tracking method (based on the SLAM technique incorporated in HoloLens 2) and p_component is the 3D position of the component saved previously in the DT. It is important to note that, because the alignment is performed in the previous step using tracking techniques, the DT’s coordinate system coincides with the real-world coordinate system (see Fig. 5 - initial alignment). When the QR code is detected (see Fig. 5 - second step) and the model of the railway is aligned with the real one (explained in depth in Sect. 4.3.2), Holorailway starts to guide the operator with a 3D arrow (see Fig. 5 - third step) and visually indicates where the component should be installed. An example of how it is seen is depicted in Fig. 6. The GUI displays a red arrow, which guides the operator to where the component should be installed. Likewise, the blue part represents the DT of the wagon previously aligned with the real wagon. The yellow part marks the position where the component should be set up. A text box is also depicted to provide the operator with instructions, such as the component’s code or the option to use a voice command if the element needs to be duplicated. As can be observed, the application is quite intuitive due to its minimalist design.

The operator should be able to clean the area and install the component without being obstructed by the hologram (see Fig. 5 - fourth step). This is why a shader is applied to the object to indicate its position. As the operator approaches, a shader appears to visually guide the operator until it disappears when the operator is close enough to the element. This can be represented mathematically in Eq. (2),where d is the distance (unit: m). In the particular case of duplicated components, the process ends when the operator uses the specific voice command “placed”. For voice commands, it was proven that in spite of working in an industrial setting, the voice commands were detected accurately due to the incorporation of directional microphones on HoloLens 2, which recognized the operator who used the headset. However, to make the system more efficient, it was determined to use voice commands only in required cases like the one mentioned. Therefore, the system has been designed to minimize interaction mainly because of its efficiency, effectiveness, and usability, and because interaction is not required in many situations. The headsets guide the operator to the area where the pieces need to be assembled by indicating their exact position and orientation. This guidance is done accurately and quickly, making the process of locating the position and orientation more efficient.

In summary, the need for identification codes, bills of materials, and installation drawing are eliminated. The process now consists of scanning the components’ QR code, following the indications of the headset to the installation position, cleaning the area, and installing the part, reducing the total installation time of the insulation material significantly.

This section explains the pre-operational procedures, specifically how the system is created and the requirements used to make Holorailway run. Although the aforementioned hardware incorporates useful techniques, it should be integrated with other methods not only to complete the entire process but also to make the system more precise and reliable.

The previous sections explained how the application was created. The following will describe the processes that work in the background while the application is running and the requirements needed. This section describes how the tracking is done and how the QR detection is carried out.

In the tracking step, the system not only uses a marker-based technique using Vuforia but also takes advantage of the tracking method used natively in HoloLens 2, which is a markerless method known as SLAM. By applying both techniques, it overcomes some of the limitations that both kinds of tracking methods are not able to solve independently, such as the accumulative error obtained by markerless methods in dynamic scenes or problems with occlusions when the tracking is done using a marker-based algorithm.

As previously mentioned, the marker-based method is used to perform the initial alignment. Once it is finished, the camera pose could be estimated during the process using the different sensors in HoloLens 2 using SLAM. This technique reconstructs the environment while it estimates the position and orientation of the camera with respect to the environment. This also allows the user to know the position in the projected virtual environment so that the user can naturally move around the DT. Additionally, the use of spatial mapping is essential for AR applications as it provides a more immersive experience, making it possible to place objects on real surfaces, create familiar, real-world behaviors and interactions with the holograms, and facilitate navigation in the environment.

Both tracking techniques are applied to obtain high spatial accuracy; therefore, the alignment of holograms in 3D space should be precise. To evaluate the spatial accuracy generated by HoloLens 2, a 3D reconstruction of the wagon was obtained using the Truncated Signed Distance Function (TSDF) volume method [63] to prove the proper functioning of the SLAM camera pose estimation process, avoiding the accumulated error time known as drift. Specifically, the TSDF volume approach could be used with HoloLens 2 to create the 3D color reconstruction of the physical spaces using depth, RGB frames and head poses. A visual example of the reconstructed model is presented in Fig. 8, which shows the 3D reconstruction of the wagon obtained with the TSDF volume method compared with its DT. Additionally, the DT follows a validation process by the engineering department of the company involved. Specifically, the JT file used in the assembly is directly exported from Siemens NX (12.0.2) modeling software and is validated to ensure that it is a faithful representation of the engineering model. Then, the quality department of the company employs various methods for measuring errors, including visual and manual inspections, as well as specialized software like PolyWorks (2021 IR11.5), to compare tolerances between the digital and the physical models. For instance, in the case of a wagon chassis with dimensions (depth, height, width) 20 000 mm × 3 000 mm × 3 000 mm, the maximum error allowed is equal to 15 mm in depth and 5 mm in height and width.

The second step required during the pipeline is the detection of the QR codes previously appended in the JT file. To detect the QR codes, the method integrated into HoloLens 2 is used. According to the current best practices in Microsoft guidelines, the QR code detectors work well under good lighting conditions and with proper size and distances. However, they do not perform well if the codes are small, poorly visible, or too far from the headset. Specifically, in our case the QR code used is approximately 5 cm × 5 cm and there is a margin around all sides of the code and no printed content. The distance between the QR code and the HMD is the length of the operator’s arm, which could be between 30 cm and 50 cm, and they should view this QR frontally to guarantee the condition of the manufacturer.