Immersive Inspection: Intuitive Material Analysis using X-Ray Computed Tomography Data in AR

Over the last years researchers discovered promising opportunities for IA across diverse analytical domains [6, 12‐14]. It aims to improve data understanding and decision making by bridging the gap between the users and their data through more natural, engaging, and immersive analysis. Although IA encompasses a variety of technological modalities, recent frameworks have shown an increasing tendency to leverage AR and VR capabilities. Particularly in the context of Industry 4.0, AR acts as a key enabling technology, offering increased productivity, accuracy, and autonomy in the quality sector [4, 5].

For materials science, a field of research characterized by complex spatial structures and large amounts of high-dimensional data, IA offers a significant potential for transforming tasks such as materials characterization, quality control, and metrology. For these tasks, technologies such as AR offer new ways of engaging with data analysis. This may significantly change how we perform and interact with data in future as outlined in the following paragraphs:Based on these opportunities, we conclude that immersive analytics presents a compelling research direction for materials science, with AR emerging as a particularly promising technology for industrial NDT applications.

Our focus regarding applications is on IA systems that employ spatial data, particularly within the domain of material science and that facilitate an analysis. There are few immersive analytics systems available analyzing volumetric data as well as high-dimensional data, and they are divided between AR- and VR-focused systems. However, systems that allow for the material characterization of the generated NDT data by means of immersive techniques are not widely available. This highlights a significant gap in literature. A large proportion of applications in the industrial environment involves assembly, maintenance, and training tasks. Such tasks are not relevant to our research focus. Here, we rather present the most relevant work for AR and VR based analysis and visualization of NDT data.

Wang et al.  [23] proposed the manipulation of CT datasets and their transfer function in VR. Their system allows tweaking various rendering parameters using hand gestures, but it does not offer any analysis options. Tadeja et al. [24] use VR for creating an aerospace design environment for digital twinning. Their tool allows for loading CAD geometries of aircraft components and respective exploration of their performance parameters through 3D scatterplots. ImNDT by Gall et al. [16] offers material experts an immersive workspace for the analysis of multi-dimensional data from NDT. The authors developed visual metaphors and interaction techniques for their VR system, which can only be applied to secondary data and do not allow a visualization of voxel data. Like in any other VR system, users are also here isolated from the real world. Gall et al. [25] extended the ImNDT system to allow for Cross-Virtuality-Analysis (XVA). They added an AR view mode for the VR user to enable collaboration with users of a large touch screen. This study found that XVA as form of analysis is beneficial, but suffers in the presented application from the poor resolution of the used VR cameras.

Schickert et al. [26] demonstrated the use of AR for material inspection. The authors developed an application for a tablets to superimpose CAD models extracted from indications of ultrasound and radar images onto a concrete specimen. The tool positions CAD models in the location of a QR code, but does not provide the ability to modify the geometry or analyze additional secondary data. The system introduced by Ferraguti et al. [27] supports workers inspecting polished surfaces by projecting measurement data directly onto the component’s surface for intuitive quality evaluation. While visual analysis is possible, the system does not facilitate the exploration of additional data that is required for our material characterization. Corsican Twin by Prouzeau et al. [28] combines VR and AR devices to create an immersive authoring tool. It allows users to annotate the virtual replica of a real-world environment in VR, and displays the charts or CAD models to AR users in the real-world environment through placed QR codes. For material characterization, the approach does not provide opportunities to explore large amounts of high-dimensional data in addition to the real samples.

The NDT data used is mainly obtained from composite materials, more precisely from fiber-reinforced polymers. The primary data is obtained by using various XCT imaging devices and consists of 3D reconstructed volume datasets, stored as a binary file (*.raw). Multidimensional data, i.e., secondary data, is derived from the generated primary data. The secondary data used in this framework was derived via the fiber characterization pipeline of Salaberger et al. [30]. In this process a Hessian matrix is used for finding the medial axis of the individual fibers after applying Gaussian blurring for noise reduction. The result is a multi-dimensional table of attributes featuring the individual characteristics for each fiber, such as its length, diameter, start/end point, and volume, among others. This information is stored in a comma-separated file (.csv).

A major advantage of our framework is that it requires no additional data preparation compared to conventional analysis systems. In accordance with the experts, we have designed the framework to accept similar primary data formats that are used in standard toolkits for materials science, such as open_iA [31] or VG Studio [32]. The primary data, stored as binary files, can also be accompanied by MetaImage header files (*.mhd) consistent with standard toolkits such as ITK [33], facilitating seamless integration into existing workflows. Similarly, secondary data maintains compatibility with the output formats of typical material characterization pipelines since it represents a text file with rows. This deliberately standardized approach to data handling ensures that analysts can move between traditional desktop-based analysis and our AR-based analysis without additional data conversion or pre-processing steps, minimizing adoption barriers and enabling our framework to serve as a complementary tool within existing analytical ecosystems.

Bild vergrößern

Given the current lack of appropriate AR systems for materials characterization, indicated in Section 2, we developed a new framework to seamlessly integrate complex primary and secondary XCT data with real-world objects. This enables researchers and analysts to inspect material properties and structures in place, offering them a virtual workspace. To facilitate further development, the framework will be released as open source on GitHub [34].

The proposed framework is based on the Unity engine (Version 2021.3) [35] for rendering immersive environments. Unity is used because of its wide variety of capabilities and compatibility with numerous immersive devices and platforms. For this reason it has been adopted for a lot of immersive applications and also serves as an effective framework for InfoVis applications, demonstrating its versatility across different areas [13, 36, 37].

Our framework’s architecture has been deliberately designed for cross-platform compatibility, enabling deployment across various immersive hardware configurations. By leveraging Unity as our development environment, the system can be adapted for deployment on more economically accessible platforms including smartphones, tablets, and lower-cost AR headsets, thus extending its reach to a wider user community. This scalability is particularly important for industrial adoption scenarios where cost considerations may influence implementation decisions. Based on the reasons mentioned in Subsection 2.1, we are using the Microsoft Hololens 2 [38] as our see-through AR HMD. The device provides advanced spatial recognition and eye-tracking, which enhances interaction with objects and virtual representations.

To enable the tracking of objects, we use the Vuforia Engine [39] which enables the robust detection and tracking of multiple targets, such as images and objects. The decision to incorporate Vuforia as a marker-based tracking system was made due to its status as an industry standard, its performance, and its cross-platform compatibility. This tracking solution is optimized for precision in industrial environments and maintains compatibility with all major AR devices, ensuring that our framework remains device-agnostic.

Our system enables the analysis of both primary and secondary derived datasets. Users may select and load XCT datasets manually using a file dialog or by physical object recognition using the cameras on the AR HMD. The latter can be performed in two different ways, which are image target recognition or shape recognition. Using image target recognition, the system detects and tracks images attached to a sample that were previously stored and linked to a specific XCT dataset. The shape recognition uses existing shape data, such as a CAD model or a surface mesh created via photogrammetry, to detect the sample and then load and display the corresponding XCT dataset. Figure 1 illustrates the detection process using a CAD model. The primary dataset is linked to the real object, which acts as a proxy for interactions. The XCT dataset is displayed using various rendering techniques, such as direct volume rendering, maximum intensity projection, or similar.

The user can load secondary data, such as lists of attributes from features of interests such as defects or other spatial objects. This information is extracted from a multi-dimensional table of characteristics stored as .csv file. In the case of fiber reinforced polymers this includes information on their length, diameter, and orientation. If spatial information, i.e., start/end points of fibers, is available in the derived data an abstracted representation can be used for further exploration. In our application, we use a model-based surface representation to depict the fibers as cylinders. This visualization is created using the fibers’ start and end points, along with their radii. This approach allows for a simplified and less cluttered visual representation of the volume, facilitating the exploration of larger datasets.

Abstract data is also highly important for a detailed analysis of materials. For records of secondary data in the .csv file, which cannot be spatially interpreted, such as stress tensor fields and statistics of characteristics, various charts as histograms or scatter plots may be generated. The visualization techniques and interaction paradigms will be explained in more detail in Subsection 3.3. Since the framework runs as a standalone application on the Hololens, it does not require a PC in an office or the factory hall. Therefore, location independent analysis can be performed anywhere at any place and any time. To extend the analysis, additional systems, such as traditional desktop-based analysis systems, can be included in the inspection (see Fig. 1). This is possible because see-through AR allows the natural environment to be retained, so the application can be used as a complementary interface to an existing traditional system.

Bild vergrößern

After identifying a sample, the corresponding XCT volume and abstract representation (depending on what is available) are displayed alongside the physical material. Users interact with spatial representations using hand gestures. By grabbing a representation with two hands and pulling the hands apart or together, a zooming metaphor is applied. Similarly, the visualizations are rotated and positioned by intuitive gestures. Furthermore, natural interaction with volumes is also possible by just interacting with the physical sample at hand. This results in a more natural, embodied inspection of the material.

The framework offers various visualization techniques to analyze the secondary data, i.e., abstract data, in addition to the spatial data. These include histograms, 3D scatterplots, bar charts, and density plots. Figure 2 shows a collection of abstract visualization techniques arranged in an office setting. These charts may be created at arbitrary places in the world and are interactive during the analysis. Interactions with the virtual representations are again carried out using hand gestures through grabbing a handle in the form of a cube placed at the lower left side of each representation. The gestures applied to the handle allow for zooming, rotating, or similar interactions using a single or both hands. Visualizations may be generated for all attributes present in the secondary dataset. The histogram visualization displays the frequency distribution of a selected attribute, allowing for a quick understanding of the most common value ranges. The 3D scatterplot reveals correlations between three attributes and provides a deeper understanding of multi-dimensional relationships. The bar chart allows experts to compare various abstract data, enhancing their understanding of differences or similarities. The distribution plot provides experts with an analysis of the shape of the distribution of an attribute for understanding the variability, central tendency, and the presence of potential outliers in the data. The visual representations provide users with a quick overview of the material’s characteristics and structure, allowing them to conduct analysis in an immersive environment.

Four domain experts participated in the study, comprising two specialists in materials science and computer tomography, and two experts in material science visualization. All participants had extensive experience with conventional material characterization workflows using industry-standard 2D tools like, VG Studio [32] and open_IA [31]. The participants stated that they had no prior experience with AR tools for material analysis.

Bild vergrößern

Bild vergrößern

As use case for this evaluation we selected a material system with polyethylene terephthalate fibers in a polypropylene matrix material. The primary dataset of this material system was obtained from a respective CT scan and secondary data was subsequently derived by the experts through the fiber characterization pipeline outlined in Subsection 3.1. The investigated sample was cut out of a standard multi-purpose test specimen manufactured by injection molding. The CT dataset yielded a size of \(250 \times 250 \times 300\) voxels and contains 214 fibers, for each of which 20 distinct characteristics were computed. The validation of the data is not within the scope of our study. As the material is very thin and features a very homogeneous texture, two image targets were used for tracking. The image target on the front of the sample is visible in Fig. 3. As all study participants possess a background in analyzing fiber reinforced polymers, an exploration task was defined to ensure the expert’s workflow was in no way constrained. This enabled us to assess how the experts utilize our tool and identify any issues with it.

For our study we used the Microsoft Hololens 2 see-through AR HMD as immersive device to run the application. At the beginning of the study, participants received instructions on the general controls and displays available in the application. They were able to follow this tutorial live alongside the instructor on a 2D screen. A cognitive walk-through methodology was employed to assess both interaction design and analytical workflow efficacy. Participants were given the freedom to use the application while sitting or standing as well as to ask about controls. To gather qualitative insights, they were instructed to use the ’Thinking Aloud’ technique during the testing phase, expressing their planned actions and any conclusions they drew.

The analysis starts with fitting and setting up the HMD. Then, the participants pick up the prepared material sample from the table to inspect it. After the detection phase of the sample, the relevant data stored on the Hololens is loaded automatically. The user is presented with the volume rendering of the XCT dataset and its model-based surface representation, both hovering over the sample (see Fig. 3). The virtual representations are manipulated in space using the physical sample. Additionally, pre-selected data visualizations for the secondary data are displayed next to the sample in the environment. In our use case, these are two histograms for different fiber lengths and a three-dimensional scatterplot encoding various spatial dimensions, which can be seen in Fig. 4. The first histogram visualizes the distribution of straight fiber lengths (the distance between the start and end points of the fiber), while the second histogram shows the distribution of actual lengths considering the curvature of each fiber. This approach enables the identification of clusters related to specific fiber lengths and curvature. The scatter plot shows the correlation of the fibers based on their diameter, area and length. This visualization enables users to draw conclusions regarding whether the fibers tend to elongate or thicken more, affecting their surface area. The representations of abstract data in the environment can be freely moved and enlarged by experts to adapt to their current investigation phase or question.

During the exploration phase, which on average lasted 25-minutes, experts were able to inspect materials and test the application’s functionality. Valuable feedback on the benefits of an immersive exploration of material datasets was provided by subsequent discussions with the experts. This feedback yielded that the presented framework seems to be promising for various material characterization tasks, with a significant advantage being the ability to interact directly with the material sample. The synchronization of translations and rotations between the physical samples and virtual representations creates a natural interaction, eliminating the need for traditional mouse and keyboard operations. Additionally, as the data is automatically loaded when the user starts interacting with the real sample, no further input is required, and the analysis is intuitively initiated. Some participants with prior experience in VR systems mentioned that the AR device is much lighter and more comfortable to use for longer data exploration. Additionally, it requires no cables and does not isolate them from their office environment and colleagues.

The experts however also identified some disadvantages: The analysis was challenging in bright environments due to low color fidelity, rendering the distinction of some representations from bright backgrounds difficult. Further comments referred to current hardware limitations of AR devices. Arguably, the field of view (FoV) is rather small, which becomes particularly obvious when enlarging volumetric data. In this case it is impossible to see the whole object within the available FoV. This requires more head movement, which can lead to fatigue. Finally, some experts expressed interest in analyzing larger primary datasets exceeding 500 megabytes. The use of the Hololens 2 in standalone mode with active object tracking and visible rendering leads to performance bottlenecks in this case and an interactive analysis without delay or unstable tracking can no longer be guaranteed.

Experts finally suggest that the system has significant potential for analyzing samples in combination with conventional analysis systems. Incorporating additional visualization techniques can enable more detailed analysis by highlighting defects in volume representations or providing insights for abstract 4D data.