The use of 3D digitizing tools is becoming the base for subject-specific products, such as the orthopaedic production process of orthoses and prostheses. This paper aims at comparing the metrological behaviour of low-cost devices (Kinect 1 and 2 by Microsoft, Structure Sensor by Occipital) and high-resolution active sensors (O&P Scan by Rodin4D, NextEngine Ultra HD, Konica Minolta Vivid 9i, GOM ATOS II 400 and Artec Leo) for the survey of human body parts. A calibrated flat plane and a test-field composed of eight calibrated spheres of different radii and placed at different heights were used to evaluate the standard quality parameters (flatness, probing errors in form and size and the standard deviation) for each device as recommended by the VDI/VDE 2634 guidelines. Subsequently, three different parts of a mannequin were surveyed as samples of human body parts. The results demonstrated the higher accuracy of fixed devices with respect to handheld ones, among which Artec Leo and Structure Sensor provided a satisfying level of accuracy for the orthopaedic application. Moreover, the handheld devices enabled performing a fast reconstruction of the mannequin parts in about 20 s, which is acceptable for a person that has to remain as still as possible. For this reason, the Structure Sensor was further tested with five motion approaches which identified that smooth motion provides the lowest deviation and higher reliability. The work demonstrated the appropriateness of handheld devices for the orthopaedic application requirements in terms of speed, accuracy and costs.
Hinweise
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
1 Introduction
The main goal of the present work is to characterize the metrological behaviour of a set of 3D devices, from low-cost to high-level ones, considering their application in the production process of orthopaedic centres. The use of Computer-Aided Design and Manufacturing (CAD-CAM) tools in orthotics and prosthetics (O&P) production increased exponentially in the last 30 years. Moreover, in the recent years, the advances in the additive manufacturing technology revolutionized the production processes of custom-made devices. With a particular focus on orthopaedics, the manufacturing of orthoses and prostheses is moving from the traditional manual process to a more digital one [1].
As briefly summarized in Fig. 1, the starting point for such design and manufacturing process is the acquisition of the 3D external geometry of the anatomical region of interest. For this reason, it is important to characterize the possible errors of the initial acquired 3D models, which are then passed to CAD systems for designing the orthoses and prostheses and finally sent to the 3D printers or CNC machines, by means of CAM modules or proper file creation (e.g. g-code). After the fabrication of the orthopaedic device, it is usually finished with small adjustments (e.g. smoothing the edges or adding soft-foam pads) and then tested with patient. Choosing a proper device for 3D acquisition of patient’s anatomy will indeed limit the error propagation of the whole design and production process of orthoses and prostheses.
Fig. 1
Orthopaedic production process flowchart (O&P stands for Orthotics and Prosthetics)
×
Anzeige
1.1 Overview
The first type of device for 3D survey used and tested in orthopaedic centres was the triangulation laser scanner; however, the cost of the device and the time needed for the acquisition made this technique not appropriate for the application [2]. The use of handheld devices for 3D survey in the field of O&P became common during the past few years.
Since 2010, when Microsoft Kinect was launched on the market, new low-cost devices, more affordable thanks to cheaper technologies, were periodically made available. The first application of these devices was interactive gaming experience, but their technology paved the way to different applications, among which the 3D survey in the field of O&P. In the wake of the Microsoft device, other triangulation-based low-cost devices (budget under 500 €) were produced, such as Asus Xtion, PrimeSense and Structure Sensor (Fig. 2). In 2013, Microsoft presented a new version of the Kinect (Kinect 2 for Xbox One) implemented with a Time of Flight (TOF) technology. Finally, other triangulation devices based on white or blue light were presented: the most representative are the Artec Eva, Spider and Leo, the Mantis Vision, the Creaform HandySCAN and Go!SCAN 3D.
Fig. 2
Low-cost devices used also for reverse engineering. Microsoft Kinect V1 and V2 in the top row, Structure Sensor, Prime Sense and Asus Xtion, in the lower row, from left to right
×
In medical applications, the 3D scanning of a human body involved mainly highly rated laser scanners for acquiring the parts of interest, in order to improve diagnosis or to facilitate the creation of 3D printed products [3‐5]. However, the medical community recently started to show interest in the low-cost devices because of their ease of use and the time saving during the survey. Their use can also be easily transferred from highly specialized institutes to small satellite clinical centres spread over the territory. The interest of producers of orthopaedic CAD-CAM solutions led to the adaptation of these kind of devices to the biomedical applications, creating both software and new proprietary hardware devices. Examples are the Rodin4D app that allows using directly the Structure Sensor with iOS devices, or the BioSculptor and the Vorum systems, composed of different types of 3D sensors and dedicated proprietary software. These systems allow to save and modify the models and to send them directly to a 3D printer or a computer numerical control (CNC) machine for the creation of a physical cast model for orthoses or prostheses production, passing through their built-in CAM modules.
In order to satisfy other fields interested in full-body 3D scanning, for example, the animation movies and fitness-related apps, new devices were developed. The simplest ones work either moving a common 3D scanner all around the person or adopting a combined use of multiple devices that acquire the body from different points of view. Examples are the Artec Shapify Booth or Vitronic VITUS 3D body Scanner. Other proposed systems are composed of a set of fixed cameras that work with the photogrammetric technique, such as in the case of Staramba 3D INSTAGRAPH. The advantages in O&P applications are the higher speed of acquisition (under 15 s and down to 1 s in some cases), but the limits of these systems for orthopaedic centres are both the high costs, starting from about 10 K €, and the large dimensions requiring a dedicated room [6]. Moreover, if the focus of the orthotic product were limited to a single limb or even a foot, it would be probably necessary to have an additional device dedicated to small parts. For these reasons, the present work does not consider the full-body systems and focuses only on the general-purpose devices.
Anzeige
The two Kinects were created as steady units for motion capture, considering the degrees of freedom (DOF) of the objects in front of them. Until now, almost all the research in 3D modelling with low-cost devices regarded the Kinect, both version 1 and 2. The first Kinect was released in 2010 starting from the Project Natal [7]. From proprietary device, the Kinect was soon opened to different aims, especially the use of the device as a moving unit for the acquisition of a steady environment and the collection of its geometric information as 3D data. The Kinect Fusion was the first project analysing the possibility to calculate the 6 DOF of a rigid object with a handheld Kinect, allowing also aligning the single frames for the creation of a 3D model [8]. Several publications dealt with the possibility of using this device as a 3D acquisition instrument, analysing also its calibration to evaluate its potential as a low-cost 3D instrument [9‐11]. The Kinect sensors, both version 1 and 2, were used for several 3D applications in cultural heritage [12], for robotics [13] and for human body scanning [14]. These devices were used for tracking surgery tools [15], for rehabilitation [16] and for improving the design of foot orthoses and leg prosthesis [17, 18].
On the other hand, the Structure Sensor was recently developed for acquiring 3D data of the environment simply connecting it to an iOS mobile device. No available researches were found regarding the use of the Structure Sensor in orthopaedic applications.
General problems related to motion and deformation of the acquired data in orthopaedics were analysed by [19], proposing and testing a deformable alignment algorithm both on static and real-time acquisition with structured light devices.
1.2 Metrology and accuracy analysis
The Joint Committee for Guides in Metrology (JCGM) stated that metrological traceability is “the property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the measurement uncertainty. Metrological traceability of a measurement result does not ensure that the measurement uncertainty is adequate for a given purpose or that there is an absence of mistakes” [20].
Two different standards are available to evaluate the functionality of a range device, the acceptance test and the reverification test. The first one is done comparing the results of the 3D data with calibrated artefacts of known shape and size and is accepted as long as the error in the data is below or equal to the manufacturer’s stated limits. The reverification test is the repetition of the first test done during defined periods to analyse the trend over time.
The accuracy of a measurement is “closeness of agreement between a measured quantity value and a true quantity value of a measurand”, while the precision is “closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar objects under specified conditions” [20].
Several previous works have evaluated the geometric verification of laser scanners with spherical test objects. González-Jorge et al. [21] presented a new standard artefact with five Delrin spheres placed on an aluminium block at the same distance and seven cubes of different dimensions. This work intended to upgrade the quality control of the instruments adopted by the users, and it focused on the evaluation test of the terrestrial laser scanner Riegl LMS-Z390i. The procedure used only one point cloud acquisition, and a MATLAB® algorithm was applied to calculate accuracy (the distance between the centres of the spheres), precision (the evaluation of the measurement of the top face of the largest cube) and resolution along X, Y and Z axes (the error in the measurement of the cubes). Similarly, Franaszek et al. [22] analysed the directional and the orthogonal error functions surveying four different spheres, made of different materials and of different sizes (radius dimensions: RA = 101.6 mm, RB = 76.2 mm, RC = 76.2 mm and RD = 50.8 mm), with three different instruments at a distance varying from 5 to 100 m. Unfortunately, the authors did not indicate the devices they used, only the specifications.
Guidi et al. [7] presented an easy characterization process for reliable comparison of different range sensors, in particular the ATOS II (GOM), the NextEngine Ultra HD (NextEngine Inc.), the SG100 and the SG1000 (ShapeGrabber Inc.), the Vivid910, the V9i and the Range7 (Minolta). The authors used test objects of known shape and size with an accuracy calculated modelling each object in a CAD environment, manufacturing them with a computer numerical control (CNC) machine tools and measuring each physical prototype with a coordinate measuring machine (CMM) with an accuracy below 4 μm in the xyz directions. Uncertainty estimation was calculated through the standard deviation of each range map acquired by the different scanners with respect to the ideal geometries, while the resolution was estimated differently along the optical axis of each range device (z) and along the other two directions (xy).
Kersten et al. [23, 24] calculated the geometrical accuracy of different medium and low-cost handheld devices. The reference data were acquired by measurements with two structured light projection systems (AICON smartSCAN and GOM ATOS I 2M). As test objects, the authors used a planar granite slab (size 300× 300 mm2), a cross-shaped body with steel spheres (max. distance 450 mm of five spheres with a diameter of 65 mm), a bust of Einstein from gypsum (height of 160 mm), a wheel hub from cast irons with the dimensions 232 × 120 × 232 mm3 and a so-called Testy. The reflecting or shining surfaces were sprayed with lime spray.
1.3 Error estimation guidelines
The measurement uncertainty of each device is affected by systematic errors associated to the concept of accuracy and unavoidable random errors associated to the concept of precision, for example, the electronic noise. The latter can be only statistically characterized for making the end user aware of the measurement system intrinsic limitations [7]. The global error can be evaluated by acquiring a certified test object and measuring the deviation of the model from the ideal one. The error can also be divided in spatial or temporal, which is observable across multiple consecutive frames [25].
When such a theoretical entity has to be evaluated for an actual instrument, including a 3D sensor, the value has to be experimentally estimated from the instrument output. For this reason, the International Vocabulary of Metrology (VIM) also defines trueness as “closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value”. Trueness is a more practical parameter that can be numerically estimated as the difference between a 3D value assumed as true (because measured with a method far more accurate), and the average of a sufficiently large number of samples acquired through the range device to be characterized. Such parameter refers therefore to the systematic component of the measurement error with respect to the real data and can be minimized through an appropriate sensor calibration.
For 3D sensors, accuracy might be evaluated both for the axial direction (z) and for the lateral one (on the x-y plane). In general, accuracy on depth is the most important and varies from few hundredths to few tenths of millimetres for triangulation-based sensors and frequency modulation–continuous wave (FM-CW) laser scanners; it is in the order of 1–2 mm for CW laser scanners and in the order of 2–20 mm for pulsed wave (PW) laser scanners. A practical value for estimating precision is to calculate the parameter defined by VIM as uncertainty. The uncertainty of a measurement is defined as a standard deviation of the data. It is a “non-negative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used. The measurement uncertainty is associated with a stated quantity value attributed to the measurand. A modification of this value results in a modification of the associated uncertainty” [20].
As accuracy is influenced by systematic errors, precision is mostly influenced by random errors, leading to a certain level of unpredictability of the measured value, due to thermal noise in the sensor’s detector, or, in case of laser-based devices, by the typical laser speckle effect.
The ISO/CD 10360-8, published by the “ISO Technical Committee 213 working group 10 Geometrical Product Specifications (GPS)—Acceptance and re-verification tests for coordinate measuring machines (CMM) with optical distance sensors”, is associated with short-range measurements. The tests are performed on a single point cloud (part 2) or on a multi-view registered point cloud (part 3) that can be also filtered and pre-processed. This standard describes a method of measuring reference objects as spheres and a plane to define the basic features of the optical system to be analysed. The test objects have to be calibrated, made of ceramic, opaque metal or of a type of optically cooperative material. Generally, two different tests have to be performed as requirement: the acceptance test, which refers to the acquisition of a calibrated object and the comparison with a reference one (usually a CAD 3D model), and the re-verification test, which involves the repetition of the first test done after some acquisition of the device.
Similarly, the standard for short-range 3D imaging is the German standard VDI/VDE 2634 derived from the VDI/VDE 2617 standard for coordinate measuring machines (CMM) [26]. The standard 2634 is related to mechanical 3D measurements and to short-range measurements with non-contact triangulation (part 1) or pattern projection (part 2) 3D systems. VDI/VDE 2634 is a guideline oriented to close-range devices with relatively small measurement volumes, which is quite essential in defining the standard artefacts, considering only three simple objects like plane, sphere and ball-bar, and in proposing very simple checks over them. Since the very early developments of 3D imaging devices, several additional objects were proposed, for possibly exploring not only an absolute error with respect to a nominal value.
1.4 Application requirements and goal
With the idea of comparing the metrological behaviour of devices for the application in orthopaedic centres (e.g. for producing orthoses and prostheses with assistive or rehabilitative purposes), we have to consider a set of constraints for the choice of the appropriate device.
Low cost: the devices should be used also in small orthopaedic centres running on low budget, i.e. below 1000 €. Only very large centres might have enough production volumes to justify an investment on more expensive devices.
Accuracy within 2 mm: the orthopaedic technicians defined that the maximum device error should be comparable with the error related to the creation of a negative cast with plaster of Paris, they quantified in about 2 mm. The value also considers the skin compressibility that can be detected on thinner layers, such as the iliac crest for scoliosis braces.
Mesh resolution about 5 mm: the spatial resolution on the skin model was defined to be fine enough to identify shape variations without detecting particular details, such as scars or veins, by the orthopaedic technicians.
Time to scan within 30 s: this limit is related to the fact that the patient has to stand and/or stay as still as possible while the technician goes around him/her to acquire the shape of the region of interest. For example, considering the case of a wrist orthosis, the patient should not move the hand in respect to the forearm and at the same time the fingers should not change the relative positions. Moreover, the physicians and the orthopaedic technicians often apply a pre-correction to the patient’s body to reduce the required modelling changes.
Scan area limited to 2 × 2 m2: typically, the 3D scanning has to be performed in the visiting room that are below 3 × 3 m2 and contain other equipment, such as desk, chairs, wardrobe and examination table.
Limited illumination problems: the device should not be too sensitive to light variation because the orthopaedic centres might be lit by the Sun during the day and/or only by artificial lamps, depending on room placement and weather conditions.
2 Methods and tools
The aim of this work is to evaluate the metrological accuracy of devices currently used for medical purposes in orthopaedic centres (i.e. the Structure Sensor by Occipital and the O&P Scan by Rodin4D) and of low-cost sensors that may be used for human body surface scan (i.e. Microsoft Kinect 1 and 2, NextEngine HD). The comparison of the results was done with professional laser scanners, the Konica Minolta Vivid 9i, the GOM ATOS II 400 and the Artec Leo. The work has been organized in 3 subsequent phases, as represented in Fig. 3.
Fig. 3
Logical sequence of the methods and tools of the present work. Firstly, tests were performed with standard objects to identify the common quality parameters (flatness, probing and distance errors), secondly the test on organic shapes related to the orthopaedic field and finally the test on 5 different motion approaches
×
The starting point of the research was to evaluate the systematic error of each device with the use of a calibrated plane made of float glass and another test-field composed by eight calibrated spheres assembled with custom 3D printed supports of different heights, placed on a base of 300 mm × 400 mm. The chrome-steel (AISI 5210) spheres had diameters ranging from 19.05 to 44.45 mm. Due to their mirror finish, the spheres were sprayed with a matt white powder. The plane and the spheres test-field were used in order to highlight which, among the professional devices, was the best to be used as a reference, similarly to what was demonstrated by [24, 27]. The volume size of the reference body with spheres corresponded to the measuring volumes of most of the examined 3D devices.
The second step consisted in comparing the results with the chosen reference device when surveying three different test objects chosen to replicate the parts of the body of a patient, considering the different level of detail and dimensions: a polystyrene hand model, a chest and a thigh from a mannequin. The only scanner that was not tested on the spheres is the O&P Scan because of issues related to the metallic spheres in terms of alignment with the magnetic field.
The third and last comparison regarded 5 motion approaches in order to acquire the chest model. In this phase, the device that was tested is the Structure Sensor in free handheld mode, mounted on an iOS mobile device as currently used in many orthopaedic centres.
From the first round of analysis, we could identify the best device in terms of absolute metrological result. The best device of phase one was then used as reference in the second round of analyses on the mannequin parts, chosen for analysing the typical organic shapes of orthopaedic applications, with variable dimensions and detail level. Finally, due to differences in fixed and handheld acquisitions with both Artec Leo and Structure Sensor, we decided to run a last analysis on motion approaches, which was helpful not only to identify the best movement but also to provide a practical guideline to technicians.
In the present work, the attention was focused on the spatial error, neglecting the temporal error. This choice derives from Guidi et al. [7], which already stated that, in the practical application, a spatial characterization allows obtaining more statistically significant results. Moreover, we noticed that the temporal error should be mitigated by the time averaging that software as Skanect and Artec Studio perform during the fusion of the different frames. In addition, we neglected the effects of temperature in time that were proved to influence the performances of these devices over a time span of several tens of minutes [28]. In our case, the time acquisition was short and limited to a maximum of 30 s for the low-cost devices.
2.1 Devices
Three low-cost general-purpose devices were evaluated in this work: the Kinect 1 and the Kinect 2 by Microsoft and the Structure Sensor by Occipital. In addition, the data regarding the O&P Scan by Rodin4D, deriving from our previous work [29] of only the plane and the mannequin parts, were included for comparison. Medium- and high-level scanners (Konica Minolta Vivid 9i, GOM ATOS II 400, ARTEC LEO and NextEngine Ultra HD 3D scanner) were used for selecting which to take as the reference for comparing results on the mannequin parts, based on the results on the standard objects. Moreover, for the comparison of the plane and the spheres, also the manual CMM device MicroScribe MLX was tested as a tentative ground truth with respect to optical sensors. The main characteristics of the devices are summarized in Table 1.
Table 1
Specification of the tested devices summarizes the main characteristics of the tested devices in terms of technology, acquisition range, declared accuracy and cost
Device
Technology
Acquisition range (m)
Declared accuracy (mm)
Cost (€)
GOM ATOS II 400
B/W fringe pattern triangulation
N/A
N/A
∼ 50k
Konica Minolta Vivid 9i
Laser triangulation
0.5–2.5
0.1–0.2
∼ 80k
NextEngine Ultra HD
Laser triangulation
0.19–0.71
0.13
∼ 3k
MicroScribe MLX
Manual CMM
N/A
0.05–0.13
∼ 8k
Artec Leo
White structured light
0.35–1.2
0.1
∼ 26k
Microsoft Kinect 1
IR pattern triangulation
0.5–6
N/A
∼ 100
Occipital Structure Sensor
IR pattern triangulation
0.3–5
∼ 4
∼ 400
Microsoft Kinect 2
IR TOF
0.4–5
N/A
∼ 200
Rodin4D O&P Scan
Handheld laser triangulation
0.05–0.3
0.5
∼ 10k
The Microsoft Kinect 1 is based on the infrared (IR) structured light technology. The device can obtain a depth estimation of the scenery at 30 FPS, by using the monochrome near-infrared (NIR) camera and the NIR projector with a baseline equal to 75 mm, comparable to the human eyes distance. The device uses triangulation technique to compute the depth information, thanks to the known and fixed dot pattern projected on the scene.
The Microsoft Kinect 2 is based on a system originally patented by Canesta [30]. It involves the modulation of an IR light source with a square wave and a flash camera to determine the distance from the object, by using the TOF technique.
The Occipital Structure Sensor is an evolution of Kinect 1, adopting the same IR structured light principle, but embedding a sensor with 4 times the previous resolution. It was created to be coupled with an iOS device in order to have a cheap and user-friendly hand device to capture 3D environments, but it can also be used connected to a computer through specific software. For acquiring the colour texture of the object, the apps use the RGB camera of the iOS device after a quick adjustment of the relative position in respect to the depth sensor.
The O&P Scan by Rodin is a handheld laser 3D digitizer, commercialized to be used in the medical field for acquiring the parts of the body of the patient to be analysed. The O&P Scan uses a magnetic field to position itself in the space and align the scans of the 3D surface. The acquisition is based on the triangulation of a fixed laser line with respect to the camera.
The NextEngine Ultra HD is a 3D laser scanner of medium level, based on the multistripe laser triangulation (MLT) technology. The device is equipped with two 5.0-MP CMOS RGB array sensors and a twin array of 4 laser lines, which tilt in respect to the camera. Moreover, it has a white-light LED illuminator for acquiring the texture.
The Konica Minolta Vivid 9i is also based on the principle of laser triangulation, with a CCD 640 × 480-pixels sensor. Also in this case, the device is usually fixed on a tripod and the single laser line tilts in respect to the camera. The higher quality of the system is demonstrated both by the quality of the three interchangeable lenses (wide, middle and tele) and by the possibility to finely control the laser intensity, according to the object to be acquired.
The GOM ATOS II 400 is a pattern projection range sensor, based on the triangulation principle. It is equipped with one pattern projector and two cameras with fixed focal distance (f = 17 mm) observing the object to be measured from different points of view in order to reduce the effects of occlusions. The system works with the triangulation method by projecting a set of black and white fringes that are perpendicular to the line connecting the cameras. The pattern is then shifted multiple times in order to obtain a denser point cloud.
The newest device tested in this research is the handheld scanner Artec Leo that works with the white structured light projection and it is the evolution of the Artec Eva, both in terms of declared acquisition results and for its stand-alone capability. It is completely independent from a computer when acquiring, and it allows analysing also a preview of the 3D model on its small screen. Its capability of acquiring up to 80 FPS when streaming to a pc and up to 22 FPS with real-time fusion is combined with the high sensor resolution, which means that it is able to acquire up to 3 million of points per second.
The MicroScribe MLX is a manual CMM, which works as a passive robot arm that registers the position of the point touched with the small spherical probe (3 mm of diameter) mounted on the end effector. Its declared accuracy, based on the 100-point ANSI sphere test, is equal to 0.3 mm.
2.2 Error parameters
In order to analyse the performance of the different devices, the following parameters were selected and computed, according to the VDI/VDE guidelines.
Flatness measurement error (F): the range of the distances of the measurement points from the best-fit plane calculated according to the least square method
Probing error in form (PF): the range of the radial distance between the measured points and a best-fit sphere determined according to the least-squared method with free radius. Equivalent to the difference between the maximum and the minimum distances of the point-cloud from the best-fit sphere centre
Probing error in size (PS): the difference between the measured diameter and the reference diameter of the calibrated sphere
Sphere-spacing distance error (SD): determined from the difference of measured and calibrated values of the distance between the centres of two spheres
The typical flatness error considers the maximum and the minimum point distances from the ideal plane. This would also consider possible outliers that could be easily removed when reconstructing the 3D object with filtering. For this reason, we decided to present the standard deviation of this type of error, as it is more statistically representative of this measurement error.
Regarding the spheres comparison, we evaluated PF and PS for each of the eight spheres of the test-field and SD for all the 28 combination of distances. In the results, we report the average, the minimum and the maximum for each of these parameters.
In addition, when analysing organic shapes, we decided to consider the global error. Similarly to the sphere probing errors, the cloud to mesh distance of the reconstructed point cloud in respect to the reference model was thus measured. In particular, it is possible to fit the distances distribution with a Gaussian distribution, the mean of which gives an idea of the scaling error, while the standard deviation represents the accuracy of the reconstruction.
2.3 Test objects
Five test objects were used to evaluate the different devices: a calibrated flat plane, used to calculate the flatness error; a test-field with 8 spheres to test the parameters PF, PS and SD recommended by the guidelines VDI/VDE; and three anatomical parts of a mannequin to simulate the parts of a human body, chosen to investigate the global uncertainty on different dimensions and detail levels—a hand, a thigh and a chest. The characteristics of the different test objects are summarized in Table 2.
Table 2
Objects used in this work, with material, dimensions and colour specifications (* range of diameters)
Test object
Material
Dimensions (mm)
Colour
Calibrated plane
Glass (opaque)
700 × 528 × 28
White
Spheres
Metal
19.05:44.45*
White
Hand
Polystyrene
200 × 95 × 50 approx.
White
Thigh
Polyethylene
420 × 130 × 120 approx.
Skin tone
Chest
Polyethylene
700 × 500 × 230 approx.
Skin tone
The reference plane, made with float glass, is 700-mm wide and 528-mm high. The particular manufacturing process of this glass allowed having a plane with a peak deviation from the theoretical plane in the order of few micrometres, suitable for testing the devices chosen for this research, characterized by measurement uncertainties in the range of millimetres. Due to the transparency of the material, not compliant with an active range sensing device, the surface was painted matt white, with the process used in the car industry. In this way, the painting was uniformly distributed without distorting the geometry of the plane. The flatness error was analysed considering the standard deviation of the acquired point clouds in respect to the ideal CAD plane, which means that the point cloud is fitted with an ideal plane and the distances of all the points from this plane are calculated.
The sphere test object (Fig. 4) had a base of 300 mm × 400 mm onto which 8 spheres were assembled with custom 3D printed supports of different heights. The sphere centres had a distance from the base plane varying between 45 and 205 mm. The positions were chosen to test the acquisition volume with just one scan, in opposition to the use of multiple shots of a two-ball bar located in different configurations [31]. The chrome steel (AISI 5210) spheres had diameters ranging from 19.05 to 44.45 mm. Due to their reflectance, the spheres were sprayed with a matt white powder.
Fig. 4
Sphere test object containing 8 spheres of different dimensions at different heights from the base plane, varying between 45 and 205 mm
×
The organic shapes of the mannequin parts and their dimensions (Table 2) were chosen because they were representative of human body segments that are usually acquired in the orthotics and prosthetics (O&P) applications. Despite the mannequin of the laboratory had also the arms with hands (Fig. 5, skin-toned one in the middle), another white polystyrene hand (Fig. 5, white one on the right) was preferred because the geometrical detail level on the backside and on the nails was not only more realistic but also more challenging for testing the scanners on small details. Compared to human skin (Fig. 5, real hand on the left side), the tests objects have a more uniform colour and are smoother. The mannequin chest and thigh are also slightly more reflective, as visible on the hand in the middle of Fig. 5, but still less than naturally oily or sweated skin.
Fig. 5
Tested white hand model on the right, compared to a real hand on the left and to the one of the mannequin in the middle
×
2.4 Software and tests for best settings
The acquisition with Vivid 9i, the ATOS II 400, the NextEngine Ultra HD, the CMM MicroScribe MLX and the Rodin4D O&P Scan were performed using the commercial software of the respective manufacturers. The Structure Sensor and the Kinect 1 were connected to the pc with USB cable, and the Skanect software was used for the acquisitions. The Kinect 2 was connected to the pc by USB cable too but used the Kinect Fusion Explorer available in the SDK Browser v2.0.
The Artec Leo and the handheld mode of the Structure Sensor, when mounted on the iOS device, represent a particular case. In these two cases, the devices were completely independent from the computer. The first was developed to be completely stand-alone, the frames are roughly aligned directly on the device itself, but a further fine registration and more processing for the fusion of the point clouds are performed on the computer with the proprietary software Artec Studio. The second uses an app that can be freely downloaded on iOS devices and that creates directly the final model with a lower resolution than the Skanect software on computer, due to the device limited computational power.
Some preliminary tests were performed to select the best settings for each software. The software Skanect always subsamples the large amount of points, deriving either from the Structure Sensor or from the Kinect 1, with fixed spatial resolution independent from the sensor’s resolution, but depending on the size of the bounding box and on the Fusion Fidelity level. The points are sampled so that there are about 125, 250 or 500 levels for each side of the bounding box for the Fidelity set to low, medium and high, respectively. Consequently, the bounding box has to be as small as possible to contain the object to be reconstructed in order to obtain the maximum spatial resolution.
The Fusion settings for the Kinect 2 SDK works in a similar manner, but the commands are more explicit. The “volume voxels per metre” defines exactly the sampling levels (available values 128, 256 and 512). The count is more complicated when the intention is to change the bounding box dimensions, because it is defined by the ratio between the “volume voxel resolution” and the “volume voxels per metre”, as shown by the tests reported in Table 3.
Table 3
Setting combination for Kinect 2 SDK fusion
Volume voxels per metre
128
256
512
256
128
Volume voxels resolution (X = Y = Z)
128
256
512
512
512
Bounding box size [m*m*m]
1*1*1
1*1*1
1*1*1
2*2*2
4*4*4
Squared mesh edge dimension [mm]
8
4
2
4
8
For the Artec Leo, we used the recommended sequence of operation for cleaning and aligning the frames in the Artec Studio 14 Professional software. The only differences for the plane and the spheres were the removal of the extra frames (only 220 frames were preserved to maintain the comparable time limit of 10 s, having set the frame rate to 22 FPS) and the use of the geometry only (without the texture to avoid any influence) for the registrations. Moreover, the key frame ratio was set to 1, meaning that the 100% of frames was considered for the alignment, only for plane and spheres. It was set back to the default value of 0.1 (meaning only 10% of the frames are considered) for the mannequin parts due to computational limits when processing the larger number of frames.
Other tests were performed for the remaining software for Vivid 9i (Polygon Editing Tool), NextEngine (Scan Studio) and ATOS II 400 (ATOS Professional), which did not show critical problems. Particular attention was paid to the light intensity depending on the object colour, while the other parameters were left as default. Finally, the distances of the devices from the objects were set accordingly to the dimensions and the available scanning volume.
2.5 Acquisitions
Each test object was surveyed with all the instruments placed on a tripod at a fixed distance. The only exceptions were the O&P Scan, because it is a handy scan with fixed angle between laser and detector, thus used only in free hand movement, and the CMM MicroScribe, that had to be placed in the reach zone for the probe to touch the plane and the spheres. Additionally, the acquisitions with the Structure Sensor and Artec Leo used both settings of fixed on a tripod and handheld mode, when acquiring both the spheres and the mannequin parts.
The objects were all acquired emulating the application environment, that is, the orthopaedic centre room with artificial lights (neon) and trying to avoid the use of markers, with the exception of the ATOS II 400. The reason for this choice is both that patients have usually difficulties staying still in respect to external references and because applying markers to the target objects could be trivial for the complete acquisition of their geometry.
The acquisition distance and the setting parameters were selected considering the specifications for each instrument (e.g. the field of view) and the object properties (e.g. texture and colour). Table 4 summarizes the acquisition distance of the devices from each object.
Table 4
Average distances between the devices and the acquired objects
Device
Acquisition distances (mm)
Plane
Spheres
Hand
Thigh
Chest
GOM ATOS II 400
1100
900
900
900
900
NextEngine Ultra HD
720
780
635
635
700
MicroScribe MLX
Contact
Contact
N/A
N/A
N/A
Konica Minolta Vivid 9i
1310
850
870
1100
1350
Artec Leo
1270
780
540
670
650
Occipital Structure Sensor
1000
550
450
450
750
Microsoft Kinect V1
1000
720
600
750
850
Microsoft Kinect V2
900
800
750
800
800
Rodin4D O&P Scan
100
N/A
100
100
100
2.5.1 Flat plane
The calibrated flat plane was placed on a table, and all the devices were put on a tripod slightly oblique in front of it, to limit possible reflections.
The Vivid 9i mounted a middle lens, choice that depended on the dimension of the plane and on the decision of limiting the maximum distance from the object, considering always the conditions of the final application, thus discarding the tele lens. Given the fact that the surface of the plane is opaque white, the parameter of the intensity for the scan was set equal to 12.
The ATOS II 400 always requires the use of at least three markers visible from both cameras when acquiring an object. In order to avoid placing the markers on the plane, they were attached to a support right under the plane. Due to the configuration of the instrument, the volume that can be acquired is limited, and it was also hard to avoid the reflected light of the projector. However, we decided to use a single scan avoiding the possible errors related to the alignment of multiple scans.
The NextEngine Ultra HD works with the ScanStudio proprietary software. The resolution was set to HD and the target colour to neutral, which is a setting similar to the laser intensity on the Vivid 9i. Even if the scan was a single one, the device used all its four laser lines the device has, requiring about 5 min to be completed. Unfortunately, due to the limited field of view of the fixed lens, only a portion of the plane could be acquired, even if the plane was place at the maximum limit for the WIDE mode.
The Artec Leo acquired over 300 frames. We preserved a limited number of acquisition frames (i.e. 220), in order to limit the acquisition to exactly 10 s and be comparable to the time necessary for scanning with the Vivid 9i and ATOS II 400.
For both the Structure Sensor and the Microsoft Kinect 1, it was used the Skanect software, imposing a cubic bounding box of 1 m3. The acquisition was stopped after 10 s, as for the Artec Leo. This setting gave the possibility to average the measurements of the single frames (i.e. 300 thanks to the frame rate of about 30 fps), thanks to the fact that the devices were placed in a fixed position. The obtained 3D models were then exported in the highest resolution available.
A comparable setup was used for the Microsoft Kinect 2, using in this case the SDK software specifically implemented for this device. After the preliminary tests, it was concluded that the best setting was to fix the voxels per metre at 768 and the volume voxel resolution at 512, meaning that the bounding box was about 0.6 × 0.6 × 0.6 m3 and the mesh size was about 1.3 mm. Also in this case, the acquisition was stopped after 10 s.
The MicroScribe had to be placed on the table, next to the plane for its limited reach zone. It was used in manual mode, meaning that the points were acquired one by one by pressing on the pedal while moving manually the probe over the plane. The total number of points kept for the analysis was equal to 133. Since the acquisition is performed manually while the software writes the coordinates in a text file, there is no visual feedback for the points distribution. For this reason, the points are not as equally distributed on the surface as with the other devices.
The O&P Scan is a handy scan, and the acquisition of the plane was performed generating several strips to cover the entire surface of the plane, manually moving the scanner in front of the plane. The software thanks to the magnetic field transducer automatically aligned all the strips. To obtain the exportable 3D model, the software RODIN4D NEO was used, with the lowest value of smoothing to limit possible deformations.
2.5.2 Spheres
The general settings for the different devices were similar to the acquisition of the flat plane, locating the scanners perpendicular to the planar base of the test-field. The main differences were the relative positions in respect to the test object. The distances between the planar base and the centre of the sensors are reported in Table 4.
The NextEngine distance from the planar base is larger than the maximum limit for the WIDE mode because we did not want to acquire the plane but the spheres mounted on it. Thanks to the custom supports, the spheres were closer to the sensor and they could be acquired properly.
In the Skanect software, used for acquiring with the Structure Sensor and the Kinect 1, the bounding box dimension was set equal to 0.4 m per side. This allowed improving the spatial resolution of the sampled points, with respect to the plane settings.
Moreover, both the Structure Sensor and the Artec Leo were used both in fixed mode on the tripod and in handheld mode going around the test-field, in order to evaluate also their usual acquisition approach for the final application.
After testing for normality, differences in PS and PF between the fifteen different combinations of scanners and methods were tested with a one-way ANOVA analysis and with Bonferroni correction for post hoc tests. Differences with p values lower than 0.05 were considered as statistically significant.
2.5.3 Hand
Since CMM devices are good for sampling only a limited number of points for measuring few distances, but not for performing a 3D reconstruction, and due to the results obtained with the tests on the flat plane and on the spheres, the MicroScribe was neglected as possible reference in the case of the mannequin parts.
For the acquisition of the hand, the Vivid 9i was coupled with a middle lens. The parameter regarding the light intensity was set equal to 13, and 28 scans were acquired moving the object in front of the laser scanner and were then aligned and merged using Innovmetric Polyworks.
The ATOS II 400 was set similarly to the acquisitions performed for the flat plane and the spheres. Indeed, due to the fixed camera position, it was not possible to vary the distance of the bounding box. For this reason, 45 scans were acquired rotating the hand, in order to cover any region.
The NextEngine performed 12 acquisitions in a full rotation, with the hand placed vertically on a turntable. Then, 10 additional single scans were completed by manually defining the orientation of the hand on supports in order to cover as much as possible the scanned surface.
For the Leo, the Structure Sensor and the two Kinect versions, the hand was placed on a stool that was rotated in front of the devices at a fixed distance with two rounds, varying their relative height. The Structure Sensor and the Microsoft Kinect 1 were placed at a distance from the object of 450 mm and 600 mm, respectively, and the bounding box in Skanect was set equal to 0.3 × 0.3 × 0.3 m3 to maximize the point cloud density. For the Kinect 2, the best setting for the SDK was to fix the voxels per metre at 512 and the volume voxel resolution at 256, with a consequent bounding box of 0.5 m per side. In addition to the models acquired in fixed position, the handheld scanners Artec Leo and Structure Sensor were used with free hand movement. They were moved around the objects, creating almost a loop at variable distance. In general, the motion took place staying far enough to acquire the whole model but in some moments getting closer and focusing on the details (e.g. the fingers) or tilting the scanner to acquire from a different point of view. A similar approach was used also for the Structure Sensor, which was mounted on an iPad. The acquisition was computed using Captevia, the app developed by Rodin4D and setting the bounding box manually on the screen. Finally, the O&P Scan was used for all the objects by moving it in strips (mainly vertical) all around the objects at a distance of about 15 cm from the object and placing the magnetic marker right under the objects.
2.5.4 Thigh and chest
All the acquisitions were performed with settings similar to the hand ones, with some differences related to the dimensions and the detail level of the objects.
The Vivid 9i was in this case coupled with a wide lens to face the larger objects. The laser intensity parameter was set to 18 because the skin-tone colour of the object was not as light as the white hand. The acquisition consisted in 19 scans for the thigh and 25 scans for the chest, which were then imported, aligned and merged in Innovmetric Polyworks to obtain the two final 3D models.
For the ATOS II 400, the setup was the same as for the hand, but the number of scans that were performed became 43 for the thigh and 37 for the chest. The large number of scans is related to the limited bonding box per scan, the required overlap for multiple markers to be visible in the following scans and the sharp edges of the models where the mannequin had the connection with other parts (e.g. the rest of the leg or the arms for the chest).
Also for the NextEngine Ultra HD, the setup did not change. The thigh was placed on the turntable, and two full rotations with 12 scans each were performed for acquiring the lower and the upper regions, but additional 6 scans were performed for the top and bottom surfaces by manually placing the object in front of the scanner. Instead, for the chest, 27 scans were acquired separately by manually placing the object, due to its larger dimensions that did not allow performing the automatic rotation on the turntable.
For the other scanners, the objects were placed on a rotating stool. Two complete rounds were performed for the thigh and three for the chest varying the relative heights. The acquisition distances were maintained constant, as reported in Table 4.
The bounding box was set at 0.6 × 0.6 × 0.6 m3 for the acquisition of the thigh and at 0.8 × 0.8 × 0.8 m3 for the chest, for both Structure Sensor and Kinect 1. Not having the same flexibility of control using the SDK for Kinect 2, the best setting was to fix the voxels per metre at 512 and the volume voxel resolution at 384, with a consequent cubic bounding box of 0.75 m per side.
After the fixed mode, the Leo and the Structure Sensor were used in handheld mode similarly to the hand, keeping the object completely fixed, while the operator was moving the devices around the hand.
2.5.5 Handheld approach
Due to the final application concerning a fast scan of the patient, we wanted to evaluate how the technician’s movements around the target could influence the scan.
The mannequin chest was thus chosen to be acquired multiple times with the Structure Sensor mounted on an iPad, in order to evaluate the best approach to be used in the orthopaedic centres in a fast but reliable way. The tests were performed with the following movements of the operator walking around the mannequin, holding the tablet onto which the Structure Sensor was mounted.
1.
One lap with the scanner at a fixed distance and without other motions, to mime the ideal circle that was obtained by rotating the objects in front of the fixed devices
2.
Two laps with the scanner at a fixed distance and without other motions, to test possible alignment improvement in respect to a single circle
3.
One lap with the scanner at a fixed distance and moved up and down, with a sinusoidal movement
4.
One lap with the scanner at a fixed distance and pitched up and down, rotating around its horizontal axis
5.
Operator allowed to move the scanner freely, changing distances and rotations in order to acquire the whole surface
All the previous approaches were limited to about 20 s to perform the full rotation around the mannequin, except for the second case that took about 40 s for the two laps. Each approach was repeated three times to provide statistical confidence and compare repeatability.
3 Analyses and processing
The first analysis regarded the estimation of the global and the systematic errors for the different devices related to the flat plane acquisition, following the basic steps presented by Guidi et al. [7].
The second analysis considered the performance evaluation on the spheres reconstructions, evaluating the three parameters PF, PS and SD described in Section 2.3.3, similarly to [23, 27]. Finally, considering the results of both analyses, the reference device was set for the subsequent comparisons on the mannequin parts.
For the analysis of the errors with the flat plane, [7] used a Taubin smooth filter [32]. The filter is a low-pass filter that does not apply any subdivision to the mesh. In other words, the filter should remove the noise, commonly known for having high frequencies, without further modification to the points’ position.
The analyses were performed using CloudCompare, “the Open Source Project for 3D point cloud and mesh processing software”. The version used was the latest stable release 2.9.1, installed on a Windows 10 notebook with i7-6700HQ CPU, 16GB of RAM memory and NVIDIA GeForce GTX 960M (2GB GDDR5 video memory). In addition, Autodesk Meshmixer (version 3.5.474) was used for pre-processing the acquired data (i.e. manual rough cleaning of the environment), and MeshLab (version 2016.12) was used for applying the Taubin smooth filter and for scaling the models acquired with Kinect 2 with a factor of 1000, in order to change the measurement units from metres to millimetres, so that all the models were coherent with the engineering measurement units.
In all the comparisons among 3D models, the point clouds were extracted from each model in the same way using CloudCompare. The point clouds were analysed in respect to the reference mesh models and the cloud-to-mesh comparison was thus chosen. The cloud-to-cloud option was not considered appropriate because the output values were only positive distances point-point and thus the scaling effects was not detectable. The cloud-to-mesh instead solves both issues, giving the correct distance of the points from the reference mesh (that represents the ideal or the best one), with both positive values (outwards) and negative values (inwards), thus highlighting also the possible scaling effects.
3.1 Flat plane
After a rough cleaning step to remove the surrounding environment, the acquired models of the plane were imported in CloudCompare. Here, an ideal plane was created using the “Primitive factory” function available in the software. The dimensions were set equal to the real plane ones, 700 mm by 528 mm.
All the models were roughly aligned manually to define the initial configuration and then registered automatically, setting a partial overlap of 80%. This step allowed using only the points related to the flat plane discarding the ones of the border that could be misleading the convergence.
Afterwards, the models were cropped using a bounding box of 680 mm by 500 mm by 40 mm, aligned with the plane in order to keep only the points of the plane. A fine registration was performed by setting the final overlap to 100%, using up to 5,000,000 points (value beyond the maximum number of points available for any device, to include the entire point clouds) and fixing the translations on the plane, while allowing the perpendicular shift and the rotations. This last constraint allowed aligning the scanned planes to the ideal plane avoiding any lateral shift.
The models were transferred to MeshLab where the Taubin filter was applied. For this step, the 3D meshes were used. The parameters adopted were λ = 0.95, μ = −0.98 and 50 iterations. These values were set in compliance with the original work [32], which defined the following limits: λ > 0, μ < − λ and the threshold kpb > 0 (preferable between 0.01 and 0.1), where kpb = 1/λ + 1/μ = 0.0315 in our case. After the filter was applied, all the models were transferred back in CloudCompare, and the point clouds were extracted.
An exception regarded the MicroScribe, which provided only a point cloud, so, it was not possible to apply the Taubin filter. In order to analyse the possible effect of a filter, the SOR (statistical outlier removal) option available in CloudCompare was used.
Finally, all the comparisons were performed with the cloud-to-mesh option, by using the vertices of the model with respect to the ideal plane created with the primitive factory. Both original data and filtered data were used to measure the global error and the systematic component, respectively.
A particular focus regarded the latest released device, Artec Leo. The frames were processed in Artec Studio and exported in 3 different ways, after a fine registration, that we refer to as Smooth Fusion, Sharp Fusion and 220 frames. The first corresponded to the processing recommended for organic shapes, using an outlier removal filter and performing a Smooth Fusion for obtaining the final model. The second is similar to the previous, but adopting the Sharp Fusion algorithm that, according to the authors, should limit smoothing to the reconstructed surface. The last considered all the 220 frames as individual raw point clouds in *.stl format, which were then assembled together with a MATLAB script without applying any cleaning and smoothing filter nor any registration.
3.2 Spheres
The acquired models were cleaned manually in order to extract and divide the points belonging to each sphere. The point clouds from the MicroScribe were cleaned in order to preserve a consistent number of 30 points for each sphere, which satisfies the requirement of “at least 25 surface points” of the VDI/VDE guidelines.
All the point clouds were then analysed with a MATLAB script that performed a least squared fitting of a sphere with freely variable radius and centre coordinates [33]. Once each point cloud is fitted with a sphere, the script computes the error in the size approximation in respect to the calibrated dimensions of the spheres, obtaining the probing error in size (PS), while the single points are evaluated with respect to the approximated sphere to define the probing error in form (PF). Finally, the distances between the approximated spheres were computed to esteem also the sphere-spacing distance error (SD), where the reference distances regarded the device that best performed in the other parameters.
Also in this case, the analyses on the Artec Leo were repeated for the three cases of Smooth Fusion, Sharp Fusion and 220 frames, for the data acquired in both fixed and handheld mode.
After testing for normality, differences in PF, PS and SD between the fifteen different combinations of scanners and methods were tested with a one-way ANOVA analysis and with Bonferroni correction for post hoc tests. Differences with p values lower than 0.05 were considered as statistically significant.
3.3 Mannequin parts
The models of the hand were roughly cleaned removing most of the surrounding environment, each model separately. This step was also a requirement in order to import all the models in a single comparison file of CloudCompare. Since no smoothing filter had to be applied, the point clouds were immediately extracted and the following steps were computed without the mesh models. Similarly to the analysis on the flat plane, the models were roughly aligned manually and then finely registered automatically, with the constraint of 90% final overlap. Subsequently all the models, aligned in respect to the reference one, were cropped in the region of the forearm at the same location. Finally, a fine automatic registration was performed using the 100% of the final models, and the cloud-to-mesh distance was evaluated.
The analysis of the thigh was very similar to the previous one (hand), with the exception of a double cropping of the models, both in the distal and proximal regions of the thigh, after the first 80% registration. Another change in the settings of the final 100% of overlap was that the vertical translation (along the axis of the leg) was locked, so that no further shift could happen when aligning the cropped point clouds due to possible scale effects on this pseudo-conical shape.
For the chest, the steps were the same used for the hand but having a multiple cropping to all the four connections of the mannequin: on the neck (for the head), on the lower part of the abdomen (for the pelvis) and on both shoulders (for the arms). In this case, the fine registration 100% final overlap was performed with all the degrees of freedom unlocked.
3.4 Different handheld motions
A pre-processing step was required to remove the surroundings and the flying parts due to noise in the acquisitions. With the goal of evaluating the repeatability of the measurements and the variability with the 5 approaches (listed in Section 2.5.5), the acquired chest models were aligned and compared approach by approach in CloudCompare using the “cloud-to-mesh distance” option. In each group, the three models were compared to each other and the standard deviations of the six permutations were averaged. Finally, the difference of the approaches was compared, and the statistical significance was evaluated.
Moreover, a second analysis regarded the right hip side which is commonly bony for the iliac crests. This was selected as region of interest for the typical application of scoliosis braces. These orthoses have an anchorage on the pelvis for applying the corrective forces on the thorax; thus, the acquisition of this region should be as precise as possible for modelling the brace. A similar conclusion could be drawn for the more recent application of custom-made exoskeletons, which require an interface anchorage around the waist. So, a further comparison was computed also for the portion of models extracted and, similarly, the statistical significance of the difference between the approaches was evaluated.
After testing for normality, differences in standard deviation between the five different approaches and the two different areas (whole body and right hip) were investigated with a two-way (approach × area) ANOVA analysis, with Bonferroni correction for post hoc tests. Differences with p values lower than 0.05 were considered as statistically significant.
4 Results and discussion
To understand the results reported in the sections below, it is important to remember that the global error, obtained analysing the models without any filter, is composed of a systematic component, which can be highlighted applying the Taubin filter and is related to low frequency, and a random error, that has high frequency and that is subtracted from the global one with the application of the filter [7]. Moreover, the lower the standard deviation values of the cloud to mesh distances are, the lower the error is and the better the device performances are. In the comparison of the mannequin parts, the mean value of the fitted Gaussian distribution to the global errors is instead related to scaling effects, which could be corrected with a proper calibration of the devices.
4.1 Flat plane
In order to evaluate the flatness measurement error, the distance cloud-to-mesh was computed among the ideal plane and each of the acquired models, both with and without the application of the Taubin filter. The results were fitted using Gaussian distributions, and the values of mesh average dimension and standard deviation are reported in Table 5. A selection of graphical results is also depicted in Fig. 6, while the complete series of graphical results is reported in Appendix.
Table 5
Values of mesh size and standard deviation of the cloud to mesh distance for the flat plane models, before and after the application of the Taubin filter
Device
Mesh average dimension [mm]
Std Dev [mm]
Without any filter
With Taubin filter
GOM ATOS II 400
3.5
0.073
0.064
NextEngine Ultra HD
0.3
0.082
0.073
MicroScribe MLX
N/A
0.078
0.074
Konica Minolta Vivid 9i
1.3
0.331
0.284
Artec Leo—Sharp Fusion
1
0.596
0.576
Artec Leo—Smooth Fusion
1
0.577
0.563
Artec Leo—220 frames
N/A
0.747
0.689
Occipital Structure Sensor
1.9
1.767
1.66
Microsoft Kinect V1
1.9
3.187
3.142
Microsoft Kinect V2
1.56
1.025
1.014
Rodin4D O&P Scan
2
4.492
4.435
Fig. 6
Example of the flat plane comparison for Konica Minolta 9i, Artec Leo and Structure Sensor. The saturation value (Lsat) corresponding to the minimum (blue) and the maximum (red) of the coloured scale bar are reported below each scanner name
×
As expected from the manufacturers’ declared specifications, the ATOS II 400 had the lowest values of standard deviation among the devices, both without and with the application of the Taubin filter, respectively, 0.073 and 0.064 mm. The MicroScribe MLX was estimated to have better results than the other 3D devices. However, it has to be noticed that the limited number of acquired points (133) could have reduced the averaging effect, thus increasing the standard deviation values.
The Minolta laser scanner showed a behaviour in line with the previous results presented in [29]. Figure 6 shows that an improvement could be obtained with a proper calibration in order to remove the vertical stripes effect and the horizontal bowing.
For the Artec Leo, we can notice how the Smooth Fusion performs better than the Sharp Fusion on the plane, due to its geometry (Fig. 12 in Appendix). As expected, both the algorithms behave much better than the raw point cloud of all the frames merged together, thanks to their averaging effect in the sub-sampling step. It is also clear that the smoothing filter has a stronger effect on the Sharp Fusion model than on the Smooth Fusion one.
The global error results of Structure Sensor and Kinect 2 are coherent with the ones found in [7], who tested a single-frame point cloud. In the present tests, the acquisition consisted in many frames captured in 10 s, so the small differences can be related to the fusion algorithm that created a time averaging effect. This could explain why the random error on the acquisition of the plane, at more or less the same distance between the device and the object, in [7] was about half of the global error.
The Kinect 1 presented the worst results in the reconstruction of the flat plane with values over 3 mm that are almost double the ones of Structure Sensor, which is its technical evolution in terms of hardware.
The O&P Scan instead was expected to have better results than the 3 low-cost devices, but it showed its two main disadvantages: the bad behaviour of the magnetic field for determining the relative position of the device with respect to the sensor, producing the spherical warping, and the small width of the laser blade and of the field of view that determined the need of multiple strips, visually comparable to the ones left by a paint brush (see Fig. 12 in Appendix).
For all the devices, the systematic error affects more the 3D acquisition, as proved by the slight decrease of the standard deviation values after the application of the Taubin filter. The strongest effects of this filter resulted on the models processed without any smoothing option or any averaging effect. For example, Vivid 9i and ATOS II 400 present the best improvements, together with the point cloud of all frames for the Artec Leo. A reduced effect is instead visible from results of the fusion of the frames that was performed with Artec Studio or that is automatically performed for creating the model by Skanect and Kinect SDK.
Finally, analysing the values of Table 5 alone is not sufficient to establish whether the device is performing at its top level. We should instead evaluate also the error distribution over the model, as reported in Fig. 6 and in Appendix, because a better calibration could be needed. For example, the Vivid 9i shows a problem of stripes and bowing and the Structure Sensor presents a clear deformation due to the sensor construction. On the other hand, the Leo presents a random distribution of the error, which means that the calibration was correctly computed, despite the overall deviation is higher than Vivid 9i, and a lower improvement could be achieved.
4.2 Spheres
The ANOVA analysis showed significant differences for both PF and PS errors between the fifteen different combinations of tested scanners and methods (p < 0.001), which allowed to identify different groups of error range.
Post hoc analysis for PF error highlighted the presence of four distinct groups, highlighted with colours in Fig. 7.
ATOS II 400, NextEngine Ultra HD, MicroScribe, Vivid9i, and Artec Leo with Smooth and Sharp Fusion in both fixed and handheld mode had statistically significantly lower errors in comparison to all other scanners and methods.
Errors associated to Artec Leo with 220 frames (both fixed and handheld mode) were statistically significantly worse than most of the scanners and methods of the first group, but better than Structure Sensor (both fixed and handheld mode), Kinect 1 (both fixed and handheld mode) and Kinect 2.
Structure Sensor (both fixed and handheld mode), Kinect 1 in fixed mode and Kinect 2 performed statistically significantly worse than all others, excluding Kinect 1 in handheld mode, which performed statistically significantly worse than all other scanners and methods.
Fig. 7
Probing error in form (PF). Numerical values of average and standard deviation over the 8 spheres and graphical representation of averages with histogram bars and min-max with error bars. Green, yellow, orange and red coloured left squares highlight the four groups of sensors identified by the statistical analysis, with errors increasing from green to red
×
Post hoc analysis for PS error also highlighted the presence of four similar groups, with few variations (Fig. 8).
ATOS II 400, NextEngine Ultra HD, MicroScribe, Vivid9i, and Artec Leo with Smooth and Sharp Fusion (in both fixed and handheld mode) had statistically significantly lower errors in comparison to Structure Sensor in fixed mode only, Kinect 1 (both fixed and handheld mode) and Kinect 2.
Artec Leo with 220 frames (both fixed and handheld mode) and Structure Sensor in handheld mode performed statistically significantly better than Kinect 1 in fixed mode only and Kinect 2.
Structure Sensor in fixed mode, Kinect 1 (in both fixed and handheld mode) performed better than Kinect 2, which performed worse than all other scanners and methods.
Fig. 8
Probing error in size (PS). Numerical values of average and standard deviation over the 8 spheres and graphical representation of averages with histogram bars and min-max with error bars. Green, yellow, orange and red coloured left squares highlight the four groups of sensors identified by the statistical analysis, with errors increasing from green to red
×
Post hoc analysis for SD only highlighted that the Kinect 2 scanner provided statistically significantly higher errors in comparison to all other methods.
Considering the results of statistical analysis and the performance values, the best results were obtained with the ATOS II 400 also for the spheres. The other devices performed similarly in terms of results ranking when considering the fixed configuration, with the exception of the Kinect 2 that became the worst.
The free hand mode of both the Leo and the Structure Sensor provided better results than keeping the devices fixed on a support, despite the same time limit (10 s). The improvements could be related to the best alignment of the frames when a larger portion of the object is acquired. For example, such improvement is clearly visible when acquiring an object in a 360 rotation around it, minimizing the overall alignment error, as opposed to the propagation of the error when the rotation around the object is not complete.
In the graph presented in Fig. 8, the plots for Kinect 2 show the large errors in the approximation of the sphere, which can be clarified analysing the reconstructed model. The spheres did not maintain the shape but became more similar to cones, probably related to an issue in the SDK fusion.
The NextEngine, which is a medium-cost device, presents a very high level of results. The main problem of this device, though, is the amount of time required of about 5 min per single scan to be acquired and pre-processed.
The sphere-spacing distances (SD) in Fig. 9 show clearly the error related to the sphere reconstruction of the Kinect 2. Its wide variation was thus cropped for readability purpose. In addition, the Minolta results have a variation that is larger than declared in datasheet, but we could relate the issue with the calibration pattern described above and shown in Fig. 6, which could define improper positioning and relative distances of the spheres.
Fig. 9
Graphical representation of the sphere spacing distances, average in orange with error bars (min-max)
×
4.3 Hand, thigh and chest
Due to the results obtained both with the analysis of the flat plane and of the spheres, the ATOS II 400 was selected to be the reference device. For this reason, the cloud-to-mesh distance was computed between each of the acquired point clouds and the model obtained with the ATOS II 400.
The results were fitted using Gaussian distributions, and the values of standard deviation are reported in Table 6, together with the value of average mesh size. For the comparison of these models, no filter was applied. All the comparisons are represented in Fig. 10, except for the Artec Leo Smooth Fusion reconstructions because comparable to the Sharp Fusion.
Table 6
Values of mesh size and standard deviation of the cloud to mesh distance for the hand, thigh and chest models, evaluated in respect to the model acquired with GOM ATOS II 400
Device
Mesh average dimension (mm)
Std Dev (mm)
Hand
Thigh
Chest
Hand
Thigh
Chest
GOM ATOS II 400
0.5
0.5
0.5
–
–
–
NextEngine Ultra HD
0.25
0.6
0.9
0.142
0.083
0.201
Konica Minolta Vivid 9i
1.4
1.5
3
0.242
0.227
0.358
Artec Leo-Fixed-Sharp Fusion
1
1
1
0.254
0.120
0.199
Artec Leo-Fixed-Smooth Fusion
1
1
1
0.320
0.148
0.260
Artec Leo-Handheld-Sharp Fusion
1
1
1
0.226
0.102
0.214
Artec Leo-Handheld-Smooth Fusion
1
1
1
0.281
0.121
0.287
Occipital Structure Sensor-Fixed
0.6
1.2
1.6
0.624
0.380
0.586
Occipital Structure Sensor-Handheld
3
3.5
4.5
0.495
0.393
0.657
Microsoft Kinect V1
0.6
1.2
1.6
0.873
1.419
2.016
Microsoft Kinect V2
2
2
1.6
2.117
1.431
2.587
Rodin4D O&P Scan
1
2
3
1.065
1.376
3.783
Fig. 10
Results obtained by the different devices, organized by column and in groups of saturation level, acquiring the hand, the thigh and the chest of the mannequin in the rows. The saturation value (Lsat) corresponding to the minimum (blue) and the maximum (red) of the coloured scale bar are reported below each group of models
×
The NextEngine scanner was the one that best performed among all the devices, in all the acquisitions. Both NextEngine and Minolta Vivid 9i presented a similar trend for the standard deviation in respect to the analysed object. The minimum was obtained for the thigh, thanks to the simpler shape in respect to the hand, which showed more difficulties for the reconstruction of the fingers. The worst results instead appeared on the biggest object, the chest model.
Surprisingly, in respect to the previous results on standard objects and declared specifications, the Artec Leo performed better than the Minolta in most cases. In particular, the use of the Sharp Fusion always produced lower deviation than the Smooth Fusion. This effect appeared both on sharp edges that had to be preserved (i.e. fingertips) and on completely smooth objects, such as the thigh and the chest. This is not only explainable considering the loss of sharp details (first case) but also for a scaling effect applied by the smoothing filter (second case), as observed also with the spheres.
The Kinect 1 obtained good results in terms of standard deviation with the 3D objects, better than the one obtained with the flat plane. Comparing the objects, the value of deviation on the hand seems to be better than the ones with the thigh and the chest. However, analysing the models in Fig. 10, we can notice that the hand is particularly good only on the wrist, palm and back, but the reconstruction of the fingers is very poor. For this reason, we can state that a single parameter is not sufficient to describe a device behaviour on the entire object.
The Structure Sensor mounted on a tripod demonstrated to have a stable deviation in all the models with respect to the references. When used as a handheld device by the orthopaedic technician, mounted on an iPad with the Rodin4D app, the device performed again quite well, also with respect to the Rodin4D O&P scan, which has higher cost. The lower spatial resolution is clearly caused by the settings of the iOS app, due to the computational power limits of the tablet in respect to a normal computer. Overall, comparing the results with the requirements set for the selected application, the Structure Sensor in handheld mode is still compatible with the goal of digitizing human parts in order to design and produce patient-specific products, such as an orthosis or a prosthesis.
For both the Kinect 1 and the Structure Sensor, the values of the standard deviation showed that the averaging, due to the high number of frames acquired during the scans, and the closed-loop paths followed around the objects produced better results.
The Kinect 2 showed sensibly higher values of standard deviation with all the 3D objects, in respect to the ones obtained with the flat plane. The main reason was already clear during the real-time view of the acquisition and depended on the very poor alignment achieved by the SDK software. The models are visibly deformed; for example, the hand had connected fingers and the thigh was warped. In addition, the acquisitions themselves required slow and smooth movements not to lose the tracking. Moreover, trying to use the maximum resolution, it presented issues in the capturing frequency that decreased from the normal 30 FPS down to 2 FPS, making impossible to track the object with the same speed of the other devices.
The O&P Scan demonstrated, similarly to the Structure Sensor, that the alignment of the strips on 3D objects performs better than on the flat plane for the closed-loop fusion. However, the device showed an increase in the error with the increase of the dimensions of the objects. This could be related to alignment problems of the magnetic field technology and its unprecise tracking in the working environment.
In conclusion, analysing the numerical results in Table 6, there is a clear distinction with three levels of devices, as organized in Fig. 10 for the saturation values, respectively 1, 3 and 5–6 mm.
4.4 Application-related choice of the device
In general, the hardest model to be acquired was the hand model due to occlusions between the fingers, which required particular attention to create a good reference model with GOM ATOS II. The laboratory devices, working only on a fixed support (GOM, Minolta and NextEngine), provided the best results in terms of spatial resolution and accuracy of the measurements. The problem of these devices is the type of technology that requires multiple scans from fixed viewpoints. Each of them takes a long time (from 10 s of Minolta Vivid 9i up to 5 min of NextEngine Ultra HD) to be completed.
Moreover, it is not easy to move the devices around the patient’s body. Thus, considering the example of orthopaedic application of scoliosis braces, the patient would have to move in front of the device, maybe standing on a large turntable, with the possible effect of changing also the geometry during the acquisition, due to both joints and soft tissues movements (e.g. arms position).
From the technical point of view, the best device for the orthopaedic application (e.g. designing patient-specific products for assistance and rehabilitation), among the others tested in this work, resulted to be the Artec Leo, not only for the low deviation from the reference models but also for the possibility to quickly move around the objects without any wire and without the need of an additional tracking system. However, the high cost limits its applicability to a wide number of orthopaedic centres.
Similarly to the Artec Leo, the Structure Sensor can be easily used in handheld mode, when mounted on a tablet. The accuracy is worse than the previous device, mainly considering small details (e.g. fingers and nails), but it still satisfies all the application requirements. For example, amputees with prosthetic sockets typically use silicon liners at the interface, while the common scoliosis braces are usually worn over a light cotton layer. Thus, according to the technicians, the softness of the skin layers of patients can mitigate errors of about 2 mm in bony regions and even larger errors elsewhere thanks to thicker skin layers.
For the Kinect 1, we can state that it could be still a very cheap and acceptable device for acquiring medium to large body parts, depending on the final goal. For example, if we have to develop a new wrist orthosis, neglecting the fingers, the shape could be still similar to the real one, but the error would not be acceptable due to the thinner skin layers around the bones. On the other hand, the Kinect 1 would still be a valid device for creating a socket for the prosthesis of an above-knee amputee, because the thicker layers of soft tissues around the bone and the use of silicon liners could easily mitigate the deviation.
Even though the Kinect 2 showed good performance on the flat plane, the poor frame alignment of the SDK fusion algorithm that resulted from the tests with the spheres and the mannequin parts determined the impossibility in a realistic acquisition of patients for orthopaedic purposes.
The Rodin4D O&P has a nominal high accuracy, but the problems due to the magnetic tracking system must have pushed towards the use of other technologies. The orthopaedic solution companies, indeed, are offering today devices based on white or blue light, with software that perform the automatic alignment of the frames without additional markers.
4.5 Handheld motions results
The ANOVA analysis showed significant main effects of approach (p < 0.001) and area (p < 0.001) and a significant interaction effect (p < 0.001). Post hoc analysis showed that the deviation for approaches 3, 4 and 5 errors was statistically significantly lower when scanning the right hip area with respect to the whole body. For the whole chest scan, post hoc tests revealed that approach 5 provided higher errors than all other approaches (p < 0.001). For the right hip area, approach 5 showed statistically significantly higher errors with respect to approaches 2, 3 and 4. In addition, for the right hip, approach 3 performed better that approach 1 (p < 0.001).
Figure 11 illustrates the average standard deviation obtained comparing the different acquisitions using the 5 motion approaches listed in Section 2.5.5, both for the whole chest and for the selected bony region of the hip side. Even if the second method corresponded to doubling the rotation at a similar speed (requiring also to double the acquisition time), the results did not show a significant improvement in respect to the single straight loop.
Fig. 11
Average standard deviation of the cloud-to-mesh distance comparisons with five different scanning approaches and error bars representing the deviations from the average, when scanning the whole chest (left blue graph) and the right hip region (right orange graph). 1: one lap at fixed distance and no motion. 2: two laps at fixed distance and no motion. 3: one lap at fixed distance moving up and down. 4: one lap at fixed distance tilting up and down. 5: one lap and free movement. Upper bars indicate the significant difference between the approaches, the asterisk * identifies the approach (5) that is significantly different from others while the hashtag # highlights the approaches on the whole chest that have significant differences with the right hip
×
It appears that the motions 1, 2 and 4 can be overall comparable. The two full rotations (motion 2) provide more repeatability (smaller error bars in Fig. 11), but the time required, having doubled from 20 to 40 s, could be incompatible for some paediatric patients for which standing still over longer time is an issue.
The best results on the hip region were obtained with the motion 3, probably thanks to the vertical movement up and down that allowed acquiring a wider part of the lower regions, with a corresponding better alignment of the point-cloud. Unfortunately, this motion did not perform so well on the whole chest, giving also a large variability of results. This could be explained thinking that the sinusoidal motion was performed manually, thus varying each time. Moreover, the limit of 20 s for completing the full rotation, combined with the vertical movement, meant that the device was moved in a faster way. The stability during the acquisition of each frame, the possibility of unfocussed frames (as it happens with video cameras) and the alignment of all the frames could then be influenced negatively.
Also pitching the device up and down (motion 4) provided an improvement for acquiring the hip region, because the wider acquisition of the lower regions could have helped the alignment. The whole chest did not show the same variation of motion 3, because of the more stable position in the fixed height.
The last approach (motion 5) corresponds to the free movement for the operator. This resulted to be the worst for the measured differences and the variability in the measurements, represented by the black error bar. This can be related to the continuous fast movement variations that must have negatively affected the reconstruction.
We can thus conclude that it is better for the operator to move quietly, acquiring the whole surface with few movements, instead of continuously and rapidly changing the point of view. Indeed, slowly varying the device position could have more influence on the quality of the acquisition rather than the attempt to zoom on some regions.
5 Conclusions
A set of 3D devices was compared using the standard test objects, as required by the VDI/VDE 2634 guidelines (flat plane and spheres) [26]. Beyond this common practice in the field of metrology, we wanted to evaluate the 3D devices behaviour also in respect to the orthopaedic application. Since the production process of orthoses and prostheses is moving towards the digital transformation required for using the additive manufacturing technology [1], the acquired 3D model would be the input for designing custom-made orthoses and prostheses with CAD tools, as briefly represented in Fig. 1. We thus need to limit the error on the input of the process to obtain a reliable output. For this reason, we also used three mannequin parts that are representative of different anatomical regions, typical of the orthopaedic applications. The hand, the thigh and the chest enabled to investigate the behaviour of the devices with organic shapes, as the human body, with different levels of detail and different acquired volumes.
The results demonstrated the higher accuracy of the fixed devices in respect to the handheld ones, as shown also in [5]. However, the total time and the setup required to complete the acquisitions with the mannequin parts did not meet the requirements for a possible final application with a real patient. The handheld devices instead enabled performing a fast reconstruction of the chest model in about 20 s. This is an acceptable limit for a person that has to remain as still as possible in order to be reconstructed. In the comparison among Artec Leo, Structure Sensor, the two Kinect (V1 and V2) and the Rodin4D O&P scan, the results clearly showed the higher quality of the Artec Leo. On the other hand, the old but expensive manual laser scanner O&P scan has proved to have issues for the alignment of the acquired stripes, probably related to the magnetic tracking system.
Among the low-cost devices, the Structure Sensor demonstrated better accuracy than the two Kinect, in accordance with [7]. Another advantage of such a device is the possibility to be used both connected to a computer, using the software Skanect, for obtaining models with high resolution, and mounted on a tablet to work wirelessly in any location. This last configuration is considered to be important for the end users of orthopaedic centres that would acquire the patient’s chest in any available room and without the risk of cables in between the legs.
Finally, the Structure Sensor used as a handheld scanner was also tested with different user’s motions around the chest to evaluate the stability of the approaches, because the acquired model would be the input for designing custom-made orthoses and prostheses with CAD tools. Performing two full laps around the chest instead of a single lap, keeping a constant speed, did not show significant improvements. The analysis on the smaller region of the hip demonstrated that moving the scanner up and down, or tilting it, while moving around the patient improves the acquisition. Moreover, the free and uncontrolled motion has to be avoided by technicians, for the clear bad results.
Regarding the use of handheld devices in general, the operator should take care when moving the device around the patient, preferring smooth movements to quick ones. The movement speed can create possible issues in the alignment of the frames because of the limited frame rate (resulting in blurring). Moreover, having a stable path also improves the repeatability of the acquisitions.
The analyses showed that a simple test of the devices on a standard object, such as the flat plane or the spheres as required by VDI/VDE guidelines [26], is only a preliminary indication of the performances of the sensors. It remains an important step for the definition of the systematic error that could be removed using a proper calibration, thanks to standard test-fields. We can also say that the synthetic parameters (such as flatness error, probing errors in size and form and the spacing distance error) can give a quick idea for comparison, but a visual check to the results could give extra information about the device behaviour, for example, about the calibration errors shape or reconstruction quality and detail level. At the same time, the results can clearly change in real application.
Therefore, a second test in conditions similar to the final application (i.e. reconstructing human body segments in the case of this project), related not only to the object but also to the lab setup (available room space, budget, time and illumination) and to the acquisition approach, is always needed. Furthermore, depending on the field of use of the devices, a standardized calibrated object a specific reference representative of the final application should be created. In the ideal case of having also the original CAD model, the results could be compared without the bias due to the error of the device selected as a reference. Finally, future works should consider other handheld scanners and include reconstructions of real patients.
Acknowledgements
The authors wish to acknowledge also the support of the orthopaedic technicians of the centre “Biomedica su Misura” of Brescia and of Alessandro Ferrario for his support in data collection.
Code availability
The code that supports the findings of this study is available from the corresponding author, upon reasonable request.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Qualitative comparison of the acquisition results on the flat plane for all the devices. Note that the colour scale bar is different for each device to maximize the visual comparison of the possible pattern, and the saturation values are reported in millimitres
Die im Laufe eines Jahres in der „adhäsion“ veröffentlichten Marktübersichten helfen Anwendern verschiedenster Branchen, sich einen gezielten Überblick über Lieferantenangebote zu verschaffen.
