Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation

doi:10.1016/j.compmedimag.2014.11.003

Computerized Medical Imaging and Graphics

Volume 40, March 2015, Pages 147-159

https://doi.org/10.1016/j.compmedimag.2014.11.003 Get rights and content

Highlights

•
A new graphics pipeline for computer-generated integral imaging is presented for autostereoscopic 3D image surgical overlay.
•
An automatic 3D image calibration paradigm for removing the distortion of 3D images is proposed.
•
The undistorted 3D image after the calibration yields sub-millimeter geometric accuracy.
•
A novel AR device for 3D image surgical overlay is presented based on the proposed methods.
•
A phantom application simulating oral and maxillofacial surgery was performed which has confirmed the usability of the AR system.

Abstract

Autostereoscopic 3D image overlay for augmented reality (AR) based surgical navigation has been studied and reported many times. For the purpose of surgical overlay, the 3D image is expected to have the same geometric shape as the original organ, and can be transformed to a specified location for image overlay. However, how to generate a 3D image with high geometric fidelity and quantitative evaluation of 3D image's geometric accuracy have not been addressed. This paper proposes a graphics processing unit (GPU) based computer-generated integral imaging pipeline for real-time autostereoscopic 3D display, and an automatic closed-loop 3D image calibration paradigm for displaying undistorted 3D images. Based on the proposed methods, a novel AR device for 3D image surgical overlay is presented, which mainly consists of a 3D display, an AR window, a stereo camera for 3D measurement, and a workstation for information processing. The evaluation on the 3D image rendering performance with 2560 × 1600 elemental image resolution shows the rendering speeds of 50–60 frames per second (fps) for surface models, and 5–8 fps for large medical volumes. The evaluation of the undistorted 3D image after the calibration yields sub-millimeter geometric accuracy. A phantom experiment simulating oral and maxillofacial surgery was also performed to evaluate the proposed AR overlay device in terms of the image registration accuracy, 3D image overlay accuracy, and the visual effects of the overlay. The experimental results show satisfactory image registration and image overlay accuracy, and confirm the system usability.

Introduction

Augmented reality (AR) based surgical navigation allows virtual organs overlaid on real organs to provide surgeons with an immersive visualized surgical environment. Compared with virtual reality based surgical navigation where a flat 2D monitor is used to display a virtual surgical scene, AR navigation where a virtual surgical scene is further registered to reality could provide enhanced realism and more intuitive information for surgical guidance [1]. There are three types of AR visualization technologies in the current surgical navigation systems: video-based display [2], [3], [4], [5], [6], see-through display [7], [8], and projection based AR [9], [10], [11]. Video-based display superimposes virtual organs on a (stereo) video stream captured by endoscopic cameras or head-mounted displays (HMD). Because camera videos cannot reproduce all the information obtained by the human visual system, see-through display and projection based AR were proposed for overlay on user's direct view using translucent mirrors or projectors. The virtual organs used in most direct view overlay systems are 2D-projected computer graphics (CG) models of organs derived from preoperative medical images. Compared with visual perception of a 3D object, 2D projection lacks two important visual clues that give a viewer the perception of depth: stereo parallax and motion parallax [12]. Depth perception in image-guided surgery enhances the safety of the surgical operation. Therefore, for those applications which superimpose virtual organs directly on a real surgical site, a 3D image is preferred so that consistent and correct parallax is maintained when observed from different locations. Depth perception can be obtained from the parallax encoded in the 3D image to give surgeons a distance sensing about surgical targets. “3D image” in this paper refers to the image encoding the parallax information which can be extracted by some optic device or human eyes to give a viewer the depth perception: the third dimension.

Among 3D image display technologies are stereoscopy [13], integral imaging (or integral photography) [14], and holography [15]. Stereoscopy creates depth perception using two view images, like the left and right images seen by the human visual system. The disparities between the two images encode the parallax information which can be extracted using such as a parallax barrier or polarized glasses for display. However, stereoscopy has very limited data bandwidth (only two images). The parallax information is only provided at two predefined view points. That is why we cannot see more even if we shift our eyes in a 3D film cinema. On the opposite side, holography directly records the wavefront of the light from a 3D object. Since all optical data irradiated from the object can be captured and reproduced with minimal loss, the parallax information can be encoded in a complete way. However, the data bandwidth is too huge to be handled in real time using current available devices with satisfactory resolution and viewing angle. In addition, holography requires very complicated and expensive devices to capture and reproduce 3D images, which limits its application. Integral imaging exists between the stereoscopy and the holography. It has medium data bandwidth and provides stereo parallax and full motion parallax within its viewing angle. The devices for extracting the encoded parallax information are very simple: It requires only a high resolution liquid crystal display (LCD) and a lens array in front of the LCD to display 3D images. The resulting 3D image presents stereo parallax and continuous motion parallax which can be directly observed by viewers without wearing special glasses. These exciting advantages make integral imaging under active research. Although integral imaging was first invented by Lippmann more than 100 years ago [16], it began to draw great attentions in the last decade since high resolution digital cameras and LCDs became available. A thorough review on integral imaging can be found in [17] and [18]. Current study of integral imaging focuses on 3D image pickup methods [19], image reconstruction methods [20], [21], and viewing quality enhancement [22], [23].

Our group first introduced integral imaging into surgical navigation for AR visualization [24], [25], [26], where an autostereoscopic 3D image is displayed by a lens array monitor (3D display) using computer-generated integral imaging (CGII), and is overlaid on the surgical site by a translucent mirror (AR window). The spatial registration between the 3D image and the surgical site is performed manually using a point-based registration method involving the use of a commercial optical tracking device. In our recent publication [27], an AR solution for dental surgery using 3D image overlay was proposed, where a stereo camera was employed to replace the commercial optical tracker for automatic real-time image registration and instrument tracking. For surgical overlay, the overlaid 3D image is expected to have the same geometric shape as the original organ. However, the resulting 3D image suffers from distortion owing to the inconsistent parameters between the digital recording and the optical reconstruction. In our previous work, the 3D image deformation issue was not well addressed and there was no quantitative evaluation on the geometric accuracy of 3D images. To reduce the distortion caused by the mismatch of the lens array and the LCD monitor, it took a lot of time to manually adjust the device itself. However, there was no guarantee that the resulting distortion could be controlled under certain level due to the lack of real-time feedback of current distortion level. The manual adjustment was performed in a trial-and-error fashion, which was very time consuming.

This paper proposes an automatic closed-loop 3D image calibration algorithm using a stereo camera to measure the distortion of 3D images as the real-time feedback. The final distortion error can be controlled under a small tolerance and the whole calibration procedure can be done within several seconds. To the best of our knowledge, there is no relevant work on compensating 3D image distortion. The second novelty of this paper is the new design of the graphics pipeline for computer-generated integral imaging. We have obtained the best rendering performance compared with our previous work. We also have made great improvement on the AR device for surgical overlay. Unlike the prototype in our previous work, the new design is compact, integrated and flexible. We performed comprehensive phantom experiments to evaluate the new AR device in terms of the accuracy and the visual effects of the 3D image overlay.

The rest of the paper is organized as follows. Section 2 describes the principle of integral imaging and introduces the new 3D image rendering pipeline together with its implementation. Section 3 presents the 3D image calibration algorithm. Section 4 describes the new AR overlay device based on the proposed methods. Section 5 presents evaluation experiments and the results. Finally, Section 6 concludes the paper.

Section snippets

CGII rendering

The basic principle of integral imaging is illustrated in Fig. 1. In the pickup procedure as shown in Fig. 1(a), the 3D object is captured by a lens array as individual elemental images recorded on the imaging plane (e.g., charge coupled device, CCD), which is located behind the lens array. For display, an LCD is used to display the previously recorded elemental images to reproduce the original routes of rays irradiated from the 3D object, causing a 3D image to be formed (Fig. 1(b)). The pickup

Necessity of 3D image calibration

We hope that the 3D image has high geometric fidelity (i.e., the same physical dimensions) as the original one for the purpose of surgical overlay. Theoretically, the 3D image has the same physical dimensions as the original 3D object if the parameters (p_x, p_y, l_x, l_y, g) are exactly the same in the rendering and display phases. Current commercial LCDs have accurate pixel pitches, and the lens array pitches can be guaranteed by precise manufacturing. However, the distance from the lens array to

AR overlay device

So far, we can display an undistorted 3D image of either a surface model or a medical volume at a specified location with respect to the 3D display. For the purpose of surgical overlay, we design a novel AR device for 3D image surgical overlay based on the proposed methods.

CGII rendering performance

The specifications of the 3D display for rendering performance evaluation is shown in Table 1. A CT volume of a head with 512 × 512 × 388 voxels and a MR volume of a heart with 512 × 512 × 186 voxels were used to test the direct raycasting method. A surface model of a lower jaw with 205,138 triangles, which was segmented and reconstructed from a CT volume, was used to test the view point moving method. For each dataset, the CEIs with two different resolutions were synthesized using the proposed

Discussion and conclusion

For the sake of 3D image surgical overlay, there are three important problems which should be addressed. First is how to display a 3D image given patient's medical data; second is how to generate a 3D image which has high geometric fidelity as the original one; third is how to overlay a 3D image on a specified location with respect to a tracking device. To answer the three questions, a CGII rendering method as well as the corresponding OpenGL pipeline implementation was presented to generate 3D

Conflict of interest

None.

References (29)

S. Nicolau et al.
Augmented reality in laparoscopic surgical oncology
Surg Oncol
(2011)
L.-M. Su et al.
Augmented reality during robot-assisted laparoscopic partial nephrectomy: Toward real-time 3D-CT to stereoscopic video registration
Urology
(2009)
D. Teber et al.
Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? preliminary in vitro and in vivo results
Eur Urol
(2009)
R. Krempien et al.
Projector-based augmented reality for intuitive intraoperative guidance in image-guided 3D interstitial brachytherapy
Int J Radiat Oncol Biol Phys
(2008)
H. Liao et al.
Precision-guided surgical navigation system using laser guidance and 3d autostereoscopic image overlay
Comput Med Imaging Gr
(2010)
R. Shekhar et al.
Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography
Surg Endosc
(2010)
V. Ferrari et al.
A 3-d mixed-reality system for stereoscopic visualization of medical dataset
IEEE Trans Biomed Eng
(2009)
M. Figl et al.
A fully automated calibration method for an optical see-through head-mounted operating microscope with variable zoom and focus
IEEE Trans Med Imaging
(2005)
K. Masamune et al.
Non-metal slice image overlay display system used inside the open type MRI
G. Fichtinger et al.
Image overlay guidance for needle insertion in CT scanner
IEEE Trans Biomed Eng
(2005)

R. Wen et al.

Projection-based visual guidance for robot-aided RF needle insertion

Int J Comput Assist Radiol Surg

(2013)

A. Osorio et al.

Real time planning, guidance and validation of surgical acts using 3D segmentations, augmented reality projections and surgical tools video tracking

Proc SPIE

(2010)

N.A. Dodgson

Autostereoscopic 3D displays

Computer

(2005)

M. Lambooij et al.

Visual discomfort and visual fatigue of stereoscopic displays: a review

J Imaging Sci

(2009)

Cited by (91)

Illuminating precise stencils on surgical sites using projection-based augmented reality
2024, Smart Health
In this paper we propose a system that connects surgeons to remote or local experts who provide real-time surgical guidance by illuminating salient markings or stencils (e.g. points, lines and curves) on the physical surgical site using a projector. The projection can be modified in real time by the expert using a GUI and can be seen by all in the operating room (OR) without the use of any wearables. This system overcomes the limitations of AR/VR headsets which can overlay information through a headset, but are obtrusive, not very accurate with movements, and visible only to the surgeon excluding others in the room. Overlaying information, at high precision, directly on the physical surgical site that can be seen by everyone in the OR can become an useful tool for skill transfer, expert consultation and training, especially in telemedicine.
In addition to the projector, the system comprises of a RGB-D camera (e.g. Kinect) for feedback, together designated as the PDC (Projector Depth Camera) unit. The PDC is driven by a PC. The RGB-D camera provides depth information in addition to an image at video frame rates. A high resolution mesh of the surgical site is captured using the PDC unit initially. During the surgical planning, training or execution session, this digital model can be marked by appropriate incision markings on a tablet or monitor using touch based or mouse based interface, on the same local machine or after being transmitted to a remote machine. These markings are then communicated back to the PDC unit and illuminated at high precision via the projector on the surgical site in real time. If the surgical site moves during the process, the movement is tracked and updated quickly on the surgical site. Our method specifically overcomes the obtrusive, exclusive, and indirect attributes of headsets and displays while maintaining high accuracy of registration with movements.
Novel Use of Virtual Reality and Augmented Reality in Temporomandibular Total Joint Replacement Using Stock Prosthesis
2024, Journal of Oral and Maxillofacial Surgery
This technical innovation demonstrates the use of ImmersiveTouch Virtual Reality (VR) and Augmented Reality (AR)–guided total temporomandibular joint replacement (TJR) using Biomet stock prosthesis in 2 patients with condylar degeneration. TJR VR planning includes condylar resection, prosthesis selection and positioning, and interference identification. AR provides real-time guidance for osteotomies, placement of prostheses and fixation screws, occlusion verification, and flexibility to modify the surgical course. Radiographic analysis demonstrated high correspondence between the preoperative plan and postoperative result. The average differences in the positioning of the condylar and fossa prosthesis are 1.252 ± 0.269 mm and 1.393 ± 0.335 mm, respectively. The main challenges include a steep learning curve, intraoperative technical difficulties, added surgical time, and additional costs. In conclusion, the case report demonstrates the advantages of implementing AR and VR technology in TJR’s using stock prostheses as a pilot study. Further clinical trials are needed prior to this innovation becoming a mainstream practice.
Clutch & Grasp: Activation gestures and grip styles for device-based interaction in medical spatial augmented reality
2023, International Journal of Human Computer Studies
Presenting medical volume data using augmented reality (AR) can facilitate the identification of anatomical structures, the perception of their spatial relations and the development of mental maps compared to more commonly used monitors. However, interaction methods explored in these conventional settings may not be applicable in AR environments, or perform differently. In terms of mode activation, gestural interaction was shown to be a viable, touchless alternative to traditional input devices, which is desirable in sterile medical use cases. Therefore, we present a user study (n $=$ 21) comparing hand and foot gestures with voice commands for the activation of interaction modes within a projector-based, spatial AR prototype to visualize medical volume data. Interaction itself was performed via hand movements captured by a data glove. Consistent, statistically significant results across measured variables suggest advantages of voice commands. In addition, a second experiment (n $=$ 17) compared the hand-based interaction with two motion-sensitive devices held in power and in precision grip respectively. All modes were activated using voice commands. No considerable differences between tested grip styles could be determined. The findings suggest that the choice of preferable interaction devices is user and use case dependent.
Registration methods for surgical navigation of the mandible: a systematic review
2022, International Journal of Oral and Maxillofacial Surgery
Image-to-patient registration in navigated mandibular surgery is complex due to the mobile nature of the mandible compared with other craniofacial bones. As a result, surgical navigation is rarely employed in the mandibular region. This systematic review provides an overview of the different registration methods that are used for surgical navigation of the mandible. A systematic search was performed in the MEDLINE Ovid, Scopus, and Embase databases on March 25, 2021. Search terms included synonyms for mandibular surgery, surgical navigation, and registration methods. Articles about navigated mandibular surgery, where the registration method was explicitly mentioned, were included. The database search yielded a total of 2952 articles, from which 81 articles remained for analysis. Four main registration methods were identified: point registration, surface registration, hybrid registration, and computer vision-based registration. The mobility of the mandible is accounted for by either keeping the mandible in a fixed position during preoperative imaging and surgery, or by tracking the mandibular movements. Although different registration methods are available for navigated mandibular surgery, there is always a trade-off between accuracy, registration time, usability, and invasiveness. Future studies should focus on testing the different methods in larger patient studies and should report the registration accuracy.
Imageless Robotic Knee Arthroplasty
2021, Operative Techniques in Orthopaedics
Citation Excerpt :
Registration is the process of aligning preoperative images with the intraoperative location of the patient's joint on the operating room table.3 Image-based registration involves the correlation of preoperative images such as computed tomography (CT) or magnetic resonance imaging (MRI) images with the intraoperative location of the joint using techniques such as paired-point matching registration,4 shape-based (surface) registration,5 and two dimensional to three dimensional (3D) or 3D to 3D using intraoperative fluoroscopic images.6 In contrast, imageless registration does not require resource-intensive imaging such as CT or MRI, but instead may use standard 2D radiographic images, intraoperative joint surface information, and/or kinetic movement information from the intraoperative registration of the hip and ankle centers.7
Robotic-assisted total knee arthroplasty is increasing in prevalence and has been shown to enable improved accuracy in implant positioning in total knee arthroplasty (TKA). Robotic assisted TKA can be categorized into image-guided and imageless techniques. In image-guided robotic TKA systems, preoperative imaging, most frequently computed tomography, is used to map bony anatomical landmarks to preoperatively obtained image to plan bone resections and implant sizing and positioning. Imageless robotic-assisted TKA does not require preoperative advanced imaging and intraoperatively maps bony anatomy to guide bone resection and implant sizing and placement. The purpose of this article is to describe the surgical technique for imageless robotic-assisted TKA and to provide a concise review of literature regarding surgical outcomes.
Augmented reality in craniomaxillofacial surgery: added value and proposed recommendations through a systematic review of the literature
2021, International Journal of Oral and Maxillofacial Surgery
Citation Excerpt :
This required more subjects to obtain a statistically significant assessment of the system16,17,24,31–33,53,55. Similarly, the number of subjects in the phantom studies was low (18 articles included fewer than five subjects2,6,18,19,27,28,31,34–42,48,52). Despite this limitation, the studies could showcase the use of various AR devices.
This systematic review provides an overview of augmented reality (AR) and its benefits in craniomaxillofacial surgery in an attempt to answer the question: Is AR beneficial for craniomaxillofacial surgery? This review includes a description of the studies conducted, the systems used and their technical characteristics. The search was performed in four databases: PubMed, Cochrane Library, Embase, and Web of Science. All journal articles published during the past 11 years related to AR, mixed reality, craniomaxillofacial, and surgery were considered in this study. From a total of 7067 articles identified using AR- and surgery-related keywords, 39 articles were finally selected. Based on these articles, a classification of study types, surgery types, devices used, metrics reported, and benefits were collected. The findings of this review indicate that AR could provide various benefits, addressing the challenges of conventional navigation systems, such as hand–eye coordination and depth perception. However, three main concerns were raised while performing this study: (1) it is complicated to aggregate the metrics reported in the articles, (2) it is difficult to obtain statistical value from the current studies, and (3) user evaluation studies are lacking. This article concludes with recommendations for future studies by addressing the latter points.

View all citing articles on Scopus

View full text

Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation

Highlights

Abstract

Introduction

Section snippets

CGII rendering

Necessity of 3D image calibration

AR overlay device

CGII rendering performance

Discussion and conclusion

Conflict of interest

Surg Oncol

Urology

Eur Urol

Int J Radiat Oncol Biol Phys

Comput Med Imaging Gr

Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography

Surg Endosc

A 3-d mixed-reality system for stereoscopic visualization of medical dataset

IEEE Trans Biomed Eng

A fully automated calibration method for an optical see-through head-mounted operating microscope with variable zoom and focus

IEEE Trans Med Imaging

Non-metal slice image overlay display system used inside the open type MRI

Image overlay guidance for needle insertion in CT scanner