Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation
Introduction
Augmented reality (AR) based surgical navigation allows virtual organs overlaid on real organs to provide surgeons with an immersive visualized surgical environment. Compared with virtual reality based surgical navigation where a flat 2D monitor is used to display a virtual surgical scene, AR navigation where a virtual surgical scene is further registered to reality could provide enhanced realism and more intuitive information for surgical guidance [1]. There are three types of AR visualization technologies in the current surgical navigation systems: video-based display [2], [3], [4], [5], [6], see-through display [7], [8], and projection based AR [9], [10], [11]. Video-based display superimposes virtual organs on a (stereo) video stream captured by endoscopic cameras or head-mounted displays (HMD). Because camera videos cannot reproduce all the information obtained by the human visual system, see-through display and projection based AR were proposed for overlay on user's direct view using translucent mirrors or projectors. The virtual organs used in most direct view overlay systems are 2D-projected computer graphics (CG) models of organs derived from preoperative medical images. Compared with visual perception of a 3D object, 2D projection lacks two important visual clues that give a viewer the perception of depth: stereo parallax and motion parallax [12]. Depth perception in image-guided surgery enhances the safety of the surgical operation. Therefore, for those applications which superimpose virtual organs directly on a real surgical site, a 3D image is preferred so that consistent and correct parallax is maintained when observed from different locations. Depth perception can be obtained from the parallax encoded in the 3D image to give surgeons a distance sensing about surgical targets. “3D image” in this paper refers to the image encoding the parallax information which can be extracted by some optic device or human eyes to give a viewer the depth perception: the third dimension.
Among 3D image display technologies are stereoscopy [13], integral imaging (or integral photography) [14], and holography [15]. Stereoscopy creates depth perception using two view images, like the left and right images seen by the human visual system. The disparities between the two images encode the parallax information which can be extracted using such as a parallax barrier or polarized glasses for display. However, stereoscopy has very limited data bandwidth (only two images). The parallax information is only provided at two predefined view points. That is why we cannot see more even if we shift our eyes in a 3D film cinema. On the opposite side, holography directly records the wavefront of the light from a 3D object. Since all optical data irradiated from the object can be captured and reproduced with minimal loss, the parallax information can be encoded in a complete way. However, the data bandwidth is too huge to be handled in real time using current available devices with satisfactory resolution and viewing angle. In addition, holography requires very complicated and expensive devices to capture and reproduce 3D images, which limits its application. Integral imaging exists between the stereoscopy and the holography. It has medium data bandwidth and provides stereo parallax and full motion parallax within its viewing angle. The devices for extracting the encoded parallax information are very simple: It requires only a high resolution liquid crystal display (LCD) and a lens array in front of the LCD to display 3D images. The resulting 3D image presents stereo parallax and continuous motion parallax which can be directly observed by viewers without wearing special glasses. These exciting advantages make integral imaging under active research. Although integral imaging was first invented by Lippmann more than 100 years ago [16], it began to draw great attentions in the last decade since high resolution digital cameras and LCDs became available. A thorough review on integral imaging can be found in [17] and [18]. Current study of integral imaging focuses on 3D image pickup methods [19], image reconstruction methods [20], [21], and viewing quality enhancement [22], [23].
Our group first introduced integral imaging into surgical navigation for AR visualization [24], [25], [26], where an autostereoscopic 3D image is displayed by a lens array monitor (3D display) using computer-generated integral imaging (CGII), and is overlaid on the surgical site by a translucent mirror (AR window). The spatial registration between the 3D image and the surgical site is performed manually using a point-based registration method involving the use of a commercial optical tracking device. In our recent publication [27], an AR solution for dental surgery using 3D image overlay was proposed, where a stereo camera was employed to replace the commercial optical tracker for automatic real-time image registration and instrument tracking. For surgical overlay, the overlaid 3D image is expected to have the same geometric shape as the original organ. However, the resulting 3D image suffers from distortion owing to the inconsistent parameters between the digital recording and the optical reconstruction. In our previous work, the 3D image deformation issue was not well addressed and there was no quantitative evaluation on the geometric accuracy of 3D images. To reduce the distortion caused by the mismatch of the lens array and the LCD monitor, it took a lot of time to manually adjust the device itself. However, there was no guarantee that the resulting distortion could be controlled under certain level due to the lack of real-time feedback of current distortion level. The manual adjustment was performed in a trial-and-error fashion, which was very time consuming.
This paper proposes an automatic closed-loop 3D image calibration algorithm using a stereo camera to measure the distortion of 3D images as the real-time feedback. The final distortion error can be controlled under a small tolerance and the whole calibration procedure can be done within several seconds. To the best of our knowledge, there is no relevant work on compensating 3D image distortion. The second novelty of this paper is the new design of the graphics pipeline for computer-generated integral imaging. We have obtained the best rendering performance compared with our previous work. We also have made great improvement on the AR device for surgical overlay. Unlike the prototype in our previous work, the new design is compact, integrated and flexible. We performed comprehensive phantom experiments to evaluate the new AR device in terms of the accuracy and the visual effects of the 3D image overlay.
The rest of the paper is organized as follows. Section 2 describes the principle of integral imaging and introduces the new 3D image rendering pipeline together with its implementation. Section 3 presents the 3D image calibration algorithm. Section 4 describes the new AR overlay device based on the proposed methods. Section 5 presents evaluation experiments and the results. Finally, Section 6 concludes the paper.
Section snippets
CGII rendering
The basic principle of integral imaging is illustrated in Fig. 1. In the pickup procedure as shown in Fig. 1(a), the 3D object is captured by a lens array as individual elemental images recorded on the imaging plane (e.g., charge coupled device, CCD), which is located behind the lens array. For display, an LCD is used to display the previously recorded elemental images to reproduce the original routes of rays irradiated from the 3D object, causing a 3D image to be formed (Fig. 1(b)). The pickup
Necessity of 3D image calibration
We hope that the 3D image has high geometric fidelity (i.e., the same physical dimensions) as the original one for the purpose of surgical overlay. Theoretically, the 3D image has the same physical dimensions as the original 3D object if the parameters (px, py, lx, ly, g) are exactly the same in the rendering and display phases. Current commercial LCDs have accurate pixel pitches, and the lens array pitches can be guaranteed by precise manufacturing. However, the distance from the lens array to
AR overlay device
So far, we can display an undistorted 3D image of either a surface model or a medical volume at a specified location with respect to the 3D display. For the purpose of surgical overlay, we design a novel AR device for 3D image surgical overlay based on the proposed methods.
CGII rendering performance
The specifications of the 3D display for rendering performance evaluation is shown in Table 1. A CT volume of a head with 512 × 512 × 388 voxels and a MR volume of a heart with 512 × 512 × 186 voxels were used to test the direct raycasting method. A surface model of a lower jaw with 205,138 triangles, which was segmented and reconstructed from a CT volume, was used to test the view point moving method. For each dataset, the CEIs with two different resolutions were synthesized using the proposed
Discussion and conclusion
For the sake of 3D image surgical overlay, there are three important problems which should be addressed. First is how to display a 3D image given patient's medical data; second is how to generate a 3D image which has high geometric fidelity as the original one; third is how to overlay a 3D image on a specified location with respect to a tracking device. To answer the three questions, a CGII rendering method as well as the corresponding OpenGL pipeline implementation was presented to generate 3D
Conflict of interest
None.
References (29)
- et al.
Augmented reality in laparoscopic surgical oncology
Surg Oncol
(2011) - et al.
Augmented reality during robot-assisted laparoscopic partial nephrectomy: Toward real-time 3D-CT to stereoscopic video registration
Urology
(2009) - et al.
Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? preliminary in vitro and in vivo results
Eur Urol
(2009) - et al.
Projector-based augmented reality for intuitive intraoperative guidance in image-guided 3D interstitial brachytherapy
Int J Radiat Oncol Biol Phys
(2008) - et al.
Precision-guided surgical navigation system using laser guidance and 3d autostereoscopic image overlay
Comput Med Imaging Gr
(2010) - et al.
Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography
Surg Endosc
(2010) - et al.
A 3-d mixed-reality system for stereoscopic visualization of medical dataset
IEEE Trans Biomed Eng
(2009) - et al.
A fully automated calibration method for an optical see-through head-mounted operating microscope with variable zoom and focus
IEEE Trans Med Imaging
(2005) - et al.
Non-metal slice image overlay display system used inside the open type MRI
- et al.
Image overlay guidance for needle insertion in CT scanner
IEEE Trans Biomed Eng
(2005)
Projection-based visual guidance for robot-aided RF needle insertion
Int J Comput Assist Radiol Surg
Real time planning, guidance and validation of surgical acts using 3D segmentations, augmented reality projections and surgical tools video tracking
Proc SPIE
Autostereoscopic 3D displays
Computer
Visual discomfort and visual fatigue of stereoscopic displays: a review
J Imaging Sci
Cited by (91)
Novel Use of Virtual Reality and Augmented Reality in Temporomandibular Total Joint Replacement Using Stock Prosthesis
2024, Journal of Oral and Maxillofacial SurgeryClutch & Grasp: Activation gestures and grip styles for device-based interaction in medical spatial augmented reality
2023, International Journal of Human Computer StudiesRegistration methods for surgical navigation of the mandible: a systematic review
2022, International Journal of Oral and Maxillofacial SurgeryImageless Robotic Knee Arthroplasty
2021, Operative Techniques in OrthopaedicsCitation Excerpt :Registration is the process of aligning preoperative images with the intraoperative location of the patient's joint on the operating room table.3 Image-based registration involves the correlation of preoperative images such as computed tomography (CT) or magnetic resonance imaging (MRI) images with the intraoperative location of the joint using techniques such as paired-point matching registration,4 shape-based (surface) registration,5 and two dimensional to three dimensional (3D) or 3D to 3D using intraoperative fluoroscopic images.6 In contrast, imageless registration does not require resource-intensive imaging such as CT or MRI, but instead may use standard 2D radiographic images, intraoperative joint surface information, and/or kinetic movement information from the intraoperative registration of the hip and ankle centers.7
Augmented reality in craniomaxillofacial surgery: added value and proposed recommendations through a systematic review of the literature
2021, International Journal of Oral and Maxillofacial SurgeryCitation Excerpt :This required more subjects to obtain a statistically significant assessment of the system16,17,24,31–33,53,55. Similarly, the number of subjects in the phantom studies was low (18 articles included fewer than five subjects2,6,18,19,27,28,31,34–42,48,52). Despite this limitation, the studies could showcase the use of various AR devices.