Medical Imaging and Augmented Reality

We have developed a visualization environment to assist surgeons with therapy delivery inside the beating heart, in absence of direct vision. This system employs virtual reality techniques to integrate pre-operative anatomical models, real-time intra-operative imaging, and models of magnetically-tracked surgical tools. Visualization is enhanced via 3D dynamic cardiac models constructed from high-resolution pre-operative MR or CT data and registered within the intra-operative imaging environment. In this paper, we report our experience with a feature-based registration technique to fuse the pre- and intra-operative data during an

in vivo

intracardiac procedure on a porcine subject. Good alignment of the pre- and intra-operative anatomy within the virtual reality environment is ensured through the registration of easily identifiable landmarks. We present our initial experience in translating this work into the operating room and employing this system to guide typical intracardiac interventions. Given its extensive capabilities in providing surgical guidance in the absence of direct vision, our virtual environment is an ideal candidate for performing off-pump intracardiac interventions.

In lung cancer image-guided therapy, a real-time electromagnetic tracked microendoscopic optical imaging probe is guided to the small lung lesion of interest. The alignment of the pre-operative lung CT images as well as the intra-operative serial images is often an important step to accurately guide and monitor the interventional procedure in the diagnosis and treatment of these small lung lesions. Registering the serial images often relies on correct segmentation of the images and on the other hand, the segmentation results can be further improved by temporal alignment of the serial images. This paper presents a joint serial image registration and segmentation algorithm. In this algorithm, serial images are segmented based on the current deformations, and the deformations among the serial images are iteratively refined based on the updated segmentation results. No temporal smoothness about the deformation fields is enforced so that the algorithm can tolerate larger or discontinuous temporal changes that often appear during image-guided therapy. Physical procedure models could also be incorporated to our algorithm to better handle the temporal changes of the serial images during intervention. In experiments, we apply the proposed algorithm to align serial lung CT images. Results using both simulated and clinical images show that the new algorithm is more robust compared to the method that only uses deformable registration.

In current robotic surgery, dexterity is enhanced by microprocessor controlled mechanical wrists which allow motion scaling for reduced gross hand movements and improved performance of micro-scale tasks. The continuing evolution of the technology, including force feedback and virtual immobilization through real-time motion adaptation, will permit complex procedures such as beating heart surgery to be carried out under a static frame-of-reference. In pursuing more adaptive and intelligent robotic designs, the regulatory, ethical and legal barriers imposed on interventional surgical robots have given rise to the need of a tightly integrated control between the operator and the robot when autonomy is considered. This paper outlines the general concept of

perceptual docking

for robotic control and how it can be used for learning and knowledge acquisition in robotic assisted minimally invasive surgery such that operator specific motor and perceptual/cognitive behaviour is acquired through

in situ

sensing. A gaze contingent framework is presented in this paper as an example to illustrate how saccadic eye movements and ocular vergence can be used for attention selection, recovering 3D tissue deformation and motor channelling during minimally invasive surgical procedures.

In this paper, we propose a novel statistical deformable model for bone-related soft-tissue prediction, which we called Br-SDM. In Br-SDM, we have integrated Finite Element Model(FEM) and Statistical Deformable Model(SDM) to achieve both accurate and efficient prediction for orthognathic surgery planning. By combining FEM-based surgery simulation for sample generation and SDM for soft tissue prediction, we are able to capture the prior knowledge of bone-related soft-tissue deformation for different surgical plans. Then the post-operative appearance can be predicted in a more efficient way from a Br-SDM based optimization. Our experiments have shown that Br-SDM is able to give comparable soft-tissue prediction accuracy with respect to conventional FEM-based prediction while only requires 10% of its computational cost.

This paper describes a method for 3D automated preoperative planning of the femoral stem in total hip arthroplasty (THA). The stem planning is formulated as a problem to determine the optimal position, rotation, and size, on the 3D surface model of femur reconstructed from CT images. We obtain the parameters that maximize the fitness between the femoral canal and stem surfaces subject to the positional and rotational constraints. The maximization is performed by local optimization from multiple initial positions. The proposed method was experimentally evaluated by the difference from planning results of an experienced surgeon in 7 cases. The average positional and rotational differences were 1.9 mm and 2.5 deg., respectively, and there was size difference only in 1 case for the proposed method while these differences were 2.8 mm, 5.0 deg., and 5 cases for an existing method. The proposed method showed better performance than the existing method.

This paper shows the viscoelastic and nonlinear liver model for organ model based needle insertion, in which the deformation of an organ is estimated and predicted, and the needle trajectory is decided with organ deformation taken into consideration. An organ model including detailed material characteristics is important in order to achieve the proposed method. Firstly, the material properties of the liver are modeled from the measured data and its viscoelastic characteristics are represented by differential equations, including the term of the fractional derivative. Nonlinearity in terms of the stiffness was measured, and modeled using the quadratic function of strain. Next, a solution of an FE(Finite element) model using such material properties is shown. We use the sampling time scaling property as the solution for the viscoelastic system, while the solution for a nonlinear system using the Euler method and the Modified Newton-Raphson method is also shown. Finally, the deformation of liver model is calculated and pig liver of in vivo situation is obtained from medical ultrasound equipment. Comparing the relationship between needle displacement and force on real liver and liver model, we validate the proposed model.

While finite element methods have been extensively used for computational cardiology, the complicated and computationally expensive meshing procedures largely increase the difficulties of improving numerical accuracy. Furthermore, as the complexity of the element structure increases with the order of polynomial can be handled, the finite element procedures are either simple but numerically inaccurate or accurate but labor intensive. In view of these problems, we adopt the meshfree methods for computational cardiology. Using the meshfree methods, the heart is represented by a set of nodes distributed in the myocardium without any mesh, thus spatial refinements only involve distribution of extra nodes to the area of interest. Furthermore, as the order of polynomial is not limited by elements, it can be increased with relatively ease. These are desirable features as they provide the flexibility for improving numerical accuracies. In this paper, the simulation of the cardiac electromechanical dynamics using the meshfree methods will be introduced. Experiments have been done on a cubical object to provide an insight into the electromechanical dynamics, and also on a canine heart model to show the physiological plausibility of the simulation.

A wheelchair propulsion analysis system that can be used to generate the motion of propulsion and calculate the driving force and the muscular force using a human model having a skeleton and muscles was developed. To evaluate the driving force and muscular forces calculated by the analysis system, typical forward movement of two cycles of propulsion from the start-up on a flat floor and on a slope were measured and the movement including the trunk was input to the system. In addition, the muscular forces of biceps, triceps, clavicular and acromion deltoids, pectoralis major and extensor carpi radialis were evaluated by electromyogram. The calculated driving force was in good agreement with the measured driving force even on the slope and the trend of calculated muscular force corresponded with measured electromyogram data. The system is effective for analyzing the driving force and load on muscles during the wheelchair propulsion.

Fiducial marker is widely used for registration in image guided neurosurgery. In this paper we propose a novel automatic approach of localizing the center of these markers in CT volume. We first segment the volume into three parts according to shape index and curvedness of each voxel. Part1 includes some voxels on top of the markers, part2 includes some voxels at the bottom of the markers, and part3 is the background. Then we cluster voxels in part1 and part2 separately. The result groups in part1 are candidates for the top of markers, and the groups in part2 are candidates for bottom. For each group in part2, if there is a group in part1 that is close to it, this pair is regarded as a marker and the centroid of the group from part2 is the marker center. Experiments show that the marker center can be localized with sub millimeter accuracy.

The determination of the position and rotation of vertebrae is important for the understanding of normal and pathological spine anatomy. Existing techniques, however, estimate the position and rotation parameters from two-dimensional (2D) planar cross-sections, are relatively complex and require a relatively large amount of manual interaction. We have developed an automated and modality-independent method for the determination of the position and rotation of vertebrae in three dimensions (3D) that is based on registration of image intensity gradients, extracted in 3D from symmetrical vertebral parts. The method was evaluated on 52 vertebrae; 26 were acquired by computed tomography (CT) and 26 by magnetic resonance (MR). The results show that by the proposed gradient-based registration of symmetrical vertebral parts, the position and rotation of vertebrae in 3D can be successfully determined in both CT and MR spine images. As the position and rotation of vertebrae in 3D are among the most important spine parameters, the proposed method may provide valuable support in the evaluation of deformities and disease processes that affect the spine.

The foremost premise for the success of noninvasive volumetric myocardial transmembrane (TMP) imaging from body surface potential (BSP) recordings is a realistic yet efficient electrocardiographic model which relates volumetric myocardial TMP distributions to BSP distributions. With a view towards the inverse problem, appropriate model simplifications should be emphasized to balance the accuracy of the model with the feasibility of the inversion. In this paper, we present a novel coupled meshfree-BEM platform to represent the combined heart-torso structure and derive the associated TMP-to-BSP models. The numerical accuracy and convergency of the presented approach is verified against analytic solutions on a synthetic geometry. The associated simplifications are justified by comparing models at different level of complexity, which further demonstrates the benefits of homogeneous torso assumption in the inverse problem. Initial simulation experiments on a realistic heart-torso structure further show the physiological plausibility of the presented approach.

Functional neuroimaging methods give the opportunity of investigating human brain functioning. Mostly used functional neuroimaging techniques include Electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), positron emission tomography (PET) and optical imaging. Among these techniques EEG has the best time resolution, while fMRI has the best spatial resolution. High temporal resolution of EEG is an attractive property for neuroimaging studies. EEG inverse problem is needed to be solved in order to identify the locations and the strength of the electrical sources forming EEG/ERP topographies. Low spatial resolution of the scalp topography causes this localization problem more complicated. In this paper, a spatial preprocessing method, which separates a topography into two or more subtopographies is proposed. The decomposition procedure is based on defining a spatial map with radial basis functions which forms the subtopographies. A simulated data is used to exhibit the advantage of using this decomposition technique prior to EEG source localization. It is shown that the accuracy of the source localization problem is improved by using the subtopographies instead of using the raw topography.

The inverse approach from MR images to electrical propagation is very novel, but difficult due to complicated processes from electrical excitation to heart contraction. A novel strategy is presented to recover cardiac electrical excitation pattern from medical image sequences and ECG data. We used MRI images to estimate the current density and visualize it on the surface of the heart model. The ECG data also be used to achieve the time synchronization when the propagation of the current density. Experiments are conducted on a set of real time MRI images, also with the real ECG data, and we get favorable results.

In this paper, we propose a fully automated method to individually classify patients with Alzheimer’s disease (AD) and elderly control subjects based on anatomical magnetic resonance imaging (MRI). Our approach relies on the identification of gray matter (GM) atrophy patterns using whole-brain parcellation into anatomical regions and the extraction of GM characteristics in these regions. Discriminative features are identified using a feature selection algorithm and used in a Support Vector Machine (SVM) for individual classification. We compare two different types of parcellations corresponding to two different levels of anatomical details. We validate our approach with two distinct groups of subjects: an initial cohort of 16 AD patients and 15 elderly controls and a second cohort of 17 AD patients and 13 controls. We used the first cohort for training and region selection and the second cohort for testing and obtained high classification accuracy (90%).

In this paper, we proposed a new surface-based fractal information dimension (FID) method to quantify the cortical complexity. Unlike the traditional box-counting method to measure the capacity dimension, our method is a surface-based fractal information dimension method, which incorporates surface area into the probability calculation and thus encapsulates more information of the original cortical surfaces. The accuracy of the algorithm was validated via experiments on phantoms. With the proposed method, we studied the abnormalities of the cortical complexity of the early blind (EB; n=15), compared with matched controls (n=15). We found significantly increased FIDs in the left occipital lobe and decreased FIDs in the right frontal and right parietal lobe in early blind compared with controls. The results demonstrated the potential of the proposed method for identifying cortical abnormalities.

Considering the anatomical variations and unpredictable nature of surgeries, visibility during surgery is very important especially to correctly diagnose problems. Hyperspectral imaging has developed as a compact imaging and spectroscopic tool that can be used for different applications including medical diagnostics. This paper presents the application of hyperspectral imaging as a visual supporting tool to detect different organs and tissues during surgeries. It will be useful for finding ectopic tissues and diagnosis of tissue abnormalities. The high-dimensional data were compressed using wavelet transform and classified using artificial neural networks. The performance of this method is evaluated for the detection of the spleen, colon, small intestine, urinary bladder, and peritoneum in a surgery on a pig.

Liver segmentation in MR Image is the first step of our automated liver perfusion analysis project. Traditional Level Set methods and active contours were often used to segment the liver, but the results were not always promising due to noise and the low gradient response on the liver boundary. In this paper we propose a novel level set based variational approach that incorporates shape prior knowledge into the improved Chan-Vese’s model [1] which can overcome the leakage and over-segmentation problems. The experiments are taken on abdomen MRI series and the results reveal that our improved level set based shape prior method can segment liver shape precisely and a refined liver perfusion curve without respiration affection can be achieved.

A framework for optimisation of specific criteria across the shape variability found in a population is proposed. The method is based on level set segmentation in the parametric space defined by Principal Component Analysis (PCA). The efficient narrow band evolution of the level set allows to search for the instances only in the neighborhood of the zero level set and not in the whole shape space. We are able to optimise any given criterion not to provide a single best fitting instance in the shape space, but rather to provide a group of instances that meet the criterion. This effectively defines a partition in the shape space, which can have any topology. The method works for data of any dimension, determined by the number of principal components retained. Results are shown on the application to shape analysis of human femora.

We propose a novel Statistical Piecewise Assembled Model (SPAM) to address the open problem of small sample size encountered when applying Point Distribution Models (PDM) in 3-D medical data analysis. Specifically, in our SPAM, the Statistical Frame Model (SFM) constructed from the salient landmarks characterizes the global topological variability of the structure. Then the landmarks are employed to partition a complex object surface into piecewise segments. After that, the Statistical deformable Piecewise surface segment Models (SPMs) are established to define the fine details of local surface shape variations. The hierarchical nature of SPAM enables it to generate much more variation modes than conventional statistical models given a very small sample size training set. The experimental results demonstrate that SPAM can achieve more accuracy rates for model representation compared with traditional Active Shape Model (ASM) and Multi-resolution ASM.

Although there are numerous publications on amygdala volumetry, so far there has not been many studies on modeling local amygdala surface shape variations in a rigorous framework. This paper present a systematic framework for modeling local amygdala shape. Using a novel surface flattening technique, we obtain a smooth mapping from the amygdala surface to a sphere. Then taking the spherical coordinates as a reference frame, amygdala surfaces are parameterized as a weighted linear combination of smooth basis functions using the recently developed weighted spherical harmonic representation. This new representation is used for parameterizing, smoothing and nonlinearly registering a group of amygdala surfaces. The methodology has been applied in detecting abnormal local shape variations in 23 autistic subjects compared against 24 normal controls. We did not detect any statistically significant abnormal amygdala shape variations in autistic subjects. The complete amygdala surface modeling codes used in this study is available at

http://www.stat.wisc.edu/~mchung/research/amygdala

.

This paper presents a method for CT-US rigid registration in minimally-invasive computer-assisted orthopaedic surgery, whereby the registration procedure is reformulated to enable effectively real-time registrations. A linear Kalman filter based algorithm is compared to an Unscented Kalman filter based method in simulated and experimental scenarios. The validation schemes demonstrate that the linear Kalman filter is more accurate, more robust, and converges quicker than the UKF, yielding an effectively real-time method for rigid registration applications, circumventing surgeons’ waiting times.

Recent improvements in the resolution of Positron Emission Tomography (PET) imaging have not translated into equivalent advances in diagnostic accuracy. Due to long acquisition times involved, the functional imaging modality is hampered by motion artefacts due to respiratory motion. In this paper, two methods for correcting reconstructed PET images through a list-mode re-binning process are investigated. The first method rebins the list-mode data according to a globally defined 3D affine transformation. The second is a novel approach that combines multiple independent 2D affine transforms in order to exploit the specific properties of 2D tomographic reconstruction. Each affine transformation method is applied to the respiratory gated sequence of line-of-response events prior to image reconstruction, thus compensating for any respiratory motion. The deformation models are derived from a non-rigid 3D/3D registration model applied to retrospectively gated MRI acquired during free-breathing. The motion correction schemes are validated using a simulation framework with respiratory gated MRI scans of 10 subjects to generate the required activity maps for estimating emission sinograms. This allows the ground truth solution to be derived so that the motion corrected reconstruction can be compared quantitatively. It is shown that the higher degrees of freedom of the 2D affine motion compensation model is superior to the 3D affine transform provided one incorporates weak material constraints to avoid ill conditioning and can tolerate the lower SNR that 2D reconstruction implies.

A novel system for image guidance in totally endoscopic coronary artery bypass (TECAB) is presented. Key requirement is the availability of 2D-3D registration techniques that can deal with non-rigid motion and deformation. Image guidance for TECAB is mainly required before the mechanical stabilization of the heart, thus the most dominant source of non-rigid deformation is the motion of the beating heart.

To augment the images in the endoscope of the da Vinci robot, we have to find the transformation from the coordinate system of the preoperative imaging modality to the system of the endoscopic cameras.

In a first step we build a 4D motion model of the beating heart. Intraoperatively we can use the ECG or video processing to determine the phase of the cardiac cycle. We can then take the heart surface from the motion model and register it to the stereo-endoscopic images of the da Vinci robot using 2D-3D registration methods. We are investigating robust feature tracking and intensity-based methods for this purpose.

Images of the vessels available in the preoperative coordinate system can then be transformed to the camera system and projected into the calibrated endoscope view using two video mixers with chroma keying. It is hoped that the augmented view can improve the efficiency of TECAB surgery and reduce the conversion rate to more conventional procedures.

To reduce the invasiveness of surgery, we developed an outer sheath device using a flexible toothed link and pneumatic locking mechanism that works with flexible devices used in minimally invasive surgery. The outer sheath can be switched between flexible and rigid modes, and the angle of its tip can be controlled by a nylon wire. All parts of this device are made of plastic and are MRI-compatible. We manufactured a sheath prototype, 300 mm long, with a 20-mm outer diameter, and an 8-mm inner diameter. Experiment results showed that the outer sheath can protect tissues from high insertion force and secure the path for flexible devices. It can follow a curved path with a reasonable radius.

Medical technologies have undergone significant development to overcome the problems inherent in Minimally Invasive Surgery (MIS) such as inhibited manual dexterity, reduced visual information and lack of direct touch feedback to make it easier for surgeons to operate. An endoscopic tool incorporating haptic feedback is being developed to increase the effectiveness of diagnostic procedures by providing force feedback. Magnetic Resonance Imaging (MRI) guidance is possible to allow tool localisation, however this enforces the requirement of MR compatibility on the device. This paper describes the work done in developing MR compatible sensing devices using piezoelectric sensor elements in two different formats and how each can be used to locate subsurface inclusions in s mined oft substrates. Results show that the position of a hard inclusion can be deterwith both methods.

The authors have developed an Non-Invasive Ultrasound Theragnostic System to decrease the strain of patients and medical doctors. The system we propose tracks and follows movement in an affected area –kidney stones here– while High-Intensity Focused Ultrasound (HIFU) is irradiated onto the area. In this paper, a framework of the non-invasive ultrasound theragnostic system is proposed and illustrated. Specifically, the concept of the system is proposed at first. Secondly, decomposing and reconstructing (structuring) of the functional requirements are discussed. Third, the constructed system, which is based on those structured functional requirements, is illustrated. Fourth, the result of the servoing experiments of the model stone is reported to confirm the effectiveness of the proposed construction methodology and constructed system.

To improve therapy research against cancer, we propose a robotized computer assisted system for needle insertion devoted to the small animal. The system is composed of a robotic arm on which the needle is attached and two cameras rigidly linked. It needs a preoperative CT acquisition in which the biologist defines an entry point and the target he wants to reach. The needle guidance is ensured by a visual servoing and the needle is registered in the CT frame using radio-opaque markers stuck beforehand on the small animal.

Biologists estimate that such a system can be beneficial if the time preparation per animal remains below 15 minutes and if the insertion accuracy is within 1 mm. Several error sources can be identified : the error due to the system only, the organ repositioning error (due to breathing motions) and the error induced by the needle insertion.

In this paper, we report an evaluation on living rats of the system error and of the organ repositioning error. Encouraging results show that a global accuracy of 0.9 mm may be reached with an acceptable preparation time.

We present our new laser device for prospective human fetoplacental surgery including that of twin-twin transfusion syndrome. We developed a composite-type optical fiberscope (2.2-mm in diameter) that enables transmission of 40-W Yb fiber laser light alongside of coaxial fetoscopic images. Using a laser condensing lens on the fiberscope tip, Yb fiber laser light can be focused 10-mm off. Using porcine liver tissue, despite changes in delivered laser energy, the diameter and depth of cauterized areas remained constant when the liver position agreed with the laser focal point, although the laser output altered the extent of tissue ablation. In conclusion, the performance of the fiberscope can be well controlled with accurate and efficient ablation of the target tissue. This Yb fiber laser fiberscope is expected to work much better if mounted on a miniature bending manipulator and if provided with additional functions (real-time distance and blood flow measurements).

This paper describes a surgical manipulator to stabilize intrauterine fetus with ultrasound image guidance. The manipulator includes an outline of 4 mm in diameter, a mechanism with 7 joints, and a set of balloons. The manipulator is arranged as straight form and the balloon is fold to be a minimum size before the insertion. Accuracy evaluation of bending performance showed that the standard deviations were ±3.6 degrees on wired-driven mechanism and ±1.6 degrees on linkage-driven mechanism. Experimental results also demonstrated high repeatability of the mechanisms. In feasibility experiments, ultrasound images in 2D and 3D modes were examined for guiding the manipulator. The 2D images provided wide view and easily viewable display of the balloon inflation. The 3D images provided easily viewable display of bending motion of the arm and the relative position of a phantom. It is possible to operate the manipulator in utero under the ultrasound guidance in 2D and 3D by switching them in each procedural stage.

Effective hand-eye coordination is an important aspect of training in laparoscopic surgery. This paper investigates the interdependency of the hand and eye movement along with the variability of their temporal relationships based on Granger-causality. Partial directed coherence is used to reveal the subtle effects of improvement in hand-eye coordination, where the causal relationship between instrument and eye movement gradually reverse during simple laparoscopic tasks. For assessing the practical value of the proposed technique for minimally invasive surgery, two laparoscopic experiments have been conducted to examine the ability of the trainees in handling mental rotation tasks, as well as dissection and manipulation skills in laparoscopic surgery. Detailed experimental results highlight the value of the technique in investigating hand-eye coordination in laparoscopic training, particularly during early motor learning for complex bimanual procedures.

We describe a real-time virtual reality platform and a novel visualization technique designed to deliver constant rendering speed of highly detailed anatomical structures at interactive rates using portable, and low-cost computers. Our solution represents the torso section of the human body as a volumetric data set and employs label maps as the prime data format of storing anatomical structures. Multi-channel 3D textures are uploaded to the GPU and a simple pixel shader algorithm allows operators to select structures of interest in real-time. The visualization module described herein was successfully integrated into a virtual-reality 3D Anatomical Guidance System for Ultrasound Operators and its rendering performance tested on a “backpack” system.

In this paper, a surgical navigation system based on stereo-vision has been proposed. With the help of a projector, surface point data of a patient are firstly captured from two cameras by using the stereo-vision technique. Next, these point data is then registered by another surface point data obtained from the patient’s pre-stored CT images using Feature-added ICP algorithm. For surgical navigation, a stereo-vision-based tracker is also designed in this system. In this manner, the anatomical information obtained from CT is fused on the images captured by the cameras and the tracker can be mounted to any surgical device for further surgical applications. Compared to conventional image guided surgeries, the proposed system has the advantage of fusing the visual and anatomical information such that it is more accurate and convenient.

In recent years, several fetal cardiac malformations have been increasingly treated before birth with gradually improved outcome, although the technique is still demanding and invasive. We newly developed a computer-aided system for energy delivery of high-intensity focused ultrasound (HIFU) to correct cardiac morphologic abnormalities

in vivo

much less invasively. The HIFU system could be controlled in real-time by a computer-based analysis of 2D-sonographic left ventricular images for optimal triggering off HIFU. Using beating heart of two anesthetized adult rabbits, the system successfully achieved a non-touch gross ablation of the atrial septum in one animal, and in the other HIFU energy was inadvertently mistargeted on the posterior wall of the left atrium with a resultant small transmural opening. We believe that the HIFU system will be introduced with pinpoint accuracy to minimally invasive treatment of fetal cardiac abnormalities that have intact or highly restrictive atrial septum.

Medical technology has advanced with the introduction of robot technology, making previous medical treatments that were very difficult far more possible. However, operation of a surgical robot demands substantial training and continual practice on the part of the surgeon because it requires difficult techniques that are different from those of traditional surgical procedures. So we focused on a simulation technology based on the physical characteristics of organs as an intra-operative assistance for a surgeon. In this research, we proposed the development of surgical simulation, using a physical model, for intra-operative navigation. In this paper, we describe the design of our proposed system, in particular our organ deformation calculator. We performed two experiments with pig liver and silicone model to evaluate the accuracy of the calculator. We obtained adequate experimental results of a target node at a nearby point of interaction, because this point ensures better accuracy for our simulation model.

Radiofrequency Ablation (RFA) assisted liver resection results in lesser blood loss during liver resections. It involves a time consuming and error prone process of alternating coagulation and cutting until the line of transaction is completed to finally divide the liver. We proposed a new robotic mechanism to automate the ablation and division process. The robotic device comprises a 2-link manipulator, an x-y translator, a flexi-arm as well as a liver ablation and division device. The process of precise linear motion ablation and cutting has been integrated with the guidance of a robotic mechanism to uniformly coagulate and slit areas of the parenchymal. A prototype of the robotic device has been evaluated on its ability to uniformly target a large region.

This paper describes a fast image mapping system that integrates endoscopic image mosaics with three-dimensional (3-D) ultrasound images for assisting intrauterine treatment of twin-to-twin transfusion syndrome (TTTS) by laser photocoagulation. Endoscopic laser photocoagulation treatment has a good survival rate and a low complication rate for twins. However, the small field of view and lack of surrounding information makes the identification of vessels anastomosis difficult. We have developed an extended placenta visualization system with the fusion of endoscopic image mosaics with a 3-D ultrasound-image placenta model. Fully automatic and fast calibration is used for endoscope calibration in fluid. The 3-D spatial position of the endoscopic images and the ultrasound image are tracked by a 3-D position tracking device. The mosaiced endoscope images are registered to the surface of the 3-D ultrasound placenta model by using a fast GPU-based image rendering method. Experimental results show that the system may provide an improved and efficient way of planning and guidance in laser photocoagulation TTTS treatment.

In this paper, we proposed a novel non-rigid 2D-3D registration framework, which is based on Support Vector Regression (SVR) to compensate the disadvantages of generating large amounts of Digitally Rendered Radiographs (DRRs) in the stage of intra-operation for radiotherapy. It is successfully used to estimate similarity metric distribution from prior sparse target metric values against different featured transforming parameters of non-rigid registration. Through applying the appropriate selected features and kernel of SVR solution to our registration framework, experiments provide a precise registration result efficiently in order to assist radiologists locating the accurate positions of radiation surgery. Meanwhile, a medical diagnosis database is also built up to reduce the therapy cost and accelerate the procedure of radiotherapy in the case of future scheduling of multiple treatments.

This paper presents a real-time autostereoscopic visualization system using the principle of Integral Videography(IV). We develop MIP and composite volume ray casting method for IV volume rendering, and implemented the algorithm on GPU to achieve real-time rendering. The system was used to visualize 4D MR image that was generated from registration of 3D MR image and 4D ultrasound image. The registration scheme consists of inter-modality rigid registration between 3D MR image and 3D ultrasound image and intra-modality non-rigid registration between 3D ultrasound images. Registration processes were also implemented on GPU. Evaluation of processing speed showed that GPU processing time was 48x, 13x, 21x faster than CPU processing time for IV volume rendering, rigid registration, and non-rigid registration respectively. We also enabled real-time user interactivity for IV visualization system. In the future, We plan to use this system to develop intra-operative surgery navigation system for intra-cardiac surgery on beating heart.

Organ motion is one of the problems on augmented reality (AR) visualization for endoscopic surgical navigation system. However, the conventional optical and magnetic trackers are not suitable for tracking of internal organ motion. Recently, a wireless magnetic tracker, which is called the Calypso 4-D localization system has been developed. Since the sensor of the Calypso system is miniature and implantable, position of the internal organ can be measured directly. This paper describes AR system using the Calypso system and preliminary experiments to evaluate the AR system. We evaluated distortion error caused by the surgical instruments and misalignment error of superimposition. Results of the experiments shows potential feasibility and usefulness of AR visualization of moving organ using the Calypso system.

This paper describes a precision-guided surgical navigation system for minimally invasive surgery using fusion of laser guidance technique and three-dimensional (3-D) autostereoscopic image overlay technique. The images superimposed onto the patient are created by employing an animated autostereoscopic image called integral videography (IV), which display geometrically accurate 3-D autostereoscopic images and reproduce motion parallax without the need for special viewing or tracking devices. To improve the insertion accuracy of surgical instrument, we integrated the image overlay system with laser guidance for visualization of insertion point and orientation of the surgical instrument. We designed and manufactured a laser guidance device and mounted it to the IV image overlay device. Accuracy evaluations showed that the system could guide a linear surgical instrument toward a target with an average error of 2.48 mm and standard deviation of 1.76 mm. Improvement in the design of the laser guidance device and the patient-image registration of the IV image overlay will make this system practical and its use will increase surgical accuracy and reduce invasiveness.

This paper presents a method for an automated anatomical labeling of bronchial branches (ALBB) for augmented display of its result for bronchoscopy assistance. A method for automated ALBB plays an important role for realizing an augmented display of anatomical names of bronchial branches. The ALBB problem can be considered as a problem that each bronchial branch is classified into the bronchial name to which it belongs. Therefore, the proposed method constructs classifiers that output anatomical names of bronchial branches by employing the machine-learning approach. The proposed method consists of four steps: (a) extraction of bronchial tree structures from 3D CT datasets, (b) construction of classifiers using the multi-class AdaBoost technique, (c) automated classification of bronchial branches by using the constructed classifiers, and (d) an augmented display of anatomical names of bronchial branches. We applied the proposed method to 71 cases of 3D CT datasets. We evaluated the ALBB results by leave-one-out scheme. The experimental results showed that the proposed method could assign correct anatomical names to bronchial branches of 90.1% up to segmental lobe branches. Also, we confirmed that an augmented display of the ALBB results was quite useful to assist bronchoscopy.

MRI is now utilized not only for diagnosis but also for intraoperative surgical treatment. In the MRI environment, the surgeon imagines the position of target region by observing pre-acquired images and patient body during the operation, therefore the spatial position of the diseased part depends on the surgeon’s knowledge and experience. Thus information from the MRI image is not fully utilized. In order to solve this problem, we developed the prototypes of slice image overlay system for open-type MRI, which comprises the MRI compatible display device, image communication, registration system and visualization software, which is adopted to clinical 0.2T Open MRI. Phantom experiments were performed and the TRE of the overlay device was less than 1.0mm at the center of the display.

The extraction of curve skeletons from tubular networks is a necessary prerequisite for virtual endoscopy applications. We present an approach for curve skeleton extraction directly from gray value images that supersedes the need to deal with segmentations and skeletonizations. The approach uses properties of the Gradient Vector Flow to derive a tube-likeliness measure and a medialness measure. Their combination allows the detection of tubular structures and an extraction of their medial curves that stays centered also in cases where the structures are not tubular such as junctions or severe stenoses. We present results on clinical datasets and compare them to curve skeletons derived with different skeletonization approaches from high quality segmentations. Our approach achieves a high centerline accuracy and is computationally efficient by making use of a GPU based implementation of the Gradient Vector Flow.

The accurate segmentation of liver vessels is an important prerequisite for creating oncologic surgery planning tools as well as medical visualization applications. In this paper, a fully automatic approach is presented to quickly enhance and extract the vascular system of the liver from CT datasets. Our framework consists of three basic modules: vessel enhancement on the graphics processing unit (GPU), automatic vessel segmentation in the enhanced images and an option to verify and refine the obtained results. Tests on 20 clinical datasets of varying contrast quality and acquisition phase were carried out to evaluate the robustness of the automatic segmentation. In addition the presented GPU based method was tested against a CPU implementation to demonstrate the performance gain of using modern graphics hardware. Automatic segmentation using graphics hardware allows reliable and fast extraction of the hepatic vascular system and therefore has the potential to save time for oncologic surgery planning.

We previously developed a deformable model for segmenting lung fields in serial chest radiographs by using both population-based and patient-specific shape statistics, and obtained higher accuracy compared to other methods. However, this method uses an

ad hoc

way to evenly partition the boundary of lung fields into some short segments, in order to capture the patient-specific shape statistics from a small number of samples by principal component analysis (PCA). This

ad hoc

partition can lead to a segment including points with different amounts of longitudinal deformations, thus rendering it difficult to capture principal variations from a small number of samples using PCA. In this paper, we propose a learning technique to adaptively partition the boundary of lung fields into short segments according to the longitudinal deformations learned for each boundary point. Therefore, all points in the same short segment own similar longitudinal deformations and thus small variations within all longitudinal samples of a patient, which enables effective capture of patient-specific shape statistics by PCA. Experimental results show the improved performance of the proposed method in segmenting the lung fields from serial chest radiographs.

Automatic identification and extraction of bone contours from x-ray images is the first essential task for further medical image analysis. In this paper we propose a 3D statistical model based framework for the proximal femur contour extraction from calibrated x-ray images. The initialization is solved by an

Estimation of Bayesian Network Algorithm

to fit a multiple component geometrical model to the x-ray data. The contour extraction is accomplished by a non-rigid 2D/3D registration between a 3D statistical model and the x-ray images, in which bone contours are extracted by a graphical model based Bayesian inference. Our experimental results demonstrate its performance and efficacy even when part of the images are occluded.

In this paper, detection of edges in oriented fields is addressed. Haralick edge detection is an accurate scheme for estimation of the edge in a Euclidean space. However, in some applications such as edge detection for vessel segmentation because of the intrinsic orientation of structures, accuracy is only demanded in a particular subspace. This is specially usefull when a curve evolution is chosen for segmentation since gradients in parallel to vessel orientation stops evolution. Haralick edge detection is generalized on a Riemannian space using the inner product of the vectors under a space metric tensor. This eliminates the spurious gradients and preserves the accuracy on the vessel border. Examples are given and the comparison is made with the state-of-the-art flux maximizing flow indicating that significant improvements in terms of leakage minimization and thiner vessel delineation is achievable using our methodology.