2.1.

Setup

Our HiLo endomicroscope setup is illustrated in Fig. 1. A single-axis galvanometer mirror (Thorlabs GVS001) directs a laser beam (Omicron PhoxX 488, 80 mW) between two illumination paths. In the uniform path the beam travels through a $10\times$ expanding telescope and polarizing beam splitter and is focused onto the back aperture of a $20\times$ objective (Olympus Plan Achromat, 0.4 NA). This uniformly illuminates the proximal face of a flexible imaging fiber bundle (600 μm active dia; 30,000 cores). The distal face of the fiber bundle is equipped with a $2.5\times$ water-immersion micro-objective (Mauna Kea Technologies; 0.8 NA) to give additional magnification and working distance of 60 μm. The fiber bundle is 3 meters long with the distal tip 2.7 mm in diameter and 14 mm in length. The generated fluorescence from the sample is relayed through the fiber bundle, spectrally separated with a dichromatic mirror (Semrock FF506-Di03-25x36) and emission filter (Semrock FF01-536/40-25) and recorded with a CCD camera (PCO Pixelfly USB; binning $2\times 2$; double-shutter mode). The structured path is identical to the first with the addition of a transmission Ronchi ruling (Edmund Optics; $50\mathrm{lp}/\mathrm{mm}$) in a plane conjugate to the objective and micro-objective focal planes. The image of the Ronchi ruling is projected into the sample, modulating the sample fluorescence with a high-contrast grid pattern.

Fig. 1

(a) HiLo endomicroscope setup. A galvanometer mirror (GM) directs a laser beam between two paths, one delivering uniform illumination and the other projecting the image of a Ronchi ruling (RR) into the sample. The paths are combined with a polarizing beam splitter (PBS). A flexible imaging fiber bundle (FB) is equipped with a micro-objective ($\mu O$) for additional magnification and working distance. Generated fluorescence is spectrally separated by a dichromatic mirror (DM) and emission filter ($\mathrm{F}$), and images are recorded with a digital camera (CCD). (b) Representative uniform and (c) structured images.

2.2.

Image Preprocessing

Several difficulties are associated with the use of a fiber bundle for fluorescence imaging. The core fill factor of our fiber bundle is only about 30%, meaning our raw images are sparsely sampled. This sampling is irregular and imparts a quasi-hexagonal honeycomb pattern to the raw images with contrast typically much larger than the sample contrast of interest. Moreover, the fiber cores are heterogeneous in size and thus transmission efficiency. Finally, the fiber core material is autofluorescent, which introduces an extraneous background in our raw images. These problems were found to significantly undermine the quality of our final HiLo images and needed to be addressed.

Several groups have suppressed the appearance of fiber cores by applying Gaussian blurring filters,² disk filters,¹³ or spatial frequency domain filters.^14–17 Glass fiber bundles have perfectly periodic hexagonal sampling that concentrates the core pattern in a few localized regions in spatial frequency space, making frequency domain filtering an attractive option. Quartz fiber bundles like those used in this work are distinctly aperiodic and are less amenable to spectral domain techniques. Histogram equalization can be used in conjunction with simple spatial filtering to enhance image contrast for visualization purposes,¹⁸ but the resulting images are no longer proportional to fluorescence concentration (i.e. are no longer quantitative). The main drawback of simple blurring is loss of resolution and image contrast since relatively wide kernels are required to sufficiently attenuate the core pattern. Also, due to core sampling sparsity, many of the camera pixels do not measure intensity from the sample, but rather from autofluorescence or extraneous reflections. A method proposed by Perchant and Le Goualher involves finding all 30,000 core centroids with a segmentation process, measuring the maximum intensity from each core, then interpolating the scattered data on a uniformly sampled rectilinear grid.^1,19,20 This process is most comprehensive, but is computationally intensive and difficult to achieve in real time.

We introduce here a simple preprocessing routine that corrects for sparse sampling, core heterogeneity, and autofluorescence. This routine is based on an iterative segmentation-interpolation algorithm that removes the appearance of the cores without sacrificing spatial resolution or sample contrast. We begin our iterative procedure with the raw camera image itself, $I^{(0)}(\overrightarrow{\rho })$, which serves as the initial estimate of a core-suppressed image $I_{\mathrm{cs}}(\overrightarrow{\rho })$. For each iteration, a binary core mask, $M^{(n)}(\overrightarrow{\rho })$, is generated to distinguish core pixels from cladding pixels. This is done by setting a local, spatially varying threshold level determined by low-pass filtering the current estimate. Camera pixels above this threshold level are identified as core pixels because the cores are almost universally brighter than the cladding. Similarly, the cladding pixels are identified by the complementary mask $1-M^{(n)}(\overrightarrow{\rho })$. To obtain an improved estimate, $I_{cs}^{(n+1)}(\overrightarrow{\rho })$, the cladding pixels are then assigned the low-pass filtered values while the core pixels retain their original values. The full width at half maximum (FWHM) of the Gaussian low-pass filter kernel is approximately one inter-core distance. This simple segmentation-interpolation procedure is iterated until a desired smoothness is achieved; typically two to five iterations suffice. We emphasize here that the low-pass filter used in this procedure is highly local and averages only over nearest neighbor cores. The core pixels thus retain their original intensity throughout the process, while the cladding pixels are filled in to approach the average of their nearest neighbor cores. The algorithm features the low complexity and fast execution time of Gaussian blurring, and minimal loss of sample resolution and contrast. However, it should not be regarded as simple filtering since the procedure is neither linear nor spatially invariant. The algorithm is summarized below.

The next steps in our preprocessing routine involve subtracting the contributions from autofluorescence and correcting for heterogeneities in fiber core efficiency. We follow a modified procedure laid out by Perchant and Zhong.^20,21 Three calibration images are acquired before imaging begins. The first, $I_{\mathrm{bg}}$, is acquired with the laser turned off while imaging a nonfluorescent blank solution, such as water, in a dark compartment. This provides a measurement of camera bias and diffuse room light that passes the emission filter. The second, $I_{\mathrm{af}}$, is acquired under the same conditions but with the laser turned on, providing an estimate of the autofluorescence of the fiber core and cladding material. Since the autofluorescence is a function of laser power while the bias and room light are not, the two images need to be acquired separately. The third image, $I_{\mathrm{hf}}$, is acquired while imaging a homogeneously labeled fluorescent fluid. When camera bias, room light, and autofluorescence signals are subtracted out, this third image provides a mapping of the core transmission and collection efficiencies. In practice, each calibration image is derived from averaging many ($\sim 100$) raw images to improve signal-to-noise ratio. Calibration images are reacquired each time the fiber bundle probe is connected and aligned to the system.

Image preprocessing can be summarized as a two-step process. First, raw images are subject to iterative segmentation-interpolation to fill in the cladding pixels, yielding $I_{\mathrm{cs}}(\overrightarrow{\rho })$. Background subtraction and normalization then yield the preprocessed image

2.3.

HiLo Principle

The benefit of HiLo over standard widefield imaging is that it provides out-of-focus background rejection. As described in previous work, the uniform and structured images can be roughly decomposed into in-focus and out-of-focus components.

The final HiLo image is axially resolved across the entire spatial frequency spectrum, from dc to the bandwidth of our imaging optics. The lateral resolution, $\mathrm{\Delta }\rho$, is limited by the sampling of the fiber cores and is given by $2d_{\mathrm{ic}}/M_{\mu O}$, where $d_{\mathrm{ic}}$ is the average fiber inter-core separation, and $M_{\mu O}$ is the micro-objective magnification. The axial resolution of the low-frequency component, $I_{\text{low}}(\overrightarrow{\rho })$, can be approximated by considering a thin fluorescent plane and calculating the decay of the grid contrast as a function of defocus. The contrast of the grid pattern is attenuated by both the illumination and detection optical transfer functions (OTFs), and the axial resolution is defined as the FWHM of this function. Using the Stokseth approximation²⁵ for the 3D OTF, assuming similar excitation and detection wavelengths, and assuming ${\kappa }_g$ is much smaller than the imaging bandwidth, the axial resolution is given by $\mathrm{\Delta }z\approx 0.54{({\kappa }_g\mathrm{NA})}^{-1}$. The transmission of low frequencies is found to decay as ${|z|}^{-3}$ for large defocus $|z|$. In comparison, the axial resolution of the high-frequency component at the filter cutoff frequency ${\kappa }_c$ is given by $\mathrm{\Delta }z\approx 0.68{({\kappa }_c\mathrm{NA})}^{-1}$. This is slightly broader due to the action of only the detection OTF providing axial confinement and is found to decay with defocus as ${|z|}^{-3/2}$ for large $|z|$. A matching of the axial resolution at the transition between high- and low-frequency components (i.e. at ${\kappa }_c$) involves selecting the cutoff frequency appropriately. We have found that choosing ${\kappa }_c$ to be slightly lower than ${\kappa }_g$ provides a reasonably accurate match while still maintaining a reduced artifact in the low channel found at twice the grid frequency. The lateral and axial resolutions of the current system are approximately $\mathrm{\Delta }\rho \approx 2.6\mathrm{\mu m}$ (governed by the Nyquist sampling criterion associated with the fiber cores) and $\mathrm{\Delta }z\approx 17\mathrm{\mu m}$ (governed by the structured illumination grid period).

2.4.

Fast Image Processing with a GPU

Our image processing routines were developed and tested in MATLAB (MathWorks). The processing speed was found to be unacceptably slow for real-time processing ($<1\mathrm{Hz}$). To enable real-time processing and display, a library of image management and manipulation functions (memory allocation and deallocation, arithmetic operations, filtering in Fourier space) were written in CUDA-C and compiled to a graphics processing unit (GPU) target (NVIDIA GeForce GTX 280). The library was compiled as a dynamic linked library for maximum utility. A LabVIEW interface (National Instruments) was created to coordinate communication among the camera, CPU memory, and GPU memory. The multicore processor architecture (Intel i7 quad core) and inherent parallelism of LabVIEW greatly improved the speed of the endomicroscope. The entire processing pipeline—preprocessing, HiLo processing, and memory transfers between the CPU and GPU—occurs in 30 ms for an image resolution $520\times 696\text{pixels}$. The endomicroscope is thus capable of real-time image processing at speeds up to $\sim 35\mathrm{Hz}$.

3.1.

Chick Embryo Preparation

Fertilized eggs of the Silkie chick (Gallus gallus) were obtained from a local farm (Golden Egg Farm; Hardwick, MA) and incubated at 38°C and 80% humidity until approximately embryonated day E5. Imaging of the embryo was performed through a small hole in the egg shell. A fluorescent solution (50 mM fluorescein in ${\mathrm{H}}_2\mathrm{O}$) was injected into the amnion and allowed to diffuse for approximately 1 hr before imaging. For some samples, the solution was injected intravenously whereupon it bound to serum albumin and diffused through the embryo. After imaging the embryos were euthanized by hypothermia by storing the eggs at $-15°\mathrm{C}$.

A video of a thin fluorescently-labeled membrane floating through focus provides a demonstration of our sectioning capacity. The raw uniform image directly acquired by the camera [Fig. 3(a)] is included to illustrate the deleterious effects of the fiber cores. The preprocessed uniform image [Fig. 3(b)] shows a marked improvement in sample contrast and features small in-focus structures with full resolution. However, this image also features considerable out-of-focus background haze. The final HiLo image [Fig. 3(c)] preserves the in-focus structures while rejecting this out-of-focus background haze.

Fig. 3

Thin fluorescently labeled membrane floating through focus. (a) Raw, (b) preprocessed, and (c) HiLo result. The HiLo video preserves the small, bright, in-focus objects with full resolution while rejecting the out-of-focus background. (Video 1, MPEG, 7.9 MB). [URL: http://dx.doi.org/10.1117/1.JBO.17.2.021105.1]

Figure 4 shows blood vessels and other features of the CAM membrane after intravenous injection of fluorescein. Figure 5 shows a torrent of red blood cells (RBCs) passing through a CAM sinus and later collecting into a venule. The RBCs do not take up the fluorescein stain and appear as shadows on a bright background. These videos illustrate the capacity of our HiLo endomicroscope to capture fast dynamics.

Fig. 4

CAM membrane after intravenous injection of a fluorescein solution. (a) Raw, (b) preprocessed, and (c) HiLo result. The HiLo video shows enhanced contrast of blood vessels and RBCs. (Video 2, MPEG, 11.5 MB). [URL: http://dx.doi.org/10.1117/1.JBO.17.2.021105.2]

Fig. 5

Torrent of RBCs flowing through a CAM sinus and collecting in a venule. (a) Raw, (b) preprocessed, and (c) HiLo result. The HiLo video demonstrates the capacity of the endomicroscope to capture fast dynamics with minimal motion artifacts. (Video 3, MPEG, 10.0 MB). [URL: http://dx.doi.org/10.1117/1.JBO.17.2.021105.3]

3.2.

Imaging Speed and Motion Artifacts

As observed in Fig. 5, our HiLo endomicroscope is largely immune to artifacts arising from sample motion (e.g. respiration, blood flow) or probe motion. Part of the reason for this immunity comes from the fact that all the high-frequency information is derived from a single image ($I_u(\overrightarrow{\rho })$). On the other hand, the low-frequency information is more susceptible to motion artifacts since it is derived from a comparison of two serially acquired images. While the consequence of intra-frame motion during the camera integration time is loss of spatial resolution in the raw images, a much more pronounced motion artifact arises from inter-frame motion when significant translation occurs between the two raw image exposures. The magnitude of this artifact can be roughly estimated by considering two serially acquired images of a sharp edge, both under uniform illumination. In the absence of sample motion, both images are the same, and we have $C(\overrightarrow{\rho })=I_{\text{low}}(\overrightarrow{\rho })=0$ [see Eq. (4)]. If instead the sample translates laterally between images, an artifactual signal occurs in the neighborhood of the edge that appears to be in focus. The trapezoidal profile of this signal can be approximated as a box-top function of unity height and width $\upsilon ({\tau }_{\exp }+{\tau }_{\mathrm{if}})$, where $\upsilon$ is the translation velocity (assumed constant), ${\tau }_{\exp }$ is the exposure time and ${\tau }_{\mathrm{if}}$ is the inter-frame delay. The low-pass filtering that occurs to generate $I_{\text{low}}(\overrightarrow{\rho })$ serves to spatially distribute this artifact, reducing the maximum error at a given pixel. For small $\upsilon$, the error bound is $E_{\max }\propto {\kappa }_c\upsilon ({\tau }_{\exp }+{\tau }_{\text{if}})$, where ${\kappa }_c$ is the low-pass filter cutoff frequency. Under typical imaging conditions ${\tau }_{\mathrm{if}}\gg {\tau }_{\exp }$ and the error is governed by the camera readout time.

To minimize ${\tau }_{\mathrm{if}}$, we made use of a double-shutter camera (PCO Pixelfly USB). Such a camera acquires images pairwise with an inter-frame delay of only $\sim 5\mathrm{\mu s}$, followed by a paired readout. While the net frame rate for image pairs (i.e. the HiLo frame rate) is the same with a double-shutter camera as with a traditional camera, the key benefit is the reduction in ${\tau }_{\text{if}}$, which greatly suppresses $E_{\max }$. We note that in our case ${\tau }_{\text{if}}$ was limited to a minimum $\approx 1\mathrm{ms}$ by the need to switch illumination paths with the galvanometer mirror. In principle, two dedicated, electronically shuttered illumination sources could be used to overcome this limitation, though with diminishing returns when ${\tau }_{\mathrm{if}}\ll {\tau }_{\exp }$. In practice, motion artifacts are minimal when displacements are kept below $\sim {(2{\kappa }_g)}^{-1}$.

To illustrate the improved immunity to motion artifacts obtained with a double-shutter camera, we compare HiLo videos acquired with double-shutter versus traditional timing (see Fig. 6). The videos share the same raw data actually acquired in double-shutter mode, and traditional camera timing was simulated here by staggering the raw data (i.e. using uniform and structured raw images from neighboring frame pairs instead of the same frame pair). The effective inter-frame delay for the traditional camera [Fig. 6(a)] was 106 ms, while the double-shutter camera [Fig. 6(b)] was 1 ms. Motion artifacts arising from the misregistration of raw images are easily identified as bright low-frequency blobs in the neighborhood of moving RBCs. The reduced inter-frame delay from the double-shutter timing manifestly attenuates these artifacts and provides crisper imaging. This demonstration is somewhat exaggerated as scientific camera readout times are typically faster than 100 ms. Nevertheless, it serves as an illustration of motion artifact reduction.

Fig. 6

Comparison of HiLo videos acquired with traditional versus double-shutter camera timing. (a) HiLo with traditional timing, (b) HiLo with double-shutter timing, representing a 100-fold reduction in inter-frame delay over traditional camera timing. Motion artifacts arising from raw image misregistration are observed in (a) as bright low-frequency blobs in the neighborhood of moving RBCs. (Video 4, MPEG, 10.0 MB). [URL: http://dx.doi.org/10.1117/1.JBO.17.2.021105.4]