Cut, overlap and locate: a deep learning approach for the 3D localization of particles in astigmatic optical setups

For the detection of overlapping particle images, we require an algorithm which is able to distinguish objects of the same class from each other, even if they share the same center point. Most modern object detection algorithms such as SSD (Liu et al. 2016), YOLO (Redmon et al. 2016) and Mask R-CNN (He et al. 2017) output multiple candidate object detections for the same object. Therefore, they rely on non-maximum suppression as a postprocessing step. When candidate object detections of the same object class are overlapping by more than a given threshold, non-maximum suppression only keeps the detection with the highest confidence score and removes all other detections. This renders models which rely on non-maximum suppression unsuitable for our goal of detecting particle images with the same center point location.

Instead, we employed an end-to-end trainable architecture for our purpose (Stewart et al. 2016). This approach uses a convolutional neural network (CNN) to extract image features within regions of the image. The features of each image region are fed separately into a long short-term memory network (LSTM) which translates image features into object detections. This end-to-end trainable model explicitly learns to produce one detection for each object and thus allows for the detection of arbitrary degrees of object overlap.

Tensorbox is an open-source implementation of the original algorithm. We adapted this model to predict out-of-plane locations of particles in addition to their in-plane locations (see Fig. 6). The model Inception V1 (Szegedy et al. 2015) is employed as the CNN (red bounding box in Fig. 6). In this CNN architecture, the input of the model first passes through several convolutional and pooling layers (yellow and purple boxes in Fig. 6) before inception modules (orange boxes in Fig. 6) are employed instead of the conventional convolutional layers. Inception modules consist of several convolutional and pooling operations which are all applied in parallel to the same input of this layer and are then concatenated together as output. \(1\times 1\) convolutional operations in this module reduce the computational budget and allow for a wide and deep neural network architecture. This allows the architecture to learn a wide range of complex image features.

Due to the convolutional and pooling operations, the output of the CNN is three-dimensional with a lower resolution in the image dimensions. The extracted image features are stored in feature vectors along the additional third dimension. The area in the input image that a feature vector is connected to is called the receptive field. In our case, the receptive field of the last inception module of the CNN has a size of \(789\times 789\) pixel. Therefore, each feature vector “sees” a large proportion of the input image. This information from a large receptive field is employed to account for large scale defocused background patterns from undetectable particles. However, the region size in which each feature vector is employed to detect particles is only \(32 \times 32\) pixel.

These feature vectors are fed into a LSTM (Hochreiter and Schmidhuber 1997) with four recurrent units and 800 memory states. The number of memory states determines the number of neurons involved in detecting particles and locating them in 3D based on information from the feature vectors. The number of recurrent units determines the number of outputs of an LSTM and therefore equates to the maximum number of particle image detections in each image region. Each recurrent unit computes horizontal, vertical and depth locations as well as a confidence score c. Linear interpolation of shallower feature vectors at the previously determined particle image locations yields additional information about precise particle locations. These shallow features are derived from the output of the first Inception module of the model and produce an offset vector which updates the particle locations and confidence scores. The feature interpolation step is the only part of the model where the computational cost is proportional to the number of particle images. Yet, it is independent of the number particle image overlaps. The average inference time for an image with 200 particle images was 1.02 s.

The CNN is initialized with weights, which were trained on the Imagenet dataset. The entire network is trained with 10,000 semi-synthetic images, each containing in average 200 particle images from a set of approximately 45,600 individual particle images. Adam (Kingma and Ba 2014) was employed as optimizer and a cyclic learning rate schedule followed by a decaying learning rate. During the initial cyclic period, the learning rate was set to alternate sinusoidal with a frequency of 20,000 parameter updates followed by an exponential decay of the learning rate from parameter update 30,000 to 100,000. We found that this alternation in the learning rate shortened training time and lowered the final error.

The trained model is first evaluated on a test set of 1000 semi-synthetic images to test the performance of the algorithm for the task of inferring the depth position from defocused particle images. The model predictions indicate that the model architecture is capable of simultaneously decoding both in-plane and depth positions from defocused particle images. Figure 7 shows the relationship between depth prediction and the true depth values. There is no visible bias in the depth inference along the depth axis, nor is there a visible deterioration in accuracy with distance from the midpoint between focal planes. The algorithm is capable of inferring particle depth positions up until the visibility limit and covers the same depth range as the human annotator of 408 \(\upmu \hbox {m}\). The model has a slightly higher variance around the regions of the two focal planes.

At the visibility limit of the defocused particle images at a distance of approximately 200 \(\upmu \hbox {m}\) from the midpoint between focal planes, the depth location error again increases. When the detected particle images are filtered by the their confidence score, it can be seen that low-confidence scores are clearly related to the particle images at the fringe of their visibility. This is where their signal is getting close to the signal of the artificial defocused background patterns. The defocused particle images are starting to merge into the image background which might be the cause of the lower confidence scores.

The overall rate of undetected particle images (false-negative detections) was found to be 2.6%, and the rate of false-positive detections was found to be 2.7%. Undetected particle images are clearly related to the presence of particle image overlaps (see Fig. 8). The closer a neighboring particle image is to the center location of a particle image, the higher the rate of undetected particle images. The proximity of false-positive detections to real particle images reveals that most false-positive detections correspond to particle images which are interpreted as two particle images. The equal performance with regard to false-positive and false-negative detections indicates that the initially chosen fraction of 50% cropped particle image overlap resulted in a well-balanced performance of both the detection of single particle images and the distinction of overlapping particle images.

The objective might change for experiments with significantly larger or smaller particle image densities which would result in different numbers of overlaps. For high particle image densities, it could be more important to distinguish between overlapping particle images or for smaller particle image densities it could be more important to reduce false-positive predictions. It might be possible to address such a change in the objective by changing the ratio of cropped particle image overlaps in the training set. This might shift the attention of the algorithm during model training towards the one or the other task. However, such optimization of the proposed technique to different experimental conditions such as particle image density still requires further investigation.

In our semi-synthetic dataset, we are able to compute the true distribution of the relative locations of the closest neighboring particle image. The share of undetected particle images to this distribution provides us with the rate of undetected particle images as a function of the relative location of the closest neighbor. Again, the maximum of this rate is located at the points where particle images share the same center point locations. Here, the rate of undetected particle images is 20.7%. At a center-point distance of 10 pixels and 15 pixels, this value drops to 7.8 and 1.3%. False-positive detections are mainly located within a radius of 4 pixels of a real particle image center.

We further analyzed the impact of neighboring particle image proximity on the performance of the algorithm to infer in-plane and depth locations. For this purpose, we filtered all detected particle images with respect to their distance to their nearest neighboring particle image. Figure 9 shows the location error in the image plane and along the depth axis as a function of the location of the nearest neighbor. The average in-plane location errors of particle images without overlap are 0.65 and 0.57 \(\upmu \hbox {m}\) in the x and y directions, respectively. Location errors start increasing sharply if neighboring particle images are within a distance of less than approximately 10 pixels. When particle image centers share the same in-plane location, x and y location errors almost triple. This effect is even more pronounced for the depth location error. The average depth location error is 6.34 \(\upmu \hbox {m}\) for particle images without overlap. When the center points of overlapping particle images are the same, the depth location error increases by a factor of 6. Thus, particle image overlap has a stronger relative impact on the performance of determining the depth position than the in-plane locations. It appears that particle image features which relate to the in-plane locations are in comparison easier for the algorithm to decode during particle image overlap as compared to particle image features which relate to the depth position. Both x and y location errors are slightly higher for neighboring particle image locations along the x and y image axis, respectively. This could be explained with scenes where superimposed particle images are on the same side of the midpoint between focal planes and their elliptical shapes line up on the x or y axis. In this arrangement, their signals are more convoluted and the solution for separating the two signals might become less unique. The total location error is entirely dominated by the location error of the depth error. The sum of the x and y location errors only contributes to approximately 10% of the total location error. It is in accordance with previous works that the depth error is an order of magnitude higher than the in-plane errors (Barnkob et al. 2015; Franchini et al. 2019).

To test the resemblance of the semi-synthetic dataset with real images, we analyzed the performance of the trained model on videos of real images with moving spherical microspheres. We acquired an additional dataset based on the previously described experimental setup with the same fluid, particles and particle density. Image sequences of the particles are captured at 21 different depth locations 150 \(\upmu \hbox {m}\) apart from each other throughout the entire depth of the channel. At each depth location, a sequence of 452 images was acquired at a frame rate of 24 fps. Polystyrene tracer particles were added to the glycerol at a concentration of \(5\times 10\) w/v. Approximately 200 particle images are detectable with human vision on the \(1440\times 2560\) pixel camera images. The resulting particle image density of \(5.4\times 10^{-5}\) particles per pixel is the same as in the semi-synthetic training images. The particle image diameters in the x and y directions vary between approximately 7 and and almost 70 pixel depending on the depth position of the particle.

In this experiment, we applied a constant flow rate of 2.3 ml/h to the liquid which is carrying the tracer particles. The liquid is flowing in the rectangular channel approximately parallel to the midpoint between focal planes. The constant flow rate ensures that the particles are moving in a laminar flow on straight trajectories. Therefore, the three-dimensional displacements of particles are constant over time in this flow field.

These constant displacements are an ideal basis for evaluating the variance of the predictions of our trained model. We employ the model on all acquired image sequences and link the detected 3D locations of the particles with the Hungarian algorithm. This algorithm only links particle detections between consecutive frames. More advanced tracking algorithms take information about the flow field into consideration and might perform better in more complex flows (Pereira et al. 2006). All resulting trajectories below a length of 100 frames were discarded to ensure that trajectory-wise measured error quantities are well-estimated. 38.3% of all recovered trajectories contained a particle image overlap at some point in time. Due to the varying particle image shapes, we defined a particle image overlap as a neighboring particle image center being closer than 35 pixel along the x or y axis to another particle image center. Thus, particle images might not always be superimposed in the fraction of trajectories which were identified as containing a particle overlap at some point in time.

Some trajectories were a result of false-positive predictions. In such cases, one particle was consistently detected twice and can be filtered out. To test whether two close trajectories are the result of different physical particles, the average relative displacements between the two trajectories are compared. Two distinctive particles travel at different velocities if they are located at different depths. Double predictions of the same particle can only move at the same velocity since they are mostly located within a 5 pixel radius (see Fig. 8). Thus, false-positive trajectories can be filtered out by setting a minimum value for the average relative displacement between a particle and its closest neighbor. The threshold is only applied to trajectories with neighbors which are closer than a certain limit. This process efficiently filters out cases where a particle is consistently detected twice. This filtering approach removes particles with accompanied false-positive detections which both have the same trajectories along the \(x{-}y\) plane. Therefore, we expect this filtering procedure to be robust in more complex flows too, where trajectories along the \(x{-}y\) plane differ even stronger than in this study.

Individual displacements along the X axis were derived from all trajectories and visualized (see Fig. 10, left). The displacements in the direction of the X axis do not vary significantly along the Y axis. This is due to the fact that only the centermost 1152 \(\upmu \hbox {m}\) of a 7-mm-wide channel has been investigated. The effects of the walls along the Y axis on the flow field are therefore not prominent in the measured volume. The periodic change in displacement variance along the depth axis (Fig. 10, right) shows that within each individual measurement slice, the error of the measured displacement changes significantly along the depth axis.

To estimate displacement errors, we first approximated the constant displacements of particles by calculating the median displacements along trajectories. The residuals between those averages and the individual measured displacements allow us to study the variance of the model predictions from real images without the need for a fully annotated dataset. We did not observe any significant bias in the location errors of our model with respect to the semi-synthetic dataset (see for instance Fig. 7 for depth error). Thus, we assume that the variance of the model predictions from the real images is a good approximation for the total location error (Cierpka et al. 2011).

Figure 11 shows how location errors of each trajectory relate to the average depth prediction of the particle. The in-plane location errors in the x and y direction show a very similar behavior. Close to the midpoint between focal planes, their error reaches a minimum of approximately 0.3 \(\upmu \hbox {m}\), and towards the upper and lower limit of particle detectability, the error increases up to approximately ten times higher values. It can be seen that the x error increases more strongly towards small depth values and that the y error increases more strongly towards large depth values. This can be explained with the shape of the particle images at those depth positions. For small depth values, the image of a microsphere forms an ellipse, with its long axis aligned along the x axis (see Fig. 2). For such a particle image, the image gradient is larger along the y axis than the x axis. It is therefore easier to locate the center location of the particle image along the y axis than the x axis for low depth values. The opposite is true for large depth values, where the long axis of the particle image is aligned with the y axis of the image and the y location error is larger.

Previously reported in-plane location errors from the fitting of 2D generalized Gaussian distributions to particle images are 0.21 pixels for images with the same signal-to-noise ratios (Franchini et al. 2019). This refers to non-overlapping and “slightly” overlapping particle images. The error corresponds to 0.17 \(\upmu \hbox {m}\) by taking the magnification of our apparatus into account. Our minimum in-plane errors of 0.3 \(\upmu \hbox {m}\) are therefore 76% larger as compared to the current method of locating tracer particles in the image plane. The sum of the differences between in-plane location errors of the two methods contributes to approximately 2% of the average total location error. Depending on the application, this additional error might not be significant for a user.

The depth location error in our study shows a less clear trend along depth axis. It reaches minimum values of approximately 6 \(\upmu \hbox {m}\) where the masked signal reaches its maximum value and there are several regions with local peaks in depth error. Focal plane \({F}_{{yz}}\) correlates with such a local error peak. Focal plane \({F}_{{xz}}\) corresponds to higher error values; however, it does not fall directly on local error peak. This is consistent with the performance of the algorithm of Cierpka et al. (2011), which showed a local error peak close to one focal plane and another local error peak in the vicinity of the second focal plane. Maximum displacement error values of approximately 15 \(\upmu \hbox {m}\) are reached at the detectability limit. The relative increase in error with distance to the depth center of the measurement volume is therefore not as strong as in the case of the in-plane location errors. Yet, the number of outliers which experience larger errors is significantly larger for the depth location error.

Previous studies reported similar ratios of in-plane to depth location errors with ratios ranging between 1:10 and 1:30 (Cierpka et al. 2010, 2011; Barnkob et al. 2015; Franchini et al. 2019). Location errors strongly depend on the signal-to-noise ratio of the experimental apparatus (Cierpka et al. 2011). A study with images of the same signal-to-noise ratio reported a depth location error of 7.1 \(\upmu \hbox {m}\) with regard to their calibration dataset of non-overlapping particle images (Franchini et al. 2019). Their calibration dataset serves the same purpose as our semi-synthetic dataset: to train a model to predict particle locations. In our study, we achieved a depth location error of 6.3 \(\upmu \hbox {m}\) in our synthetic dataset. The average depth error of non-overlapping trajectories in our real images is 9.2 \(\upmu \hbox {m}\) over a 67% larger depth range than (Franchini et al. 2019). The difference in the errors between the semi-synthetic dataset and the real dataset of our study might be explained with a slight difference in the distributions of the data in the two dataset. The semi-synthetic dataset might include image features which are not present in the real data and vice versa. The same might be true for the stated depth location error with respect to the calibration dataset in Franchini et al. (2019). We therefore consider our measured depth location error to be of a comparable level of accuracy.

The depth range in which the trained algorithm is able to recover particle trajectories spans 373 \(\upmu \hbox {m}\) (see Fig. 11). This covers 91.4% of the depth range in which a human annotator was able to detect particle images in real experimental images. This translates into an increase in the depth range along which particles can be detected of 67 % as compared to detection by simple smoothing and thresholding (Franchini et al. 2019). This indicates that the semi-synthetic dataset was successful in representing the differences between defocused background patterns and low-intensity particle images. The performance of the employed neural network is therefore close to human performance in detecting low-signal particle images.

Location errors as a function of the closest neighboring particle image show a similar behavior as in the synthetic dataset (see Figs. 9, 12); however, there is clearly more noise in the error values of the experimental dataset. This is most likely due to a smaller number of sample points. For all location errors, there is a clear trend towards higher errors for distances to the nearest neighbor of less than approximately 10 pixel. The error values at larger distances to the particle image center point are comparable to the values in the synthetic dataset. In this region, average in-plane location errors range between 0.5 and 1.0 \(\upmu \hbox {m}\), whereas z-location errors range between 6 and 11 \(\upmu \hbox {m}\). The total location error approximately doubles to a maximum total location of 14.7 \(\upmu \hbox {m}\) when two neighboring particle images share the same center point. For neighbor locations which share the same particle image center, all location errors are below the errors of the synthetic dataset. This shows that the model is capable of detecting overlapping particle images. Yet, more challenging particle image overlaps which caused higher errors in the synthetic dataset are possibly not detected in the real images.