A contribution to 3D tracking of deformable bubbles in swarms using temporal information

Dispersed bubble-laden flows are heavily involved in chemical, biological, or environmental applications (Mathai et al. 2020; Schlüter et al. 2021; Lohse 2022). Tracking the motion of the dispersed phase, i.e. bubbles in 3D, is vital to advancing our understanding of their rise velocity, acceleration, collective behaviour, and relative dispersion.

Experimentally, the use of optical measurements to track such finite-sized objects is quite common due to their non-intrusiveness and high spatio-temporal resolution (Thoroddsen et al. 2008). Such 3D tracking of a single bubble—the simplest case—has been attempted before to study hydrodynamic bubble forces (Salibindla et al. 2020; Hessenkemper et al. 2020), the bubble rise path (Estepa-Cantero et al. 2024; Ma et al. 2023a), their shape oscillations (De Vries et al. 2002; Veldhuis et al. 2008), and bubble-particle relative motion (Sommer et al. 2024).

However, detecting and tracking individual bubbles in a bubble swarm becomes increasingly difficult as the bubble concentration grows, since the bubbles can overlap and occlude each other in the recorded images. Before we discuss the experimental challenges in 3D, we first give an overview of the main developments in bubble tracking in 2D. In general, tracking bubbles in a swarm has two main steps: first, detecting the bubbles and obtaining their positional information, and then linking the bubbles in adjacent frames. A common way to detect bubbles is to apply image processing techniques, e.g. Hough transform method (Hosokawa et al. 2009), breakpoint method (Honkanen 2009), and watershed method (Lau et al. 2013; Karn et al. 2015). These methods have good accuracy for dilute bubbly flows or fixed shaped bubbles. For dense bubbly flows with deformable bubbles, deep learning-based image processing tools have proven their capabilities under various scenarios (Haas et al. 2020; Kim and Park 2021; Hessenkemper et al. 2022; Ma et al. 2022).

Following the detection of bubbles, bubble tracks are established by linking their locations in adjacent time steps. A number of studies in the field of 2D bubble tracking have employed a strategy that is based on the nearest neighbour criterion, which constitutes a widely used approach within the field of particle tracking velocimetry (PTV) (Bröder and Sommerfeld 2002; Cierpka et al. 2013; Cerqueira et al. 2018; Schäfer et al. 2019). In addition, some works also attempted to enhance the reliability of tracking by incorporating bubble features (e.g. bubble similarity in two consecutive frames) as an additional criterion (Akhmetbekov et al. 2010; Acuña and Finch 2010). However, tracking becomes more challenging when multiple bubbles are detected in close proximity, which led to attempts at tracking bubbles in swarms by using machine learning-based 2D bubble tracking method. Wen et al. (2022) developed a 2D bubble tracking method that combines Convolutional Neural Network (CNN) models with the Kalman filter, enabling the simultaneous calculation of the velocity of multiple overlapping bubbles. However, the geometric characteristics of the bubbles are estimated using rectangular bounding boxes, which reduces the computational effort while concomitantly decreasing the accuracy of the bubble shape estimation. In a further development, Ma et al. (2023b) presented another 2D bubble tracking method that relies on a CNN-based bubble detector (Hessenkemper et al. 2022), which is capable of extracting the important geometrical properties of detected bubbles. In particular, this approach integrates a graph-based tracking formalism combined with several multi-layer perceptrons (MLPs) to enable the tracking of bubbles that are even completely occluded over multiple time steps.

Tracking bubbles in 3D typically requires the use of a multi-camera system. To make use of such systems in bubbly flows, additional effort is required to match a large amount of visual information across different perspectives and between sequential time steps. Since PTV techniques have been successfully extended from 2D to 3D and have gained substantial advancements in tracking individual tracer particles in space over the past few decades (Raffel et al. 2018), some studies have started to develop methods capable of tracking bubbles in 3D by drawing on the development of PTV. Xue et al. (2013) presented a virtual stereo and motion matching method to reconstruct 3D bubble trajectories. Mathai et al. (2018) investigated the dynamics of mixing in turbulent bubbly flows by treating bubbles as point particles. Kim et al. (2022) used the Shake-The-Box (Schanz et al. 2016) approach, which is an advanced framework for Lagrangian Particle Tracking (LPT) of tracer particles in densely seeded flows, to perform simultaneous 3D tracking of bubbles and tracer particles in a bubbly jet flow. However, these are limited to very dilute bubbly flows. Recently, a method for 3D Lagrangian tracking of very dense bubbles was introduced by Tan et al. (2023). This method, which incorporates the general idea of the tracking-then-triangulation procedure, is inspired by the Shake-The-Box approach. For different flow scenarios, this approach demonstrates a reliable tracking capability. However, the study by Tan et al. (2023) focuses mainly on fixed shaped bubbles, as the shaking step in this method requires reference bubble images.

More recently, we introduced a deep learning-based 3D tracking workflow to track individual deformable bubbles in bubble swarms (Hessenkemper et al. 2024). This approach adopts the common LPT strategy with the Wiener filter to predict the potential bubble position in upcoming frames, limiting matching candidates to a defined search range. To resolve ambiguities between deformable bubbles detected in different views, we applied a Siamese Neural Network (SNN) (Chicco 2021) to evaluate the similarity of bubble appearance between different views when multiple matching candidates are present. This method showed overall good performance in diverse flow scenarios. Nevertheless, in cases with higher bubble image densities, i.e. the presence of high overlap ratios or complete occlusion, the tracking performance deteriorates.

In this study, we enhance our previously developed 3D bubble tracking method by incorporating temporal information from established 2D tracks, allowing for more robust and accurate 3D trajectory reconstruction. Section 2 lays out the detail of the enhancement process. In Sect. 3, we demonstrate the validity of our extended 3D tracking method by calculating and comparing relevant metrics on the same artificial sequences as in our previous work. Finally, we discuss the shortcomings of our new method, including some possible solutions.

A bubble moving in 3D volume will be represented as a 2D track in an image sequence. Multiple perspectives can offer different 2D representations of the bubble motion. Figure 1a shows an example of a 3D bubble trajectory recorded by a 4-camera system. Ouellette et al. (2006) mentioned that for 3D-LPT at least 3 cameras are needed in order to reduce matching ambiguities. In this work, we consider setups with three cameras or four cameras.

Full size image

In order to retrieve the 3D bubble trajectories in a swarm, the most basic step is to detect bubble images in different views. Consider a 3D bubble swarm captured by four geometrically calibrated and temporally synchronized cameras with different viewing directions, as illustrated in Fig. 1. The 2D bubble detector detects bubbles at each frame, i.e. time step, represented as \(f \in \{0,1,2,....,F\}\), with the bubble centroid denoted as \(B_{f}^{v,i}\), indicating a bubble with the i-th ID in the v-th view at the f-th frame, in this case \(v \in \{1,2,3,4\}\) and \(i \in \{1,2,....,n\}\).

Based on bubble information from different views, we consider strategies for reconstructing bubble trajectories in 3D space: In our previous work, we followed the triangulation-then-tracking strategy; one of the main steps is the search for suitable candidates across views, which is performed by establishing multi-camera correspondences based on the projected lines technique (i.e. the double epipolar lines technique, see Fig. 2a) and defining a search threshold for matching the correct bubbles in different views. To further increase the matching reliability, an additional Siamese Neural Network (SNN) is introduced, which incorporates bubble appearance information to compute similarity scores between a bubble and its candidates matched in different views. A position in space is formed when more than three bubbles fulfil the one-to-one correspondence across views (i.e. unique cross-view match), and then tracking in space is performed. For a more detailed description, see our previous work (Hessenkemper et al. 2024).

Full size image

Although the number of matching ambiguities can be reduced by using the user-defined search threshold and SNN, they cannot be resolved completely, which is the main reason for the 3D trajectory interruptions. Tracking bubbles in 3D space while using information in image sequences aims to reduce the interruptions of bubble trajectories and increase the yield of long trajectories. One such method was introduced to the 3D-PTV community by Willneff and Gruen (2002) and relies on the use of a quality criterion that computes the correspondence between unmatched particles and established trajectories in image sequences to perform disambiguation. This work demonstrates that the capability of standard 3D tracking in space can be enhanced by incorporating image-based information.

In order to improve 3D tracking with temporal information from image sequences, we first have to establish the 2D tracks for each view. To ensure high accuracy in tracking multiple bubbles in 2D, we use the bubble tracker developed by Ma et al. (2023b). This 2D tracker is able to track bubbles even when they are completely obscured by other bubbles for several time steps. To do so, a graph-based framework is used, where detected bubbles in an image sequence are represented as nodes and possible connections in time are represented as edges between the nodes. After generating the graph, relevant node information, i.e. appearance features, and edge information, are embedded with dedicated Feedforward Neural Networks (FNNs). Then a Message Passing Network (MPN) iteratively updates the embedded information by aggregating neighbouring node and edge information, so that with each iteration information of nodes with increasing distance in time is passed to the current node to be updated. The final edge classifier then predicts valid edges between the nodes; in other words, valid connections of detected bubbles are used to form bubble tracks. Due to the MPN, edges between nodes with multiple time steps in between can also be classified as valid, which represent bubbles that are not visible for some time. More information and validation of this 2D bubble tracker can be found in Ma et al. (2023b).

The main task of the present work is to improve the performance of our former 3D tracking method by using already established temporal information from the image sequences. To this end, a cost function is proposed as a quality criterion to select the most likely combination of cross-view 2D bubbles, taking into account the corresponding temporal information of the 2D tracks. In this way, more triangulated bubbles are introduced into 3D space, waiting to have a spatio-temporal connection with other 3D positions or trajectories. Two further trajectory post-processing steps are introduced that use the cost function to link related trajectory fragments and to extend trajectories. Details are given in the following sections.

In order to evaluate the performance of the proposed extended 3D tracking method compared to the previous implementation, the same artificial sequences are used. The parameters used to generate the 3D bubbly flow sequences are shown in Table 1. In line with our previous work, the metrics used for the evaluation are as follows:The two approaches were initially tested on the ground truth data set, i.e. for the ideal case of having perfect detections as well as perfect 2D trajectories. The results, as shown in Table 2, illustrate improvements in almost all aspects following the integration of the extension within the former 3D tracking scheme. As the ground truth data do not suffer from the overlapping issue, each bubble in the sequences can have complete geometrical data, thus enabling the generation of accurate search corridors during matching. Additionally, initialized and established trajectories provide unbiased 3D positions, enhancing Wiener filter’s precision in predicting the potential bubble position in the upcoming frames. Furthermore, these perfect detections can also make the SNN more effective in achieving the disambiguation task. This is the main reason why the improvement gained by the extension in this context is not significant. Nevertheless, it evidences that the standard 3D tracking can be enhanced by incorporating temporal information from multiple 2D domains.

The extension approach was applied to the sequences based on the results of the 2D bubble detection and tracking. A comparison of MOTA and MOTP obtained by the extended 3D tracking method with the values obtained by the previous method is shown in Fig. 5.

Full size image

Looking at the MOTA results with and without extension, it can be seen that the extended 3D tracking method results in an overall improved MOTA score, which is in part due to the reduction of incorrect predictions and ID switches caused by unmatched bubbles. While in most cases the extended method is able to improve the MOTA by about 0.1, in some cases the improvement is close to zero (an example with improved MOTA is shown in Fig. 6a, while an example without considerable improvement is shown in Fig. 6b). The reasons for this are manifold: first, most of the cases involved were generated with long sequences (over 50 frames). A long 3D trajectory can potentially provide more 2D representatives, since a link failure due to a nearby ghost bubble does not affect the 3D tracking method too much. As mentioned above, the reconstruction of 3D trajectories is only achieved by taking one representative with the lowest cost value. A larger number of 2D representatives means a larger number of correspondence ambiguities, which increases the likelihood of misjudgement of the cost function, especially when the noise ratio is high (i.e. a large number of non-trackable bubbles, corresponding to the difference between \(\langle g_{all} \rangle\) and \(\langle g_{t} \rangle\)).

Full size image

However, this effect can only partially explain the lack of improvement. A key limitation is the void fraction, which is closely related to the bubble image coverage. Higher gas fractions lead to increased occlusions in the image sequences, resulting in increased noise levels and matching ambiguities. This reduces the effectiveness of 2D detection, which subsequently affects 3D tracking performance. As a result, trajectory interruptions and ghost trajectories become more likely. For instance, in the case of high bubble image coverage combined with high noise levels, the fraction of ghost tracks (\(f_{ghost}\)) is significantly increased (see the optical tracking results of case E6C2 in Fig. 6b and compare the \(f_{ghost}\) values in Table 3). In such cases, bubbles are more likely to be occluded by non-trackable bubbles, which can lead to mismatches. Obviously, not all detected bubbles should be matched if they are only visible in one view. Additionally, not all detected bubbles can be matched if they are occluded by other bubbles in other views. If such occlusions last over more than two frames, ghost trajectories can often be created. In contrast, scenarios with lower number of frames or bubble image coverage present more stable improvements in MOTA values for different setups, bubble densities, and noise levels. Looking at the MOTP results, the differences in MOTP values are less pronounced, see Fig. 5 entitled MOTP. Note that MOTP provides an indication of the spatial accuracy of the prediction of bubble 3D positions, as well as the accuracy of the estimation of bubble rise velocities. The ability of the extended tracking method to predict the bubble rise velocity is evident in Table 3, which shows the comparison of the predicted velocities with the ground truth velocities in various cases. In general, the algorithm demonstrates robust capabilities in predicting bubble rise velocities; even in challenging conditions, such as case E6C2 with higher occlusion and noise levels, it is still able to accurately estimate bubble rise velocity with an absolute difference of 4.126 mm/sec. However, in case E4C2, a large error (12.727 mm/sec) is observed, which is mainly due to the limited statistical sample size (see Table 1 for the values of \(\langle g_{t} \rangle\)), making the result more susceptible to the influence of outliers.

The results of the long-term tracking fraction \(f_{t>2/3}\) provide the most direct indication of whether the extension approach is a useful addition to the former 3D tracking method. Comparing the average \(f_{t>2/3}\) values of the two methods given in Table 4, it is clear that the inclusion of temporal information leads to significantly longer correct trajectories. This is mainly due to the introduction of the cost function, which incorporates a corridor-bubble match quality score and a physical constraint (bubble rise velocity). Based on temporal information from multiple 2D domains, the cost function provides appropriate cost values to deal with unresolved ambiguities, which is the main reason for trajectory interruptions when using the former 3D tracking method. It is obvious that not every trajectory could be improved via extensions, but even bridging a small gap in time steps is a significant improvement for the analysis of bubble hydrodynamics.

However, incomplete trajectories still remain due to the difficulty of bubble detection and reconstruction at the boundaries of the measurement volume. Therefore, most of the trajectories lack head and tail information (see the tracking results of case E6C2 in the top view of Fig. 6b). A main limitation of the extended approach is the limited performance of the 2D bubble detection and tracking at high bubble image densities; therefore, ghost-free tracking is desirable but could only be achieved in cases like E4C2, where a better optical access of the bubbles is available. As shown in Table 1, in case E4C2, the number of visible bubbles (\(\langle g_{all} \rangle\)) is relatively modest in comparison with other cases, as is the number of trackable bubbles (\(\langle g_{t} \rangle\)) within the measurement volume. The optical access in such a case minimizes 2D detection and tracking errors, thereby allowing the extended method to effectively handle ambiguities and achieve ghost-free tracking (see Table 3 for the values of \(f_{ghost}\)).

Comparing the average results in terms of MOTA, \(f_{t>2/3}\) and \(f_{ghost}\) values gained in the different steps, Table 4 reveals that the extension with full functionalities yields the optimal results. Nevertheless, the improvement achieved in the first step of this stepwise extension concept is overall more pronounced than in the last two steps. However, the majority of the ghost trajectories are introduced into space after the first step has been processed.

We notice that most ghost trajectories are relatively short. Therefore, one effective method for filtering out ghost trajectories is the incorporation of a duration threshold. As demonstrated in Table 4, the complete elimination of ghost trajectories can be achieved by implementing the full extensions (Extension 1+2+3) in conjunction with an appropriate duration threshold. Nevertheless, while the elimination of ghost trajectories is achieved, the unexpected exclusion of certain valid short trajectories, particularly those that have been correctly tracked at the boundaries of the measurement volume, may occur. Thus, the fraction of correctly established trajectories (f) is reduced, as is the long-term tracking fraction (\(f_{t>2/3}\)). Evidently, an appropriate duration threshold is always case-dependent and highly dependent on human experience as well.

In this paper, we introduce an extension approach to our previously developed 3D tracking method (Hessenkemper et al. 2024) to perform optical 3D tracking of deformable bubbles in swarms. The main idea of our extension is to incorporate temporal information from multiple 2D domains gained by the 2D bubble tracker into the former 3D tracking framework. Our extended 3D tracking method outperforms the previous method in a test with a broad range of artificial image sequences. To solve the issue of missed triangulated bubbles, a cost function considering the 2D track information is applied to introduce more triangulated bubbles into space. Additionally, linking errors caused by the prediction process in former implementation can also be partially prevented. Two further trajectory post-processing steps are described, both of which attempt to reconstruct longer bubble trajectories by connecting trajectory fragments and extending trajectory heads and tails, respectively.

When tracking bubbles in cases with a large number of non-trackable bubbles combined with high bubble image coverage, the extended 3D tracking method tends to produce more ghost trajectories compared to the former method. The main reason for the increase in ghost trajectories is that the extended method is forced to triangulate a 3D position while encountering a bubble matching conflict, even though this conflict is considered to be a group of non-trackable bubbles. Generated ghost bubbles can be linked to form a ghost trajectory over the entire time span of the sequences. Eventually, the fidelity of the generated bubble trajectories to the ground truth trajectories is largely due to the accuracy of the 2D bubble detector. The artificial sequences, including the corresponding 3D ground truth data, are available in Hessenkemper et al. (2024). The 2D track information and the code for the extended 3D tracking method can be accessed via [CODE PUBLICATION IN PREPARATION].

Furthermore, the validation of the extension with ground truth data highlights its potential value in practical applications. The idea of incorporating temporal information from multiple 2D domains could also improve 3D tracking of other objects like tracer particles. This, however, necessitates adaptations to the methods introduced in this work, since they were specifically designed for tracking deformable bubbles.