Advanced iterative particle reconstruction for Lagrangian particle tracking

IPR only reconstructs the particle distribution \(\left( {x,y,z,I} \right)_{i}\) and therefore relies on prior knowledge of the camera system. Typically, we aim at reconstructing the particle positions with accuracies below 0.1 px (much smaller for noise-free synthetic data). Therefore, we have high requirements on the calibration of the camera system, which are typically met by performing a volume-self-calibration (VSC, Wieneke 2008) on top of the geometrical calibration using a 3D calibration plate.

The gained correspondences of world coordinates to image coordinates are used to optimize the parameters of a camera model. This model represents the camera cam as a function that projects from 3D world-coordinates onto the 2D image coordinates, as well as a function cam⁻¹ that projects from 2D image coordinates onto 3D line-of-sights (represented by two points lying on said line). Further, the Jacobian matrix J with the derivatives of the camera function is required for the optimization of the IPR particles and the calibration of the cameras.

The actual implementation of the camera model can be freely chosen, as long as the aforementioned properties are fulfilled. For the synthetic test cases below, we use a perspective camera (akin to a pin-hole model). For experimental cases, such a model can be amended with second- and/or fourth-order correction terms.

The images of illuminated particles typically vary in shape and intensity, depending on their position within the measurement volume and the different viewing angles of the cameras. The attainable position accuracy of an image matching process highly depends on the conformity of the two involved images—the re-projected—and the original particle image. Therefore, the imaging properties of the particles are calibrated using the methods described in (Schanz et al. 2013a). The resulting average shapes of the particle images within several sub-volumes are modeled as a 2D Gaussian with 4 parameters for each camera:

The parameters \(a\), \(b\) and \(c\) encode the circular or elliptical shape and orientation of the Gaussian and the parameter \(I_{a,r}\) the average brightness of the particles on the current camera, relative to the other cameras. Since the particle images are assumed to be diffraction limited, the four parameters are solely dependent on the particles 3D position and the local OTF of the respective camera.

Within the IPR processing, the calibrated shape parameters are used as kernels around the projection points of the reconstructed particles while creating the projected images [as per Eq. (1)].

The task to optimally reproduce the measured image img_uv by a reprojection of a 3D particle cloud is tackled by reducing the pixel-wise squared distance between img_uv and rep_uv for each camera with as few and as bright particles as possible. To that end, we define the residual image res_uv as the pixel-wise difference between img_uv and rep_uv:

With this residual image, the cost function on camera j becomesand the total cost of the reconstruction is the sum of the costs of each camera

In an ideal situation, res_uv would be completely void of residuals of particles already contained in the state vector. However, due to particle imaging issues (noise, deviation of a particle image from the calibrated OTF, non-Gaussian particle images, particles in outer regions only visible by a subset of cameras, etc.) and due to the existence of ghost particles (at least in the first iterations of IPR), residuals of the images of already reconstructed particles are common.

The detection of particle images on the camera images is carried out using a custom procedure that understands all pixels of the image as the parameters of 2D B-splines and therefore effectively smooths the image. While the smoothing reduces noise the splines also provide an interpolation between the pixels of the image and spatial derivatives. The peak-detection iterates over the B-spline grid looking for bright spots above a certain threshold P_min. For every bright spot, a Levenberg–Marquardt algorithm (Moré 1978) is used to find the associated maximum with subpixel accuracy (approximately on the order of 0.1 px). Further, it estimates the shape of each found peak, suitable for calibrating the OTF.

From the peaks detected on the (residual) images of the different cameras, the 3D positions of the underlying particle cloud have to be estimated. To this end, given a certain peak on one camera, the corresponding peaks on the other cameras, belonging to the same particle, have to be identified. This can be achieved by the triangulation procedure, which is based on epipolar geometry and estimates a set of possible 3D particle positions given detected 2D particle peaks from a set of multiple cameras (see Fig. 2 for a scheme).

Several effects limit the abilities of the triangulation procedure:

The IPR procedure as introduced by Wieneke (2013) tries to counteract some of the mentioned drawbacks:

The triangulation process is limited by the accuracy of the peak detection and does not suppress ghost particles. Therefore, the positions and intensities of the particles are further optimized using image matching schemes, which iteratively shift the position and intensity of the particles to minimize the difference between the original camera images and the reprojection of the particles (being the residual images).

The original IPR as introduced in (Wieneke 2013) minimizes the cost function by iteratively optimizing the position \(x_{i} ,y_{i} ,z_{i}\) and intensity I_i of each particle individually by using a procedure roughly equivalent to a Newton-fitting algorithm with numerically determined first and second derivatives for \(x_{i} ,y_{i} ,z_{i}\) and a redetermination of the current optimal intensity I_i in each step. To this end, each single particle is displaced in all three directions of space by ± 0.1 pixels and the local cost is determined on all of these positions. A second-order polynomial fit is applied to the residuals and the particle is moved to the positions of minimal residual (with a maximum step of 0.1 px) in each direction of space.

This approach requires to estimate the local cost around each particle 7 times per camera in each iteration. Depending on the image size and the form of the OTF, this can be an expensive operation. Therefore, it is desirable to minimize the number of estimations. Secondly, for best results it is necessary to estimate the numeric derivative with high precision, for which the recommended procedure is to use analytic derivatives whenever possible.

We substitute the optimization using numerical derivatives with analytic derivatives of the cost function. For this, we first calculate the derivative with respect to the projected particle positions u_i and v_i and the particle intensity I_i of particle i on camera j:

IPR does not optimize the projected particle positions \(u_{i}\) and \(v_{i}\), but the 3D position \(x_{i} ,y_{i} ,z_{i}\) and the intensities \(I_{i}\), \(u_{i}\) and \(v_{i}\) are functions of the 3D position of particle i. The exact dependency is defined by the camera-model that maps the 3D positions onto pixel coordinates. Given the Jacobian of the camera function, the chain-rule can be applied to Eqs. (6) and (7), yielding

Using the same schemes, also the second derivate of the cost function with respect to the spatial coordinates can be determined analytically. It is evident that all derivatives and the local cost can be simultaneously estimated while only iterating once over the immediate proximity of the projected particle position.

The classical approach to use these cost derivatives to fit the interim solution to the measurement is to apply common fitting algorithms, like LBFGS, to solve the problem for all particles simultaneously. Unfortunately, standard libraries often do not allow adding and removing of particles at runtime (we expect it to be possible to adapt the LBFGS to these needs, however that is a work in progress). Furthermore, special features, like ignoring a selection of cameras for each particle to actively suppress ghost particles are not available. Due to these reasons, localized methods that treat each particle individually have shown a better reconstruction performance. Still, global solvers have a potential of being effective tools for the position optimization, once specialized versions are developed. A first useful development and its application are presented in Sects. 3.6 and 5.

Using the derivatives introduced above, several optimization procedures working on individual particles have been tested, including ones that query the second derivative to estimate the optimal step size, thereby requiring ideally only one sampling of the cost function.

However, only the technique that proved to be the most stable is presented here, which is the steepest-descent method, requiring only the first derivative.

Each inner iteration of shaking can be followed by a filtering step, where presumable ghost particles are removed from the current interim solution. The most important filtering procedure is removing all particles below a certain intensity threshold \(I_{\min }\). This measure is also proposed in the original IPR paper. Ghost particles usually only poorly overlap with measured peaks or draw their intensity mostly from one or two cameras only. Furthermore, the true particles drain intensity from the ghost particles during the optimization step. For that reason, ghost particles tend to have a lower intensity than true particles, increasingly so with increasing number of iterations (see Fig. 3). Opposed to Wieneke (2013), we do not define \(I_{\min }\) as a percentage of the average intensity, but as a fixed value, which proved more stable in most situations.

Other filtering methods are additionally applied. Particles that are found in close proximity in 3D space (below a threshold of typically 1 pixel) are merged, as it can be assumed that they represent the same true particle. The darker particle is deleted and its intensity is added to the brighter particle. Also, particles need to be visible within at least two cameras. As soon as they leave this volume, they are deleted.

After triangulation and position optimization with filtering, the residual images \({\text{res}}_{uv}\), as introduced in Sect. 2.3, are created. These images are the basis for the following outer IPR iteration, as the peak detection will now be carried out on \({\text{res}}_{uv}\). In order to more effectively erase known particles from the images, a factor \(f_{{{\text{Pt}}}} \ge 1\) can be used on the particle intensity while creating the projected images prior to the triangulation of new particles on the residual images.

Rendering the residual images is not only used in step 1 before the triangulation procedure, but also internally in step 4 within every inner iteration of the shaking process (here \(f_{{{\text{Pt}}}} = 1\) is used).

In Sect. 3.3, we showed how to efficiently determine the derivatives of the cost function with respect to the 4N parameters \(x_{i} ,y_{i} ,z_{i} ,I_{i}\) for all N particles (by estimating the derivatives for each particle on each camera and looping over its immediate proximity in the residual image).

As indicated in Sect. 3.3, these algorithms currently cannot compete with approaches that optimize each particle individually. Another feature of this approach, however, proves to be very useful. Namely, during the estimation of the derivatives with respect to the particle parameters it is possible to also estimate the derivatives of the cost function with respect to internal camera parameters with little additional cost. To take advantage of this, we introduced an additional global 2D shift on each camera to model small shifts induced due to vibrations during the measurement.

These two extra parameters per camera can simply be added to the gradient vector, resulting in a total of \(4N + 2N_{{{\text{cams}}}}\) number of degrees of freedom to be optimized. This way, the global shake procedure can be used to correct for vibrations in an experimental setup or for small initial deviations from the camera calibration, which might occur between subsequent realizations of an experiment.

The capability of the global shake, which can be regarded as a bundle adjustment method, to account for decalibrations is demonstrated in chapter 6.

The performance of the extended IPR processing method when applied to synthetic data is analyzed below. To this end, a synthetic test case was created, which replicates the one used in Wieneke (2013): Four virtual pin-hole cameras of 1300 × 1300 pixels each are oriented in a pyramidal viewing configuration and image randomly created particles in a domain of 50 × 50 × 16 length units. All cameras view the full volume. The particle shape is Gaussian with a diameter of 2 pixels (\(I_{a,r} = 1.0, a = 2\), \(b = 0\), \(c = 2\), see Sect. 2.2). No variations of the OTF were included here. The pixel values are always integers.

Three different cases were considered: perfect imaging conditions (case I), moderate noise (case II) and heavier noise (case III). Particle image densities ranging from 0.02 ppp to 0.16 ppp were investigated.

Unless otherwise noted, the following parameters were used for the IPR reconstructions of all cases: 20 outer IPR iterations \(N_{{{\text{IPR}}}}\) were conducted. For the sake of clarity and visualization of the results, this value remained constant, even if convergence is usually reached earlier. The allowed triangulation error ε starts at ε(1) = 0.4 px and is linearly increased to ε(N_IPR) = 0.8 px (see Sect. 3.2). Due to the low initial values of ε, the triangulation can immediately start with using a reduced set of cameras (only \(N_{{\text{c}}} - 1 = 3\) cameras are required to have a peak in order to register the particle). Each outer IPR iteration (as depicted in Fig. 1) used \(N_{{{\text{shake}}}}\) = 10 inner iterations of shaking. From those, particle filtering (intensity, duplicates, FOV) is performed for the last seven iterations. The number of brightest cameras left out in the intensity update (see Sect. 3.3) is set to \(N_{{{\text{C}}i}}\) = 2. The camera order is permutated 4 times (1–2–3–4, 4–1–2–3, 3–4–1–2, 2–3–4–1) and a triangulation is carried out for each permutation (see Sect. 3.2)). The maximum step width is initially set to \(d_{\max } = 0.4\;{\text{px}}\). For iterations, 15–17 \(d_{\max }\) is reduced to 0.1 px; for iterations, 18–20 a further reduction to \(d_{\max } = 0.02\;{\text{px}}\) is applied. The factor applied to the reprojected image was \(f_{{{\text{Pt}}}}\) = 1.0.

The created virtual images are reconstructed and the results compared to the original particle distribution.

First the performance of the algorithm will be assessed on images free of any defects (case I). The particles have a constant intensity of 1000 counts in the volume, which is distributed over all pixels of the particle image in each camera (leading to typical peaks of 300–400 counts, see Fig. 4). All pixels of the generated measurement are rounded to the nearest integer. No image pre-processing is applied. The main IPR parameters are indicated in the section above. Specific parameters for this case are: the intensity threshold for identifying peaks on the (residual) images is \(P_{\min }\) = 50 counts, the intensity threshold in the volume, below which a particle is discarded as a ghost particle, is \(I_{\min }\) = 200 counts (i.e., 20% of the average particle intensity).

Figure 5 shows the convergence behavior of IPR using the steepest-decent method (SDM) for varying particle image densities, when applied to the described noise-free synthetic dataset. Shown is the ratio of correctly identified particles \(R_{{\text{f}}}\) (reconstructed within a search radius of 1 pixel around the true particles; normalized with the number of true particles and abbreviated as ‘founds’), the ratio of ghost particles \(R_{{\text{g}}}\) (reconstructed particles, for which no true particle could be found in a radius of 1 pixel; also normalized with the number of true particles and abbreviated as ‘ghosts’) and the average accuracy of the particle positioning of the correctly found particles \({\Delta }_{{\text{f}}}\) (calculated as the average distance of the reprojected points to the original peak position).

For lower particle image densities (N_I ≤ 0.08 ppp), the method converges to a stable value of \(R_{{\text{f}}}\) \(\approx\) 1 in less than 5 iterations. At N_I = 0.14 ppp, the reconstruction converges after 15 iterations. Even for this high seeding density, only 8 particles out of 136,477 are not found. For the highest seeding density, convergence is foreseeable, but not achieved within the 20 iterations used (see also Sect. 4.4).

For low values of \(N_{{\text{I}}}\), the number of ghost particles quickly diminishes after a few iterations. At higher particle image densities though, a rise in ghost numbers occurs within the first five iterations, which can go up to \(R_{{\text{g}}} = 1\). This can be understood, as in these cases many particles could not be identified in this early state and the uncertainty of the found ones is high. Therefore, the residual images will show relevant intensities, even in spots where the particles were already triangulated. These residual intensities will be picked up by the peak finder and new false particles will arise.

However, with the identification of more and more true particles, the ghost level is gradually reduced. As soon as all real particles are found, also the ghosts have disappeared. Indeed, only for N_I = 0.16 ppp ghosts are still present after 20 iterations (as this case is not yet converged). For all other cases, the number of detected ghosts is 0.

In order to give a better insight into the workings within a single IPR iteration, Fig. 6 shows the development of ghost and found true particles at N_I = 0.14 ppp, resolved for the triangulation and all shake iterations (blue and orange lines). Additionally, the end-result of each iteration is connected (black and red lines, which then correspond to the results from Fig. 5). Looking at the first few outer iterations, it becomes obvious that initially high numbers of weak ghost particles are triangulated. These remain in the particle cloud for the first three shake iterations (the short flat parts of the curve at the beginning of each inner iteration). As soon as the filtering of the particles is performed, the majority of these is deleted, as their intensity has been further reduced by the shaking. From there, the number slowly decreases, until the next triangulation drives the number up again. At the same time, the number of found true particles is barely reduced by the shaking. With every outer iteration, more true particles are found, reducing residual peaks, which in turn gradually minimizes the number of triangulated (ghost-) particles, until all particles are identified and zero ghost particles remain.

The position accuracy is very much dependent on the presence of ghost particles and the detection of true particles. As soon as these two have converged, high levels of accuracy are reached. The reduction of the maximum step width \(d_{\max }\) at iteration 15 is very visible, e.g., for the 0.12 ppp case, the position accuracy dips from a plateau at 0.04–0.007 px. The second reduction of \(d_{\max }\) at iteration 18 yields a further reduction of \({\Delta }_{{\text{f}}}\) to 0.005 px at this seeding density. At 0.16 ppp, it becomes evident that a reduction of \(d_{\max }\) leads to a reduction in the speed of convergence, when looking at \(R_{{\text{f}}}\) and \(R_{{\text{g}}}\). Therefore, this measure should only be applied after convergence is reached.

Compared to the results of Wieneke (2013), which were gained using a comparable synthetic experiment, the range of particle image densities has been greatly extended. Wieneke reported a usable particle image density of N_I = 0.05 ppp and a sharp decrease of reconstruction quality when going beyond this value using 16 outer and 6 inner iterations. Here we find that images at N_I = 0.14 ppp produce results completely void of ghost particles and very few missing real particles (far below Wienekes values at 0.05 ppp). The position accuracy of the advanced IPR is much higher, with values Δ_f ≤ 0.01 px for all converged cases. These quite profound advances in reconstruction quality result from the combination of the several enhancements to the IPR algorithm introduced above, whose effect is quantified in Sects. 4.6 to 4.9.

The next section will discuss the influence of camera noise and particle intensity variations on the reconstruction results.

Reconstructions of noise-free synthetic images provide a mean to characterize the theoretic performance of algorithms; however, images gathered within experiments will show numerous deviations from perfect conditions. Camera noise can play an important role, as the light budget is typically limited for volumetric investigations, leading to low signal-to-noise ratios. Seeding particles are typically not perfectly monodisperse, leading to—sometimes broad—distributions of peak intensities. A single particle might even show variations in imaged intensity for the different camera projections due to effects of scattering angles (after Mie theory) (Manovski et al. 2021) or uneven particle shape. In order to characterize the effects of some of these experimental shortcomings on the reconstruction quality, synthetic test cases with two different levels of simulated camera noise and a distribution of particle size (i.e., intensity) were created (cases II and III).

For case II (moderate noise), the particles have a mean intensity of 1000 counts in the volume, with a Gaussian distribution with \(\sigma^{2} = 250\) counts, leading to image peaks of different brightness in the range of ~ 200–500 counts. A constant Gaussian noise floor with a mean of 40 counts and rms width of 25 counts is added to the image. To further simulate photon shot noise, a Poisson process is applied for every pixel, thereby adding noise proportional to the intensity of individual pixels. Case III (heavier noise) applies a Gaussian distribution with \(\sigma^{2} = 500\) counts to the particle intensities, leading to image peaks in a broad range of ~ 40–800 counts. A constant Gaussian noise floor with a mean of 80 counts and rms width of 50 counts is added to the image. Again, pixel shot noise is additionally applied. Case II was chosen to represent an experimental case with very good imaging conditions, e.g., recordings of Helium-filled soap bubbles illuminated by LED light (see e.g., Huhn et al. 2017). Case III represents tougher experimental conditions, like laser-illuminated droplets in air (e.g., Manovski et al. 2021) or particles in water (e.g., Neeteson et al. 2016; Schröder et al. 2020). We are aware that other effects (like speckles and scattering laws) are as well relevant in this context; however, these topics are beyond the scope of this work. Figure 7 shows exemplary excerpts from both cases at low- and high-particle image densities.

Image pre-processing consisted of subtracting a constant of 40 counts (case II) and 80 counts (case III), which roughly corresponds to subtracting the minimum or average image, as often performed for experimental cases. The main IPR parameters are the same as indicated in Sect. 4.1. Specific parameter for this case: the intensity threshold for identifying peaks on the (residual) images had to be increased to \(P_{\min }\) = 70 counts (case II) and \(P_{\min }\) = 90 counts (case III), in order to limit the influence of noise peaks. In order to more effectively remove false particles, triangulated from the remaining noise peaks, the intensity threshold in the volume, below which a particle is discarded as a ghost particle, is increased to \(I_{\min }\) = 350 counts (case II) and \(I_{\min }\) = 450 counts (case III), respectively.

Figure 8 shows the convergence of the advanced IPR processing for varying particle image densities, when applied to the noisy synthetic datasets. For the lower noise level (left column), it can be seen that the general convergence is slowed down compared to the noise-free case. However, all cases below N_I = 0.14 ppp converge to \(R_{{\text{f}}} \approx 0.99\) within at most 13 iterations. For N_I = 0.14 ppp, a similar convergence would likely be achieved when applying more than 20 iterations. For all but the highest particle image densities, the number of ghost particles is virtually zero, once convergence is reached. The position accuracy shows a constant level, determined by the noise level, which is slightly offset by increasing effects of particle overlap. For the converged cases, values between Δ_f = 0.09 px and Δ_f = 0.12 px are attained.

For the higher noise levels and intensity variation of case III, the ratio of found particles drops to values of \(R_{{\text{f}}} \approx\) 0.8, which are achieved up to N_I = 0.1 ppp. For higher particle image densities, lower values are seen; still, approx. 66% of the real particles are correctly identified at N_I = 0.14 ppp for this relatively high noise level. In general, the convergence is much slower, as the peak detection accuracy suffers from the image noise, requiring higher values of \(\varepsilon\) for a successful triangulation, which are only attained within later IPR iterations. Despite a high number of noise peaks that reach above the peak detection threshold, the occurrence of ghost particles is limited (\(R_{{\text{f}}} < 0.03\) for N_I < 0.14 ppp). The position accuracy is more affected by the stronger noise, with most seeding densities ranging between Δ_f = 0.14 px and Δ_f = 0.18 px. For both noisy cases, the close proximity of Δ_f for a wide range of particle image densities points to the image noise as the dominant source for the position uncertainty. Consequently, the reduction of the maximum shake width to 0.1 and 0.02 px in the last IPR iterations is of no consequence here. It is evident that especially for case III not all particles could be reconstructed, which is likely caused by the large range of particle intensities, leading to peaks that are within the noise level and below the peak detection threshold for most or all cameras.

Figure 9 (top)shows the inner-iteration-resolved convergence of case III at N_I = 0.12 ppp. Opposed to the processing of case I (see Fig. 6), the number of found true particles is now affected by the filtering process. Even at the later iterations, a certain number of true particles is deleted by the intensity threshold filter, which is likely caused by the presence of low-intensity particles, which are bright enough to be picked up by the peak search, but fall below the relatively high volume intensity threshold of \(I_{\min }\) = 450 counts. The number of ghost particles fluctuates strongly within each IPR iteration, illustrating that the noise peaks are indeed strong enough to be detected by the peak finder, but do not form coherent 3D particles, which are easily removed by the intensity filter. The increase of triangulated ghost particles with convergence can be attributed to the increasing allowed triangulation error. The lower absolute number of ghost particles at the beginning in relation to the noise-free case I can be explained by the increased peak detection threshold (90 counts vs. 50 counts).

The combination of high noise levels and a significant spread of particle intensities inhibits a reconstruction of all particles for case III. The majority of the missing real particles are of so low intensity that they fall below the specified intensity threshold. Figure 9 (bottom) documents this by plotting the ratio of found particles \(R_{{\text{f}}}^{*}\), this time normalized by the number of true particles that have been assigned an intensity above \(I_{\min }\) (450 counts). It can be seen that approx. 95% of these particles are correctly identified up to N_I = 0.1 ppp. The remaining undetected particles are missed due to the heavy noise on the particle images, inhibiting a correct detection of the particle peak.

If performing STB evaluations of an experiment with multiple time-steps (ideally fully time-resolved), a full reconstruction is still possible. Due to the predictive step within STB, particles whose trajectory has once been found are typically not lost before leaving the measurement volume. Therefore, it is sufficient to identify a particle only once in four consecutive time-steps within its residence time, thereby greatly enhancing the chances of a successful detection. Two- (Jahn et al. 2017; Lasinger et al. 2019) and multi-pulse particle tracking approaches (Novara et al. 2019a), using, e.g., particle space correlation (PSC) (Novara et al. 2016b) as a local flow predictor, could apply iteratively reduced thresholds in order to track as many particles as possible.

Depending on the type of experiment, several other imaging defects might occur (especially when using laser illumination) that are not integrated into the current simulation. For such experimental data, even lower thresholds might have to be chosen, resulting in high amounts of ghost particles in single time-steps. Again, the dimension of time has to be used to separate them from the true particles.

A useful tool is a histogram of the number of detected peaks in relation to the peak finder threshold, which can help finding a suited threshold to separate noise from signal. Another measure that would be of benefit in such cases is to develop/apply a peak detection method that is relies less on the intensity as the sole detection measure. If, e.g., searching for a specific particle image shape instead of a bright peak, the detection would be less susceptible to noise.

The effects of changing the number of outer (IPR) iterations \(N_{{{\text{IPR}}}}\) is investigated by comparing results of reconstructions using \(N_{{{\text{IPR}}}} = 5\), \(N_{{{\text{IPR}}}} = 20\) and \(N_{{{\text{IPR}}}} = 100\). The linear increase of the triangulation radius from \(\varepsilon = 0.4\) to \(\varepsilon = 0.8\) is retained, only the spacing changes according to the number of iterations.

All other parameters remain unchanged compared to Sects. 4.2 and 4.3. Figure 10 (top) shows the convergence of these reconstructions for case I at three particle image densities. It can be seen that for the two lower values (\(N_{{{\text{IPR}}}} = 0.04{ }\;{\text{ppp}}\) and \(N_{{{\text{IPR}}}} = 0.1\;{\text{ppp}}\)) the differences are minor, however the reconstruction using \(N_{{{\text{IPR}}}} = 5\) actually converges the fastest, due to the quicker increase in \(\varepsilon\). At very high-particle image densities (\(N_{{\text{I}}} = 0.16\;{\text{ppp}}\)), the differences are profound: For \(N_{{{\text{IPR}}}} = 5\) only approx. 60% of the true particles can be identified, using \(N_{{{\text{IPR}}}} = 20\) increases this value to approx. 85%. When applying 100 iterations, the reconstruction converges to a full solution after 40 iterations.

A slightly different trend can be seen when using the noisy images from case III [Fig. 10 (bottom)]. At low-particle image densities, all reconstructions converge to \(R_{g} \approx 0.82\); however, the less iterations are used, the faster the convergence due to the quick increase of \(\varepsilon\). At medium values of N_I however, this reconstruction using \(N_{{{\text{IPR}}}} = 5\) is not able to converge to the same value of R_g as the other two and at \(N_{{\text{I}}} = 0.14\) the differences between the three reconstructions are again profound (\(R_{{\text{g}}} \approx 0.42\) at \(N_{{{\text{IPR}}}} = 5\), \(R_{{\text{g}}} \approx 0.66\) at \(N_{{{\text{IPR}}}} = 20\) and \(R_{{\text{g}}} \approx 0.74\) at \(N_{{{\text{IPR}}}} = 100\)).

To summarize, increasing the number of IPR iterations is beneficial at very high particle image densities. For typical experimental conditions, using low values for \(N_{{{\text{IPR}}}}\) (between three and ten) usually works equally well and further iterations might even not improve the values due to usually more complex experimental imaging conditions than those mimicked in the present synthetic test cases.

In order to quantify the effect of the number of inner (shake) iterations \(I_{i}\) on the results, the created images have been evaluated by IPR using \(N_{{{\text{shake}}}} = 5\), \(N_{{{\text{shake}}}} = 10\) and \(N_{{{\text{shake}}}} = 20\) shake iterations within each outer iteration. Filtering was always performed starting from the fourth inner iteration. All other parameters remain unchanged compared to Sects. 4.2 and 4.3.

Figure 11 (top) shows the results for three different particle image densities of case II. It can be seen that only for the highest particle image densities noticeable differences can be seen. At N_I = 0.06 ppp, all cases perform equally. Going to N_I = 0.1 ppp, the reconstructions using more iterations show a slightly increased convergence. At N_I = 0.14 ppp, the reconstruction using \(N_{{{\text{shake}}}} = 5\) does not converge, opposed to the other two cases. The reason for the instability of this case is further analyzed in. When comparing the inner-iteration-resolved convergence (Fig. 11, bottom) to the case using \(N_{{{\text{shake}}}} = 10\) (Fig. 11, middle), it can be seen how at the first iterations the additional iterations in the latter case lead to an ongoing reduction in ghost particles within every outer iteration. This, combined with a better positioning and intensity reconstruction of the particles, leads to a less populated residual image, which in turn leads to a reduced ghost particle triangulation in the next iteration. For the case using \(N_{{{\text{shake}}}} = 5\) on the other hand, the number of ghost particles triangulated within every outer iteration is increasing more and more as the triangulation error is growing with the iteration number. The high number of total particles leads to a reduction of average particle intensity, explaining the decrease of reconstructed true particles in the last iterations, as more and more true particles are deleted by the intensity threshold. Using a reprojection factor \(f_{{{\text{Pt}}}} > 1\) might stabilize the process, at the expense of some unidentified true particles.

All in all, it can be stated that for particle image densities that are typically encountered in experimental conditions, using low numbers of shake iterations in order to limit processing time comes with little penalty.

Conducting several triangulations with permutated camera order, as introduced in Sect. 3.2.1, has a strong influence on the convergence behavior of IPR. Figure 12 compares results from case II, which were reconstructed once using four successive triangulations with camera orders (1–2–3–4, 4–1–2–3, 3–4–1–2, 2–3–4–1) and once using a single triangulation with order (1–2–3–4). All other parameters remain unchanged compared to Sect.  4.3.

It is apparent that the permutation approach greatly enhanced convergence. For both particle image densities, this case yields higher numbers of true particles, starting from the first iteration. From there, the reconstructions converge within 5 (N_I = 0.06 ppp) and 14 (N_I = 0.12 ppp) iterations, while the standard approach fails to fully converge within twenty iterations in both cases. At the beginning, the permutated cases also show higher ghost levels, which can be understood, given that more triangulations are involved. However, the ghost particles are quickly suppressed as more and more true particles are found. For the standard case at N_I = 0.12 ppp, the ghost level rises within the first five iterations and is only slowly reduced in the following.

For cases I and III, similar results were attained. The slight increase in computational effort by conducting multiple triangulations is easily compensated by the much faster convergence.

As described in Sect. 3.2.2, all reconstructions presented until now use an intensity update, which ignores the two brightest camera (N_Ci = 2), i.e., uses the average of the two (locally) weakest cameras to compute the updated intensity. In this section, we analyze the impact of changing N_Ci. To this end, reconstructions of all test cases and particle image densities were conducted, varying N_Ci from 0 to 3.

Figure 13 shows selected results for case I and III. For noise-free images at moderate seeding density (N_I = 0.08 ppp), all versions start at the same fraction of found true particles; however, the more cameras are ignored, the better ghost particles can be suppressed. In case all cameras are used for the update (N_Ci = 0), the ghost fraction is too high to be quickly corrected for and convergence is delayed. At lower particle image densities, all versions perform approx. the same. In these noise-free conditions, the intensity information from peaks on a single camera is sufficient for an accurate determination, as can be seen when looking at the result with higher particle image density (N_I = 0.14 ppp). Again, the fraction of found true particles starts at the same value for all values of N_Ci and the ghost levels vary. The case with N_Ci = 3 converges within 12 iterations. Including two cameras in the intensity update (N_Ci = 2) leads to a slower convergence, which is reached after 15 iterations. For the other cases, convergence is not attained; at N_Ci = 0, only approx. 60% of the true particles can be identified. The reason for these differences is the strong difference in ghost suppression capability. A ghost particle is typically supported by strong peaks on one or two cameras only; the other cameras show low-intensity residuals or only partial overlap with a particle image. Therefore, the chances of ghost particles retaining a high intensity are greatly reduced when using N_Ci = 2 or N_Ci = 3.

The situation changes to a certain degree when noise and other imaging defects are present. The result of case III at N_I = 0.12 ppp shows that here the reconstruction using N_Ci = 3 performs worse than N_Ci = 2 and is even becoming slightly unstable (R_g going down in the last iterations), such that the reconstruction using N_Ci = 1 is reaching the same level, despite converging slower initially. Again, including all cameras in the intensity update performs worst. The reason for the worse performance of N_Ci = 3 in this case is likely a stronger deletion of true particles by the intensity filtering due to the presence of weak particles and the influence of the noise on the intensity registered in each camera.

Due to the high relevance of noise in real experimental data, N_Ci = 2 was chosen as the standard value for a four-camera setup, as regarded here. When using more cameras, a higher value of N_Ci might also be chosen.

As introduced in Sect. 3.2.2, all previously presented IPR reconstructions apply a triangulation radius, linearly increasing within the N_IPR outer iterations: \(\varepsilon \left( i \right) = \varepsilon \left( 0 \right) + i/\left( {N_{{{\text{IPR}}}} - 1} \right)*\Delta \varepsilon\), with \(\Delta \varepsilon = \varepsilon \left( {N_{{{\text{IPR}}}} } \right) - \varepsilon \left( 0 \right)\) and \(\varepsilon \left( 0 \right) = 0.4\;{\text{px}},\; \varepsilon \left( {N_{{{\text{IPR}}}} } \right) = 0.8\;{\text{px}}\). In this section, the differences to using a fixed value of ε, as applied, e.g., in Wieneke (2013), are discussed, as well as other functions of increasing ε.

Figure 14 compares the results of reconstructions using fixed values of \(\varepsilon = 0.6\;{\text{px}}\) and \(\varepsilon = 0.8\;{\text{px}}\) to the linear ramp approach for two particle image densities from case III. At low-particle image density (N_I = 0.04 ppp), the results do not differ much, with the reconstruction using \(\varepsilon = 0.6\;{\text{px}}\) converging quickest, followed by the ramp approach. Using \(\varepsilon = 0.8\;{\text{px}}\) results in a slightly slower convergence, likely due to an increased appearance of ghost particles, caused by noise peaks and the relatively high allowed triangulation error. Eventually all approaches converge to a comparable level of approx. 82% of found particles.

At high-particle image density (N_I = 0.14 ppp), the results are different. Here, the ramp approach is consistently yielding the best results. Using a fixed \(\varepsilon = 0.6\;{\text{px}}\) shows a slower convergence and a lower end value, while the fixed \(\varepsilon = 0.8\;{\text{px}}\) case completely fails to yield a significant number of true particles. To better understand these results, Fig. 15 shows the inner-iteration-resolved convergence of the two fixed cases at N_I = 0.12 ppp, which can be directly compared to the linear-ramp result, given above in Fig. 9. It is obvious that both cases using a fixed value of ε exhibit a high level of triangulated ghost particles at the beginning of the reconstruction (\(R_{{\text{g}}} \approx 2.5\) at \(\varepsilon = 0.6\;{\text{px}}\) and \(R_{{\text{g}}} \approx 5.0\) at \(\varepsilon = 0.8\;{\text{px}}\)), which can be avoided using a low value of ε, as the ramp approach does (here \(R_{{\text{g}}}\) always remains below 0.8). At a fixed value of \(\varepsilon = 0.6\;{\text{px}}\), the reconstruction is able to recover from the initially high ghost fraction, as enough true particles are above the intensity threshold to allow convergence; at \(\varepsilon = 0.8\;{\text{px}}\) however, the intensity available from the images is distributed over so many triangulated particles that nearly all of them fall below the threshold of \(I_{\min }\) = 450 counts. From the initially triangulated 55% of true particles, less than 3% remain after filtering. The ghost particles are nearly completely deleted. Therefore, the residual image used for the next triangulation iteration represents an almost unaltered original image and the reconstruction shows no signs of convergence over all 20 iterations.

Of course, these results will vary with the chosen parameters. If the triangulations of the first IPR iterations were conducted using the full set of cameras (i.e., requiring a peak on all 4 cameras, instead of just 3), the number of triangulated particles would be reduced; however, this would require more parameters and proved to be less flexible.

To summarize, using a fixed triangulation error can yield competitive results, especially at lower seeding densities and low-noise conditions, as long as it is chosen small enough. However, this requires a very accurate calibration that is not always attainable in real-life experiments. Therefore, we recommend using a triangulation error that is increasing with IPR iteration, which intrinsically provides a way to limit the occurrence of ghost particles in the first iterations, where the residual image is still very populated.

Other approaches than a linear increase of epsilon with the outer iteration were also explored: a quadratic function \((\varepsilon \left( i \right) = \varepsilon \left( 0 \right) + \left( {i/\left( {N_{{{\text{IPR}}}} - 1} \right)} \right)^{2} *\Delta \varepsilon\)) and a square-root function (\(\varepsilon \left( i \right) = \varepsilon \left( 0 \right) + \sqrt {i/\left( {N_{{{\text{IPR}}}} - 1} \right)} {*}\Delta \varepsilon\)). We saw that the square-root function led to faster convergence at noise-free conditions and at relatively low seeding densities, where the quicker progression to higher values of epsilon does not lead to critical ghost levels. At higher seeding densities and image noise though, a slower increase of epsilon better suppresses ghost particles and converges to higher values of found particles, thereby favoring the linear and the quadratic approach. Overall, the linear approach showed the most balanced results.

As laid out in Sect. 3.3, this IPR implementation applies an analytic derivation of the cost function in order to determine the optimal direction and step for the repositioning of each particle in each time-step. Here we want to characterize the differences when comparing this approach to the one applied in Wieneke (2013), who used numeric derivatives by moving the particle Δ_p ± 0.1 px in all three directions of space, fitting a second-order polynomial to the three values in each direction and finally moving the particle to the minimum of this polynomial (or apply the maximum step of 0.1 px, in case no minimum is found). This ‘classic’ position optimization approach was implemented in the current framework, and two versions were used for reconstruction: one always using a Δ_p ± 0.1 px, the other using Δ_p ± 0.23 px and reductions to Δ_p ± 0.1 px at iteration 15 and to Δ_p ± 0.02 px at iteration 18. These values are comparable to the ones used for the analytic approach, with a maximum step with of 0.23 px in each direction of space corresponding to a maximum step width of 0.4 px in 3D space. Please note that Wienekes intensity update was not adopted—the one described in Sect. 3.3.2 is applied in all cases.

Figure 16 compares results from two reconstructions using numeric derivation to an otherwise identical reconstruction using analytic derivatives. As can be seen, the differences are minor. Only at very high particle image densities (N_I = 0.16 ppp), the curves of found true and ghost particles differ appreciably. For this case, the reconstruction using numeric derivatives with a small step size (Δ_p ± 0.1 px) converges slowest; an increase to Δ_p ± 0.23 px accelerates convergence nearly to the level of the reconstruction using analytic derivation. At lower particle image densities, all algorithms perform basically identical. The positional accuracy of the case using a static value of Δ_p ± 0.1 px is typically lower than for the cases which reduce the step width for the final iterations. The analytic approach always retains a small accuracy benefit over both numeric approaches.

The findings above indicate that using numeric derivations is a viable alternative to analytic derivations in terms of reconstruction performance. However, the availability of analytic derivations allows for the application of other concepts to solve the reconstruction problem, like the ‘global shake’ bundle adjustment scheme discussed in the next chapter.