Calibration of multiple cameras for large-scale experiments using a freely moving calibration target

The different images of the calibration target are processed to identify the curves and the nodes corresponding to the gridlines and intersections between the tiles of the checkerboard (Fig. 2). In our application, using a checkerboard calibration target is advantageous over using a pattern of dots. This is because the image gradient obtained from a checkerboard determines grid points more accurately and more robustly over the large depth-of-field of our experiment.

The calibration images are converted to the image gradient using a Savitsky–Golay image differentiation approach (Meer and Weiss 1992) to mark locations on the gridlines between the tiles. For each image, we then fit a set of polynomial curves \(\gamma _i(t)\) to the \(I=9\) gridlines between the tiles of the checkerboards using the local intensity values from the image gradient. These fitting curves are written as \(\gamma _i(t)=\sum _k {\mathbf {a}}^k_i t^{k-1}\), where \({\mathbf {a}}^k_i\) are two-dimensional vectors. Here the parameter t varies within the interval \(t \in [0 , 1]\) over the checkerboard image, such that \(\gamma _i(t=0)\) and \(\gamma _i(t=1)\) correspond to the beginning and end-points of the gridline of the checkerboard image, see the lines on Fig. 2. At last, we find the \(J=20\) intersections between all the gridlines as a set of nodes \({\mathbf {x}}_{j}\) in the image plane of each camera. Here \({\mathbf {x}}_{j}=[x_j \ y_j ]^T\) with \((\bullet )^T\) the vector transpose, where the numbering \(j=1 \cdots J\) is consistent between the different camera views (Geiger et al. 2012) (Fig. 2).

Following the path of an optical ray for a single camera in Fig. 2, the linear path is refracted across the air/water interface (Belden 2013). This causes optical distortions in the image plane for each camera and, therefore, we rectify the image plane by dewarping the optical distortion. The coordinates \({\mathbf {x}}=[x \; y]^T\) in the image plane are mapped to distortion-corrected image coordinates \(\hat{{\mathbf {x}}}=[{\hat{x}} \; {\hat{y}}]^T\) by determining a distortion mapping \(\hat{{\mathbf {x}}}=m({\mathbf {x}})\). This mapping ensures that collinear points in the object domain are projected as collinear points in the dewarped image plane and, therefore, supports linear ray tracing. For moderate optical distortions, a polynomial distortion map (Soloff et al. 1997) is sufficient. In this study, we write the distortion map asThe distortion map is determined by rectifying the curved gridlines in the N original calibration images. We follow Devernay and Faugeras (2001) and minimize the percentage of deflection along the gridlines. We consider the nodes \({\mathbf {x}}_{j}^n\) along a particular gridline \(\gamma _i^n(t)\) and their images \(\hat{{\mathbf {x}}}_{j}^n\) and \({\hat{\gamma }}_i^n(t)\) in the dewarped image plane through the map \(m({\mathbf {x}})\). We fit a straight line \({\hat{\ell }}_i^n\) through the nodes \(\hat{{\mathbf {x}}}_{j}^n\) and compute the point-line distance \(d(\hat{{\mathbf {x}}}_{j}^n,{\hat{\ell }}_{i}^n)\). The parameters \({\mathbf {c}}_k\) defining the distortion map are then determined by solving a minimization problem over all gridlines in all calibration images:This minimization problem can be solved efficiently using a steepest descend algorithm and numeric integration techniques described in Boyd and Vandenberghe (2004). This approach directly extends to larger optical distortions requiring more elaborate distortion models (see Appendix A for the air/water interface and Supplementary-Fig. 8). An example of a dewarped image can be found in Fig. 2.

We consider a physical point in the object domain \({\mathbf {X}}\) of coordinates \({\mathbf {X}}=[X \; Y \; Z]^T\) in a world coordinate system and its projected image \(\hat{{\mathbf {x}}} = [{\hat{x}} \; {\hat{y}}]^T\) in the dewarped image plane of a single camera. The calibration is defined by the mapping function F, such that \(\hat{{\mathbf {x}}} = F({\mathbf {X}})\). Our method uses the framework of projective geometry to express the mapping function F and implicitly assumes a pinhole camera model. In the following, we outline the main notations used in projective geometry; a more complete introduction can be found in Hartley and Zisserman (2004).

We make use of augmented vectors to represent points in both the image plane and in the object domain. The coordinates in the dewarped image plane \(\hat{{\mathbf {x}}}\) are augmented to the ray-tracing vector \(\tilde{{\mathbf {x}}}\) such that \(\tilde{{\mathbf {x}}} = [k{\hat{x}} \; k{\hat{y}} \; k]^T\), where k is a scaling parameter in the direction of the principal optical axis. The associated inverse function that projects \(\tilde{{\mathbf {x}}}\) back to \(\hat{{\mathbf {x}}}\) is defined as the projection \(p(\tilde{{\mathbf {x}}}) =[{\tilde{x}} \; {\tilde{y}}]^T/k=\hat{{\mathbf {x}}}\). Similarly, the world coordinates \({\mathbf {X}}\) are augmented to a homogeneous vector as \(\tilde{{\mathbf {X}}}=[X \; Y \; Z \; 1]^T\) (Hartley and Zisserman 2004). Using augmented vectors a geometric transformation, consisting of a rotation and a translation, is simply written as a matrix multiplication \([R \ {\mathbf {t}}] \tilde{{\mathbf {X}}}\), where R is a rotation matrix and \({\mathbf {t}}\) is a translation vector.

With these notations, the mapping function can be written in the following form that is widely used in projective geometry:  (Hartley and Zisserman 2004):Here K is the \(3 \times 3\) camera calibration matrix and \([R \ {\mathbf {t}}]\) is a \(3 \times 4\) matrix, with R the \(3 \times 3\) rotation matrix, and \({\mathbf {t}}\) the \(3 \times 1\) translation vector. The matrix K in Eq. 3 has the following form:where \(\alpha _x= f r_x\) and \(\alpha _y=f r_y\) are scale factors, with f the focal length of the lens in \(\mathrm {mm}\) and \(r_x\) and \(r_y\) the pixel pitch of the sCMOS sensor in \((\mathrm {px}/ \mathrm {mm})\). s is the pixel skew, characterizing the angle between the x and the y pixel axes, and \([p_x \ p_y]^T\) are the coordinates of the principal point at the intersection between the optical axis and the dewarped image plane (Hartley and Zisserman 2004). The elements of K are often referred to as the intrinsic camera parameters, representing the characteristic properties of the camera itself (Hartley and Zisserman 2004), while \([R \ {\mathbf {t}}]\) are referred to as the extrinsic parameters representing the position of the camera with respect to the world coordinate system (Fig. 2).

Together, K, R, \({\mathbf {t}}\) define the mapping function F of Eq. 3 and have to be determined for each of the cameras separately. In the following, we use the superscript \(c=1 \cdots 4\) and the notations \(K^{\text {c}}\), \(R^{\text {c}}\) and \({\mathbf {t}}^{\text {c}}\), when we distinguish explicitly between the different cameras. We omit the superscript for clarity when no distinction between the cameras is needed; the details of the algorithms can be found in Appendices.

First, the camera matrix K is determined for each of the four separate cameras by calibrating a single camera using the method developed by Zhang (2000). We consider a local coordinate system \({\mathbf {X}}^o=[X^o \; Y^o \; Z^o]^T\) attached to the planar checkerboard in the object domain, where \(Z^o=0\) corresponds to the plane of the checkerboard. In this coordinate system, the J nodes at the intersections between the gridlines have known coordinates \({\mathbf {X}}^o_j=[X_j^o \ Y_j^o \ 0]^T\). The nodes \({\mathbf {X}}^o_j\) are mapped to their images \(\hat{{\mathbf {x}}}_j^n\) following the formalism used in Eq. 3. This transformation can be written as \(p( K [R^n \ {\mathbf {t}}^n] \tilde{{\mathbf {X}}})^o_j\), where the rotation matrix \(R^n\) and translation vector \({\mathbf {t}}^n\) characterize the position of the checkerboard in the object domain. Geometrically, this corresponds to a rotation and translation of the checkerboard plane in the object domain, followed by a projection on the image plane. The camera calibration matrix K and the positioning of the checkerboard, characterized by \(R^n\) and \({\mathbf {t}}^n\), are determined following Zhang (2000) and are refined by minimizing the following functional:

To complete the calibration over the multiple cameras, we determine the rotation \(R^{\text {c}}\) and the translation \({\mathbf {t}}^{\text {c}}\) representing the position of each of the cameras in the world coordinate system. \(R^{\text {c}}\) and \({\mathbf {t}}^{\text {c}}\) correspond to the extrinsic camera parameters described in Sect. 2.3 and a first estimate is deduced directly from the calibration of single cameras performed in the previous step. Selecting two different calibrated cameras, we use the Kabsch algorithm (Kabsch 1976) to estimate the relative positions of the two cameras by comparing the positions of the checkerboards, \(R^{n,{\text {c}}}\) and \({\mathbf {t}}^{n,{\text {c}}}\), for these two views. Considering all camera pairs, we determine the relative positions between all the views and deduce a first estimate for \(R^{\text {c}}\) and \({\mathbf {t}}^{\text {c}}\); see Appendix B for detail. Next, for each of the N calibration images, we estimate position \(R^n\) and \({\mathbf {t}}^n\) of the checkerboard in the world coordinate system. We do this by averaging the position estimates \(R^{n,{\text {c}}}\) and \({\mathbf {t}}^{n,{\text {c}}}\) obtained from the four separate single-camera calibrations.

In the last step, we compute the final values for \(K^{\text {c}}\), \(R^{\text {c}}\) and \({\mathbf {t}}^{\text {c}}\) by minimizing the reprojection errors from all cameras and calibration images. For this optimization step, we use as initial conditions, the camera matrices \(K^{\text {c}}\) obtained from Sect. 2.4, the positions \(R^{\text {c}}\) and \({\mathbf {t}}^{\text {c}}\) obtained from the Kabsch algorithm and the estimates of the checkerboard positions \(R^n\) and \({\mathbf {t}}^n\). We define the reprojection error \(\varepsilon ^{n,{\text {c}}}_j\) for each of the N calibration images and in each camera view as

The reprojection error \(\varepsilon ^{n,{\text {c}}}_j\) is normalized by \(\hat{{\mathcal {A}}}^{n,{\text {c}}}_j\), which is the area in pixels of a single tile on the checkerboard projection in the dewarped image plane (see Appendix C). Hence, \(\varepsilon ^{n,{\text {c}}}_j\) provides a measure of the error relative to the size of the checkerboard. This error is relatively independent of the location of the calibration target within the large depth of field and the positions of the cameras and, therefore, independent of the apparent size of the checkerboard in the image plane. The final parameters for the calibration function F for each view are obtained from the minimization of the following summation:

The resulting camera calibration is shown in Fig. 3.

First, we consider the numerical values of the extrinsic camera parameters obtained from a calibration, see Table 1. As discussed in Sect. 2.3, these parameters characterize the spatial position and orientation of each camera. The reconstructed positions of the cameras are in agreement with the experimental setup, with cameras 4 and 1 positioned above cameras 3 and 2, and cameras 4 and 3 positioned on the left-hand side while cameras 1 and 2 are located on the right-hand side (as shown in Fig. 1), see Table 1. Furthermore, the relative distances between cameras are also in agreement with the experimental scene as we find the horizontal distance between cameras \(\varDelta W\approx 5.6 \; \mathrm {m}\) and a vertical distance between cameras \(\varDelta H \approx 1.2 \; \mathrm {m}\), as deduced from Table 1. Likewise, the reconstructed camera orientations are consistent with the cameras on the right-hand side oriented with a positive angle \(\alpha\), while the cameras on the left-hand side are oriented with a comparable negative angle \(\alpha\).

In addition to reconstructing the position of the camera, the calibration procedure reconstructs the intrinsic camera properties, which we compare to the specification of the instrumentation. The coefficients of the camera calibration matrix K of Eq. 4 are provided for each camera in Table 1. We focus on the values of the focal length of the lenses and deduce an effective focal length \(f_{\mathrm {eff}}\) directly from the coefficients aswhere r is the resolution of the camera sensor in \(\mathrm {px/mm}\), which is known from the camera specifications, \({\tilde{J}}\) represents a correction factor for the image expansion due to optical distortion (see Appendix D) and \({\tilde{n}}=n_{\mathrm {air}} / n_{\mathrm {water}}\) corrects for the magnification due to refraction at the air/water interface. Our calibration yields values for the effective focal length of the four cameras of \(f_{\mathrm {eff}} = 23.73 \pm 0.82 \; \mathrm {mm}\). This reconstruction of the focal length lies within 1–2 \(\%\) of the actual focal length \(f_{\mathrm {lens}} = 24 \; \mathrm {mm}\) of the lenses that were used. Hence, we find that both the extrinsic and intrinsic camera parameters deduced from our calibration procedure are in agreement with the dimensions and characteristics of our experimental setup.

The camera calibration is obtained by minimizing the sum of the squared reprojection errors \(\varepsilon _j^{n,{\text {c}}}\) over all four cameras, all N calibration images and all nodes on the checkerboard, see Eq. 7. The camera calibration converges to low values for \(\varepsilon _j^{n,{\text {c}}}\) see Table 1. Here, the average normalized reprojection error is on the order of \(\varepsilon _j^{n,{\text {c}}} \sim 2 \%\) of the size of a checkerboard tile, which corresponds to an error of less than \(1 \ \mathrm {cm}.\)

The camera calibration requires a minimum of two non-coplanar checkerboard images (Zhang 2000). Increasing the number of calibration images increases the sampling of the measurement volume and, therefore, improves the reliability of the calibration. We further characterize the performance of the method as a function of the number of calibration images used, by randomly selecting different subsets of checkerboard images.

We first consider the effective focal length \(f_{\mathrm {eff}}\) of Eq. 8 deduced from the matrix K to characterize the quality of the position in space of the checkerboards and the cameras. In Fig. 4, we select different subsets of calibration images ranging from \(N=2\) to \(N=50\) images and compute \(f_{\mathrm {eff}}\) from the associated calibration. Figure 4a represents the ratio between \(f_{\mathrm {eff}}\) and \(f_{\mathrm {lens}}\) as a function of N. With a low number of \(N=2\) to 10 calibration images, the ratio \(f_{\mathrm {eff}} / f_{\mathrm {lens}}\) is far from one and the focal length is under- and over-estimated by \(50\mathrm {\%}\), indicating an unreliable calibration. Increasing the number of calibration images to \(N=15\) shows that the estimated focal length \(f_{\mathrm {eff}}\) approaches \(f_{\mathrm {lens}}\) and represents a clear improvement of the calibration. Further increasing N beyond \(N=15\), the ratio \(f_{\mathrm {eff}} / f_{\mathrm {lens}}\) does not significantly converge further (Fig. 4a).

Second, we consider the reprojection errors \(\varepsilon ^{n,{\text {c}}}_j\) in Fig. 4b. For a low number of \(N=2\) to 10 calibration images, the average reprojection error is as high as \(\varepsilon ^{n,{\text {c}}}_j \sim 50\) to \(60 \ \%\). Increasing the number of calibration images from \(N=15\) to 50 shows an additional decrease of the normalized reprojection error from \(\varepsilon ^{n,{\text {c}}}_j \sim 5\) to \(2 \ \%\) (Fig. 4b) and inset. This shows that 15 calibration images are sufficient to achieve a valid calibration. Further increasing the number of calibration images improves the convergence for the camera calibration while the ratio \(f_{\mathrm {eff}} / f_{\mathrm {lens}}\) remains at a value close to 1.

For this, we use the nodes identified at gridline intersections on the N calibration images. We proceed by evaluating the four optical rays associated with each node of each calibration image. We then triangulate the location of each node by finding the point \({\mathbf {X}}_j^n\) in the object domain that minimizes the sum of the squared distances from the point to the four optical rays. We report for each node the skewness \(s_j^n\) as the average distance from the triangulated location \({\mathbf {X}}_j^n\) to the four optical rays (see Appendix E for more detail).

The calibration images were acquired over the entire depth of the tank and used to characterize the spatial accuracy, by reporting the skewness as a function of the position along the Z-axis of the world coordinate system (Fig. 5a). We find the skewness to be mostly uniform over the large depth of the measurement volume \(Z=5-25 \ \mathrm {m}\). Our calibration yields a high spatial accuracy characterized by an average skewness of less than \(1 \ \mathrm {cm}\). Only a slight increase in the skewness can be observed towards the back of the aquarium although the average skewness still remains below \(1 \ \mathrm {cm}\). This small increase is due to the decrease in spatial resolution and the decrease in angles between the optical path from the different views.

The variation in skewness over the height and width of the tank are represented in Fig. 5b, c. Figure 5b is a map of the skewness in a XY-plane at the front of the aquarium, while Fig. 5c provides a similar map at the back of the aquarium. Both are qualitatively similar and the skewness remains small over the depth of the tank. For reference, Fig. 5d, e shows the sampling density, which indicates the number of checkerboard images that were used at a location to compute the calibration. It is noteworthy that the center of the tank was better sampled than the sides and the bottom. We find that the skewness, and hence the spatial accuracy, is relatively constant over a large part of the measurement volume, but increases towards the edges of the tank. This is likely a result of the lower sampling away from the camera center, as well as unresolved optical distortions at the edges of the image plane.