Monocular 3D reconstruction of sail flying shape using passive markers

We briefly introduce the terminology of some parts of the boat that we refer to in this work. These parts are indicated in the boat diagram presented in Fig. 1:

The first step of our method is to place markers on the sail. We opted for augmented reality markers printed on waterproof adhesives, fixed on one side of the sail surface. We previously compared the detection robustness between two libraries: ARToolKit [18] and ArUco [13]. ArUco presented the best results for our tests. ArUco markers are square and binary, and the library allows creating a configurable markers dictionary by defining the number of markers and the number of bits for the inner pattern and the border. Markers are generated maximizing the inter-marker distance and the number of bit transitions. We performed experiments with different numbers of bits for the inner pattern and border size and achieved the best results for our application with 9 internal bits and 1 bit for the border. From the markers, it is possible to extract their image contours, 3D center positions and orientations in the camera’s coordinate system (Fig. 3). The extracted data are the input for the next steps, as described in the following sections.

For naval architecture purposes, it is important to retrieve horizontal sections along the sail, since they convey well its general shape. For simulation and design purposes, the sail surface is mainly defined by horizontal curves [32]. Thus, markers were placed forming horizontal stripes in strategic positions pointed out by the naval architects. We also placed markers along a vertical line on the sail, which is important to get an orthogonal orientation of the sail and verify the coherence among the horizontal stripes. Moreover, it is interesting to have a rigid reference for the sail’s markers in order to properly capture the sail behavior over time. For this purpose, some markers were fixed on the hull.

Once the markers are fixed, their positions on the sail allow to establish an adjacency map. This map defines a graph as a set \({ G } = \{{ V }, { E }\}\), where \({ V } = \{v_{i}~|~v_{i} \text { is the marker with index } i\}\) is the vertices set and \({ E } = \{e_{ij}~|~e_{ij}\) is the edge connecting the vertices \(v_i \text { and } v_j \}\) is the edges set. We established the adjacencies as shown in Fig. 4: markers on horizontal lines are connected to the markers on the right and left; markers on the vertical line are connected to the markers above and below; and markers on the hull are connected to all adjacent markers. This graph is useful to verify the topological coherence and remove duplicate detected markers (Sect. 3.3). We define the topological distance between two vertices as the number of edges connecting them. The smaller the number of edges between two vertices, closer they are. For example, in Fig. 4 the vertices \(v_j\) and \(v_k\) are the nearest vertices to \(v_i\) because only one edge separates these vertices. In other words, \(v_j\) and \(v_k\) have distance 1 to \(v_i\). The next nearest vertex is \(v_l\), which has distance 2 to \(v_i\).

It is important to emphasize that the total number of markers depends on the project (design) and the analysis objectives. More markers provide a more detailed graph and reconstruction, and redundancy may help in overcoming detection errors. The markers fixation should be performed carefully to avoid losing them during sailing, and the markers should tolerate some amount of water, wind and sail deformations. Since the adhesive glue may not be enough to avoid these issues, we also fixed scotch tape along the markers border. However, if a marker does fall off, our method considers that this marker was not detected and the reconstruction proceeds normally. For our tests, the fixation of about 122 markers took around two hours.

The next step of our method is to capture a video of the marked sail in real sailing conditions. We use a single camera placed in another boat that follows the target boat from a distance of a few meters. Considering the Finn class, three to five meters is enough to not affect the sail boat performance, retrieve the markers, and, at the same time, capture the whole sail surface. Alternatively, we could place the cameras inside the target boat. This setup has disadvantages, however, such as the need for more cameras in order to capture the whole sail surface [8, 21], and the perspective distortion of the images, especially on the sail top [10]. Positioning cameras in another boat allows to record the sail at a more perpendicular angle. It is a more generic and simple setup that can be used for a broader range of boats and can be arranged as to not interfere with the sailing of the tracked boat. The main challenge of capturing the sail is to keep the camera at a distance that allows a good marker detection while avoiding illumination problems.

Given a video frame f, for a detected marker whose index is k, its four corners \(\{{\mathbf {x}}_{k,1}, {\mathbf {x}}_{k,2}, {\mathbf {x}}_{k,3}, {\mathbf {x}}_{k,4}\}\) are extracted in image domain \({\varOmega }\subset {\mathbb {R}}^2\), while its center’s transformation (translation \({\mathbf {t}}_{k,0} \in {\mathbb {R}}^3\) and rotation \({\mathbf {R}}_{k} \in SO(3)\)) is recovered in relation to the camera’s coordinate system. \({\mathbf {R}}_{k}\) also defines the marker’s normal and tangent vectors, while its center position in camera space \({\mathbf {p}}_{k,0} \in {\mathbb {R}}^3\) is directly obtained from \({\mathbf {t}}_{k,0}\). Similarly, we can find the corners positions \(\{{\mathbf {p}}_{k,1}, {\mathbf {p}}_{k,2}, {\mathbf {p}}_{k,3}, {\mathbf {p}}_{k,4}\}\) by a rigid transformation of \({\mathbf {p}}_{k,0}\). Conversely, the image point of the marker’s center \({\mathbf {x}}_{k,0} \in {\varOmega }\) can be found by projecting \({\mathbf {p}}_{k,0}\) onto the image. Thus, for each marker we can define a matrix of 2D points in image coordinates:and a matrix of their respective 3D points in camera coordinates:Therefore, a marker of index k detected at frame f can be defined by the pair:Commonly, false positives arise during detection. Artifacts on the image may be confused with a marker, and markers may be mislabeled, as shown in Fig. 5. In order to simplify our process, we use markers with unique indices k, i.e., any marker detected more than once clearly indicates a detection error.

We identify and remove the duplicate markers using topological constraints, which are based on the graph defined in Sect. 3.1. For each index k and frame f, we have a set of candidate markers \({ C }_{k,f} = \{{ M }_{k,f}^i~|~{ M }_{k,f}^i\) is a candidate for marker \(k \text { at frame } f\}\). Initially, all markers with only one candidate, that is \(|{ C }_{k,f}|=1\), are marked as correct. Given a marker index k, such that \(|{ C }_{k,f}|>1\), for each candidate \({ M }_{k,f}^i\in { C }_{k,f}\), we compute the average distance in pixels (px) between its marker center \({\mathbf {x}}_{k,0}^i\) and the three topologically nearest vertices that are already marked as correct. If a marker is an outlier, we expect it to be far from its topological neighbors. For example, in Fig. 5, the incorrect detection of marker 30 is far from markers 28, 29, 31 and 32. Thus, the candidate with smallest average distance is selected as the marker with index k, and all other candidates are discarded. Even though this criterion is not fail proof, it works well because duplicate markers are rare in practice. After this initial selection, we have only a single candidate for each marker. Algorithm 1 shows the pseudocode of our duplicate removal algorithm. We further implemented another topological verification for the non-duplicate candidates to verify that they are really correct. However, we noted that this verification did not improve the reconstruction results. The two additional filtering steps applied during registration (Sect. 3.4.1) and reconstruction (Sect. 3.5) are more effective to remove outliers. Thus, we have chosen to handle only the duplicate markers in the detection step.

Thus, for each frame f, we define the set \({ D }_f=\{{ M }_{k,f}\}\) of markers detected and verified at frame f. Henceforth, when a marker of index i is discarded at a frame f, \({ M }_{i,f}\) is removed from \({ D }_f\).

The markers on the sail and the camera move independently over time. Their relative position changes constantly during recording, as illustrated by Fig. 6a. For each video frame f, we initially have a different coordinate system; therefore, we need to define a common reference system for all frames, as illustrated in Fig. 6b.

To perform the reconstruction, we define a central frame r, around which we intend to achieve the average sail configuration. Next, we select n frames before and n frames after r. These n frames do not need to be selected consecutively, since frames with small time differences are very similar and do not add much new information to the reconstruction. In fact, very similar frames may even cause numerical issues for the reconstruction. The spacing between frames depends on recording conditions such as boat velocity and video frame rate, and the criterion to select the frames will be detailed below. For now, without loss of generality, lets define the set that contains the selected \(2n + 1\) frames as:For each frame \(f\in { S }\), given its verified markers \({ M }_{k,f} \in D_f\), we need to find the rigid transformation that optimally aligns all the markers centers \({\mathbf {p}}_{k,0}\in {\mathbf {P}}_{k,f}\) denoted by \({\mathbf {p}}_{k,0}^{(f)}\) and \({\mathbf {p}}_{k,0}\in {\mathbf {P}}_{k,r}\) denoted by \({\mathbf {p}}_{k,0}^{(r)}\):where k represents all markers indices such that \({ M }_{k,f} \in { D }_f\) and \({ M }_{k,r} \in { D }_r\), \({\mathbf {R}}_{f,r} \in SO(3)\) is the rotation and \({\mathbf {v}}_{f,r}\in {\mathbb {R}}^3\) is the translation that align f’s reference system with r’s. Eq. (1) is a least square problem which can be solved by Singular Value Decomposition (SVD) [9]. It must be solved for each \(f\in { S }\), resulting in \(|S| - 1 = 2n\) rigid transformations.

Let:be the set of selected frames where the marker of index k was correctly detected. A marker needs to appear in a minimum number of frames \(\beta \) so that its position can be correctly optimized by the Bundle Adjustment (BA) algorithm [37] described in Sect. 3.5.1. To avoid optimization problems, if \(|{ Q }_k|<\beta \), \({ M }_{k,f}\) is removed from \({ D }_{f}\), for all \(f \in { S }\). The threshold \(\beta \) is our frequency tolerance, and its value will be discussed in Sect. 4. Thus, only markers of index k such that \(|{ Q }_k|\ge \beta \) will be reconstructed. The set of these marker indexes to be reconstructed is then defined as:After the frequency tolerance filtering, the updated sets \({ D }_f\) are used to estimate the mean positions \({\bar{{\mathbf {P}}}}_k\) of the marker k. Notice that up to this point all positions were computed using the frame r as reference. These mean positions are iteratively computed from the initial mean (iteration 0):where \(k\in { I }\). After computing this initial mean, we start an iterative algorithm to compute a weighted averaged position [9] for each marker \(k\in { I }\). For each iteration i, \({\bar{{\mathbf {P}}}}_{k}^i\) is given by:where \({\mathbf {W}}^i_{k,f}\) is a weight matrix defined as:where \(\displaystyle {w^i_{k,j} = e^\frac{||{{\mathbf {p}}'_{k,j} - {\bar{{\mathbf {p}}}}_{k,j}^{i - 1}}||}{\sigma }}\), for \(j=0,1,2,3,4\). Thus, \({\mathbf {W}}^i_{k,f}\) is a Gaussian weight matrix that favors points nearer to the average in the previous iteration. Points far from the average will have a decreasing weight and the process converges after few iterations [9]. At the end of this iterative process, we have a matrix:of the mean position of the markers points for each \(k\in { I }\). This weighted iterative estimation converges to a fair estimate by progressively penalizing points far from the mean. Thus, we can define a set:of the mean marker points positions. This estimate of mean points will be refined by the BA algorithm.

The sail videos registered throughout our experimental sessions are available at http://​www.​lcg.​ufrj.​br/​sail3D. In order to evaluate a more controlled environment, we recorded some videos with the sail ashore (Fig. 7a). This scenario allowed more control over the capture distance and the illumination. We recorded a total of 20 ashore sequences, including 4K, 2.7K and 12MP camera resolutions. After this controlled scenario, we captured the sail in a real sailing environment (Fig. 7b and c), totaling 28 sequences. Our original videos were divided in these two categories: ashore and sailing.

The original videos were split and classified into two main classes based on the capture distance to the sail: near or far. This division resulted in 39 clips with 4K resolution. Each clip presents particular features as listed in Table 2.

According to the reflection occurrence the clips can be classified as “Weak reflection” when the reflection obfuscates few markers or “Strong reflection” when many markers are obfuscated by the natural illumination. Figures 7b and c present examples of these situations. It is important to note that both reflection types can occur in the same clip. For future registrations, we can soften this issue using filters in the camera.

In some clips, the wind changes during the capture, modifying the sail shape. They were classified as “Wind change”. Fig. 8 shows three frames of the clip “near_4k_08.mp4” with wind changes.

Some clips were recorded from a great distance, which makes markers detection very difficult. These clips are classified as “Too far”. Furthermore, some clips were captured from an almost parallel angle in relation to the sail. They were classified as “Bad angle”. Ideally, the capture should have an angle as perpendicular as possible to the sail.

We used three evaluators to quantitatively assess our reconstruction:For each clip in our dataset, we compute the reconstruction centered in several frames. The area error \(e_a\) was computed for each reconstructed marker and the reprojected error \(e_r\) was computed for each marker point. In order to achieve a general evaluation of the reconstruction and fine tune the parameters, we computed the statistics of the errors: average, standard deviation, median, minimum and maximum.

In Sect. 3.5, we described our iterative weighted average of the markers points. This average uses a Gaussian weight with parameter \(\sigma \). We tested some \(\sigma \) values in the interval [0.1, 5.0] and observed its influence on the error evaluators. We noticed that \(\sigma \) is not sensitive for the reconstruction ratio, and values \(\sigma \ge 0.6\) do not disturb \(e_a\) and \(e_r\). Thus, we set \(\sigma =0.6\) for our experiments. We also analyze the threshold \(\beta \) for the frequency filter by varying its value between \(10\%\) and \(40\%\). Small values increase the number of reconstructed markers, but also increases \(e_a\) and \(e_r\). The value \(\beta =30\%\) presented the best trade-off between reconstruction ratio and errors. We performed 30 iterations, which were enough for convergence in Eq. (3).

For the RANSAC strategy described in Sect. 3.4.1, we need to define a threshold for considering a point as an inlier. In our case, this value is the acceptable distance between the registered point and the point in the central frame. We tested values between 50 and 300 mm and noticed that 100 mm presents good results considering \(e_a\) and \(e_r\). Values below 100 mm slightly decrease the errors but considerably reduces the number of reconstructed markers. On the other hand, values above 100 mm increase the reconstruction ratio at the cost of increasing errors.

The clip “near_4k_17.mp4” is the longest sequence recorded in sailing conditions from a reasonable distance. This clip presents a good sail stability, parts with weak and strong reflection. Thus, it is considered the best baseline for the dataset reconstruction and all results presented in this section use this clip. It will be used in the next subsections for comparing the results with difficult clips.

Figure 9a presents a visualization of the reconstruction centered in the frame 457 from two view points. It shows all points for each reconstructed marker.

As previously mentioned, we compute the reconstruction centered in several frames. In order to statistically evaluate the behavior for the entire clip, several statistics are computed as follows:Our frame selection procedure described in Sect. 3.4.1 depends on three parameters: n (number of selected frames), s (skip size) and m (neighborhood size). We varied n in the interval [5, 50], which results in varying \(|S|=2n+1\) in the interval [11, 101]. Figures 10a, b and c show the resulting statistics for the three evaluators. Figure 10 presents the mean statistics in function of total number of selected frames |S|. Notice that the error decreases with the increase in selected frames, but after 41 frames the variation is small. The average error was around 250 \(\mathrm{mm}^2\), which represents \(2.5\%\) of the marker area and the maximum around 1000 \(\mathrm{mm}^2\), which is \(10\%\) of the area. For the reprojected error, the error slightly increases as the number of frames increases. This is expected since we have more frames to be adjusted by the Bundle Adjustment. Despite this increase, the maximum error slightly changes after \(|S|=50\), stabilizing at around 2 pixels. The reconstruction ratio decreases with the total of selected frames, varying from 56.9% for 11 frames to 40.1% for 101 frames, i.e., as more frames are used for the reconstruction, fewer markers are reconstructed. The decrease is more accentuated after \(|S|=31\). Thus, we can summarize the analysis of Fig. 10a, b and c as:Based on this analysis, we opted to use \(n=20\), i.e., selecting \(|S|=41\) frames for reconstruction. This value ensures small area errors without penalizing the reconstruction ratio.

Figure 11a, b and c presents the evaluation results in regards to the skip size s. By analyzing Fig. 11a, we note that small values perform poorly. This is explained by the frames similarity, since the frame rate is high in relation to the scene motion. If s is increased, a longer clip is necessary to select the frames, since the interval between two selected frames will be larger. But keeping the sail stable for an extended period is usually not a trivial task. Furthermore, Fig. 11c shows that the reconstruction ratio decreases by increasing s. Regarding the reprojected error (Fig. 11b), the behavior was similar to Fig. 10b, i.e., the error slightly increase by increasing s. The explanation is also similar. Since the Bundle Adjustment should adjust frames with more variability between them, it is expected an increase in the mean error to adjust all frames. We observed that \(s=10\) is a good choice for videos recorded at 30 FPS.

We also observed that the neighborhood size m has no significant impact on the errors, but increasing m also increases the reconstruction ratio. This occurs because more frames are used to find inliers to register with the central frame. The value of m should not be greater than s to avoid overlapping the intervals. We found that \(m=5\) is a good choice for \(s=10\). Considering the values of \(n=20\), \(s=10\) and \(m=5\), we can estimate a minimum video length. In the worst case for these values, all frames are selected with spacing of 15 frames. To select 41 frames (\(n=20\)) at least 20 s of video at 30 FPS is necessary. However, larger videos allow us to also vary the central frame.

Figure 12a presents the histogram of the markers area by using \(n=20\), \(s=10\) and \(m=5\). This histogram considers the area of the markers reconstructed in all frames. Notice that the markers area tend to be close to the expected value of 10,000 \(mm^2\).

Figures 10c and 11c present values smaller than 60% for reconstruction ratio. It is important to clarify that the values presented in these figures are the average reconstruction ratio for the all clip frames. Figure 12b shows the reconstruction ratio for each frame from 202 to 1502 for the clip using \(n=20\), \(s=10\) and \(m=5\). The reconstruction ratio is around 70% before the frame 600, i.e., before 20 s of the video. After this frame, the ratio decreases, only rising again near to the clip end. This behavior is justified by the increase in the capture distance which difficults the markers detection.

Figure 13a shows the reprojection of the reconstructed points (Fig. 9a) on the central frame 457. The points are projected on the expected positions, i.e., at the center of the markers.

Figure 14 presents the rigid motion of the sail markers in relation to the hull markers. This motion is computed by aligning two reconstructions centered in different frames in relation to the hull markers. The distance between reconstructions is 15 frames, i.e., 0.5 second (frames 457 and 472). Notice that the motion occurs mainly on the sail top, which is coherent with the sail dynamics and confirmed by domain experts as the expected behavior.

To evaluate the precision and accuracy of our method in a controlled environment, we fixed 33 \(80\times 80\) mm markers in a flexible plastic surface. Consecutive markers in a row are separated by 150 mm (Fig. 17). The surface was fixed on a slightly cylindrical wood frame. We recorded 48 videos in 4K resolution of this pattern under two situations: static (24 videos) and with wind generated by a fan (24 videos). The camera was slowly moved in all axis, some videos at 2 and some videos at 4 meters from the surface, to represent motion.

We applied our method to reconstruct the surface points using our best parameters for sail reconstruction (\(n=20\), \(s=10\) and \(m=5\)). We performed 400 reconstructions centered in consecutive frames for each video. The distance of horizontally adjacent markers and the markers area were computed for each reconstruction. The statistics of area, distance and respective errors using all 400 reconstructions of the 24 videos in each situation is shown in Table 3.

Table 3 shows that the error for the detected distance between markers was below 3% of the expected value and the area error was around 1% of the expected value. This setup is useful to assess the averaging properties of our method, by using the same parameters tuned for sailing conditions. Notice that the area error average is high if compared to the area average, which is fairly close to 6400 \(mm^2\). This is due to mistakenly detected marker areas (outliers), leading to a heavy-tailed distribution. However, the standard-deviation can be reduced by filtering out the outliers by thresholding.

Finally, we performed an experiment to evaluate the reconstruction against curvature variation. For this purpose, we used four cylindrical surfaces with different radii. For each one, we used a pattern of 15 markers, varying their dimensions and inter-marker distances to better fit the surfaces. We also adjusted the RANSAC threshold accordingly due to different scales, but all other parameters were fixed. In particular, we used our best parameters for the sail reconstruction (\(n=20\), \(s=10\) and \(m=5\)). Table 4 shows the settings for each surface, and Fig. 18 illustrates the settings for the surface with largest radius.

For every surface, we recorded a video in 4K resolution by moving the camera along all axes at a distance of approximately 2 meters from the surface. We then performed 1000 reconstructions centered in consecutive frames. Since the cylinder radius and the geodesic distances between markers are known, we calculated the real euclidean distance between markers and compared them to the reconstructed data. We also calculated the real planar area formed by the markers’ corners and compared to the estimated markers. The results are presented in Table 5. The average distance error was less than 2% for all cases, and all area errors were below 3%, which is compatible, and predominantly better, than the approximately 2.5% for the sail reconstruction. Figure 19 shows the reconstructed points around the ground truth cylinders.

These experiments show the ability of our method to reconstruct surfaces with different curvatures without affecting the performance. It is important to mention that the distance verification takes into account distances between non-adjacent markers, where the difference between the geodesic and euclidean distances are larger. We even included the distance between markers at opposite extremities of each row. Moreover, all cylinders provide a curvature much larger than any expected configuration of the sail, better supporting our results for the sail reconstruction.

As described in Sect. 3, our reconstruction method is composed of five steps: markers fixation, capture, detection, registration, and reconstruction. Each step takes a different time to be performed and depends on different factors. As mentioned in Sect. 3.1, markers fixation takes about two hours, even though we expect this time to reduce significantly with more practice. The video capture depends on how long we want to analyze the sail behavior. However, as exposed in Sect. 4.3, less than 30 s of footage is already enough to achieve a suitable reconstruction.

The detection time depends on the video resolution and how many frames are analyzed for the reconstruction. For a reconstruction using our best parameters (\(n=20\), \(s=10\) and \(m=5\)), we examine at most 601 frames. In the worst case, the distance between the reference frame and the first and last selected frames will be of 300 frames, since we have in this case 20 frames separated by 15 frames (10 of the skip size and 5 of the neighborhood). Notwithstanding, the detection is performed before the frame selection step, thus, we need to detect markers in all 601 frames. The detection of 601 frames for a 4K video takes around 221 s in a \(\text {Intel}^{\circledR }\) Core™ i7-5500U 2.40GHz processor with 8GB of memory. It is important to note that the detection needs to be performed only once for each video, as it can then be reused to compute reconstructions centered at different frames and using different parameters.

The registration and reconstruction steps depend on the number of markers. For the reconstruction of frame 457 of video ”near_4k_17.mp4” (Fig. 9a), the registration and reconstruction took 21 and 41 s, respectively. This reconstruction is composed of 92 markers (460 points). The total processing time was 283 s, considering detection, registration and reconstruction.

We performed some extra tests by artificially removing some markers. In this case, markers with odd indexes were discarded. We analyzed the quality and runtime of this sparser reconstruction. As expected, the area and reprojected error were comparable to the complete reconstruction for video “near_4k_17” (Fig. 20), as presented in Sects. 4.3.1, 4.3.3 and 4.3.2 for issue cases. This means that a sparse reconstruction can present a satisfactory result and it is possible to reduce the number of markers according to the application needs. In terms of runtime, the sparser reconstruction took 10 s for the registration and 25 s for the reconstruction. However, the detection step time did not present a significant reduction since the ArUco library still runs through the entire image to detect markers, i.e., it depends on the image resolution and not on the number of markers. Thus, using fewer markers obviously reduces the fixation time, but does not improve significantly the processing time and does not hinder the reconstruction.

The results for the reconstruction of the clip “near_4k_17.mp4” were submitted to domain experts and experienced sailors for analysis. Figure 21 shows the profiles of the sail sections generated by naval engineers using the ANSYS [25] software from the markers centers of our reconstruction data. They observed that, in general, the shape of the profiles of the sail sections is very satisfactory. However, some distortions are observed near the boom (in red). Nonetheless, it is not clear if these are reconstruction errors or the sails actual shape since this region is subject to significant interference from the mast and the boom. Furthermore, some misalignment between the profiles is observed. The same observation was formulated about the initial and final points of the profiles. We noticed that these misalignments result from the actual markers positioning on the sail. Therefore, the per points reconstruction quality was considered satisfactory to generate the sail shape. Nevertheless, it was suggested that additional information about the sail bounds would entail more useful reconstructions for simulation and design evaluation purposes, and a more careful positioning of the markers would also increase the profile reconstruction quality.