Fast Estimation of Loader’s Shovel Load Volume by 3D Reconstruction of Material Piles

Image feature matching, a critical component of 3D reconstruction methods based on stereo vision, has a direct effect on the accuracy and computational efficiency of 3D reconstruction and volume estimates. Stereo matching algorithms are often divided into three categories: global stereo matching algorithm, semi-global stereo matching algorithm, and local stereo matching algorithm. Global stereo matching offers greater matching accuracy, but its applicability in 3D reconstruction is severely constrained by its high computational cost and memory usage. The semi-global stereo matching algorithm has been proposed and is widely used to increase the computational efficiency of the global stereo matching algorithm. Semi global block matching (SGBM) algorithm [16] is one well-known example. Its basic concept is to first determine the disparity of the image and then establish an energy generation value function related to the global image on the disparity map in accordance with the smoothing constraint criteria of the pre-selected scan. The method has higher matching accuracy and uses less computing power than the global stereo matching method. However, the algorithm still struggles to meet the requirements of real-time updating of earthwork volume estimation during loader shovel loading operations due to the complicated, real-time, and dynamic terrain environment. Local feature point stereo matching, in contrast to the approaches mentioned above, determines the local optimal matching cost by detecting a constrained set of pixels and is popular due to its superior computing efficiency.

The Seale invariance feature transform (SIFT) algorithm [17], speed up robust features (SURF) algorithm [18], and ORB algorithm [19] are the three primary traditional feature-matching algorithms. The improved oFAST and rBRIEF algorithms comprise the ORB algorithm [20, 21]. In this method, the key points in the image are identified using oFAST corners to detect pixels that differ considerably from the surrounding pixels. Subsequently, the rBRIEF description for each keypoint is calculated. This algorithm, a classic example of a binary description technique, outperforms the SIFT and SURF algorithms in terms of computing efficiency. Deep learning-based image processing techniques are also widely used to address image matching issues in addition to traditional image matching methods based on manual labeling. A large number of deep learning networks have also been used to extract image features. For example, Chaudhuri et al. [22] proposed a new architecture based on deep neural networks to solve the cross-modal information retrieval problem in remote sensing, which can learn a distinguishable shared feature space for all input modes, and is suitable for information retrieval with consistent semantics. Li et al. [23] proposed a large-scale remote sensing image retrieval method based on deep hash neural networks (DHNN), which can automatically learn feature extraction operations and feature hash mapping under the supervision of labeled samples. Furthermore, many deep learning-based feature matching methods [24‐26] apply graph neural networks (GNNs) to aggregate neighborhood information and form a structured representation of nodes and edges, showing good matching performance.

Deep learning-based methods can extract subtle low-level features and abstract high-level features, which can describe features more accurately. Deep learning networks, on the other hand, typically have complex structures and high computing costs, and they cannot meet the needs of real-time computing without graphics processing unit (GPU accelerators). Traditional feature description methods based on manual labeling, such as the ORB algorithm, have significant advantages in computing speed and cost, making them more suitable for 3D reconstruction of earthwork material piles and shovel loading volume estimation in real-time. However, this method has the following drawbacks: 1) The ORB algorithm employs the oFAST algorithm to detect feature points by comparing a pixel's gray value to its surroundings. This comparison allows the program to identify feature points in the image that are more noticeable than surrounding pixels. However, the algorithm employs an arbitrarily selected global fixed threshold, causing the detected feature points to be excessively clustered or even overlapped in areas with obvious features and excessively sparse or even empty areas without obvious features, resulting in a loss of information in some image areas. 2) The ORB algorithm describes the feature points using an improved rBRIEF algorithm. Due to the low feature description ability of the binary description method, it is difficult to obtain more correct matching feature points in the feature point matching phase, which seriously affects the accuracy of 3D reconstruction and volume estimation of the material pile.

In summary, it is necessary to study the feature matching method with higher computing efficiency and stronger robustness to realize the 3D real-time reconstruction of the material piles in the earthwork operation of the loader in the face of the characteristics of the material piles with an irregular shape and no surface texture, as well as the real-time dynamic change of the material pile surface characteristics during the earthwork operation.

Stereo vision technology has been widely used for the 3D measurement of objects in different fields, such as industrial manufacturing [27‐29], agriculture and fishing [30, 31], and aerospace [32, 33]. Research on earthwork volume measurements using visual techniques is gaining increasing interest. Bügler et al. [3] estimated the production efficiency of engineering activities by determining the entire amount of earth that had been dug using photogrammetric technology and combining it with video data. Borthwick et al. [34] installed a stereo camera on the large arm of a mechanical excavator to shoot the truck bucket at a distance and calculated the load volume by constructing a 3D surface model before and after the truck loaded materials. Yakar et al. [35] rectified the point cloud using the placed control points, then, based on the corrected point cloud, constructed a grid 3D model of the gravel pile and computed the volume. The aforementioned study may be used to plan the overall productivity of earthwork; however, it is challenging to evaluate operational efficiency in real time. Anwar et al. [7] proposed the use of stereo vision to estimate the volume of materials in an excavator bucket and verified the effectiveness of the method through simulation and field tests. Guevara et al. [8] used a binocular stereo camera to construct a 3D point cloud on the material surface of a bulldozer bucket, and used the Alpha-shape algorithm of Delaunay triangulation to estimate the effective shovel load of the bucket. Lu et al. [9, 10] developed a new perception system based on the stereo vision perception method, as well as advanced technologies such as point cloud registration, splicing, and surface interpolation, to realize the 3D point cloud reconstruction of materials in the loader bucket and accurate estimation of shovel loading. The aforementioned study produced an accurate assessment of the volume of materials in a single bucket during earthwork, offering reliable assurance for real-time estimation of earthwork volume and real-time evaluation of operational efficiency. However, the majority of current investigations are restricted to measuring the amount of materials in containers, such as buckets. This approach makes it difficult to employ when the bucket size varies because it necessitates the construction of an in-advance 3D model of an empty bucket. Fu et al. [36] suggested a fast estimation method for landslide deposit volumes by the 3D reconstruction of smartphone images and interpolation of the bottom surface considering the steepest gradient. This method resolves the issue of estimating the volume of irregular landslide deposits. However, it is not appropriate for the dynamic change in earthwork that occurs before and after loading in real time.

In summary, the method of single-shovel volume measurement by constructing a 3D point cloud model of the bucket and the material in the bucket needs to build a 3D model of the empty bucket in advance, which has a strong dependence on the bucket model and poor universality, and is difficult to apply to a situation where the bucket has different specifications and sizes. As a result, research on the measurement method of loader single-shoveling volume based on 3D reconstruction of material piles provides a novel idea for real-time project progress and operational efficiency evaluation.

The overall research idea of this study, which mainly includes data acquisition and processing, feature matching and correction based on QORB algorithm, dense 3D reconstruction of material piles based on MAPM, and volume estimation of the shovel load, is shown in Figure 1. First, the camera was calibrated according to the acquired binocular image to obtain the internal and external parameters of the camera, and then the image was calibrated on the same polar line. Second, a feature matching and correction method based on QORB algorithm is proposed to improve the feature point detection uniformity and obtain more correct matching feature points. Third, sparse 3D point clouds are generated by employing 2D Delaunay triangulation on the matching feature points in accordance with the determined camera calibration parameters. MAPM is built using the sparse feature points generated via matching as prior knowledge to estimate the parallax of the residual pixel points and achieve the densification of sparse point clouds. Finally, reference and matching models were constructed using 3D point clouds on the surface of the material piles before and after shoveling, respectively. The matching and reference models are then registered using the ICP algorithm. The point cloud was segmented according to the shovel loading boundary, and the shovel load volume was estimated using the Alpha-shape algorithm of Delaunay triangulation.

To achieve a fast estimation of the loader’s shovel load volume, this paper presents a novel dense stereo matching algorithm, namely, QORB–MAPM, by integrating the QORB algorithm and MAPM. This algorithm is combined with the principle of binocular stereo imaging to realize the fast matching of feature points and dense 3D reconstruction of material piles. Figure 2 shows the flowchart of this method, which involves three stages.

The detailed approach of each stage is discussed as follows.

According to this research method, it is necessary to reconstruct the surface model of material piles before estimating the shovel load volume. To verify the effectiveness of the dense stereo matching and 3D reconstruction method of the material piles proposed in this study based on QORB–MAPM algorithm, some image sets of the material piles were selected from the 8 shoveling tests, which were before shoveling, after two shovels, after four shovels, and after 6 shovels, as shown in Figure 9. In addition, in the experimental process, we also compared 3 current methods that have good performance and similar characteristics to this research method: the block matching method (BM), semi-global block matching method (SGBM) and traditional ORB-based dense matching and 3D reconstruction method (ORB–MAPM). To evaluate the performance of this research method in detail, in addition to the difference in feature point detection and matching, a comparison of traditional ORB and QORB algorithm ensures the consistency of other parts, such as feature matching and dense 3D reconstruction. For the sake of generality, this experiment used the VC++ language and the mainstream open-source computer vision library (OpenCV) for algorithm development.

First, feature points were detected. In this process, the traditional ORB algorithm and QORB algorithm in this study used oFAST to detect feature points. In this study, the improved ORB algorithm first divides the 2D space of each pyramid image into image blocks using the improved quadtree algorithm and then employs the oFAST algorithm to detect feature points for each image block. During the experiment, the number of feature points to be detected was set to 5000. The effects of feature point detection using the different methods are shown in Figure 10. It can be seen from the figure that the feature points extracted by the traditional ORB algorithms are too concentrated and overlapped in some areas with prominent edges and are too sparse or even appear as empty holes in areas with insignificant edges, resulting in the loss of local image feature information. The feature points extracted using the QORB algorithm combined with the improved quadtree algorithm in this study were more evenly distributed. The surface of the earthwork material piles with irregular surface morphology and no texture can more fully reflect the shape characteristics of the material piles, which is conducive to their 3D reconstruction of the material piles.

After feature point detection, feature point description and matching are required. In the feature point description, the traditional ORB algorithm and the QORB algorithm in this study use binary descriptors, therefore, the matching similarity is calculated using the Hamming distance to preliminarily select the matching points with the highest similarity. In the process of feature point matching, there may be incorrect matches of keypoints matched through specific similarity measurement relationships and corresponding search strategies. Therefore, the RANSAC algorithm was further introduced to purify feature points to filter out correct matches and incorrect matches. According to this research method, the homography matrix \({\varvec{H}}\) is calculated for the filtered matching feature points. The left image feature points in the incorrect matching are calculated through the homography matrix \({\varvec{H}}\) to obtain the corresponding right image mapping points. The right image mapping point can be taken as the corrected right image feature point to replace the right image feature point in the incorrect matching to obtain the final sparse matching feature points. The more sparse matching feature points can be obtained through feature point filtering based on the RANSAC algorithm and feature point mapping based on the homography matrix \({\varvec{H}}\) to improve the accuracy of 3D reconstruction and shovel load volume estimation of the material piles. The matching effects of the two methods are shown in Figure 11. Only the correctly matched feature point pairs are shown in the figure, as indicated by the green line. It can be seen from the figure that compared with the traditional ORB algorithms, the feature matching and correction method based on QORB algorithm in this study can obtain more correct matching feature point pairs, and the matching point pairs are more evenly distributed.

The obtained correctly matched feature points are then further generated into a sparse 3D point cloud in accordance with the camera calibration parameters and binocular camera stereo imaging principle. A dense 3D point cloud of material piles may be constructed by building a probability model in accordance with the obtained sparse 3D point cloud and triangular mapping. To verify the effectiveness of the 3D reconstruction of material piles based on QORB–MAPM algorithm proposed in this study, we compared it with three different methods: BM, SGBM, and ORB–MAPM. The 3D point clouds of the material piles reconstructed by different methods are shown in Figure 12. The figure shows how the ORB–MAPM, and QORB–MAPM methods in this study were able to reconstruct dense 3D point clouds that closely represent the actual material pile morphology on all test images. However, there were several noise points and holes in the 3D point cloud created by the BM and SGBM algorithms. This is because the pixel intensities are compared by BM and SGBM to calculate the generation value between the corresponding matching points. This pixel block-based descriptor performs has weak description performance, which increases the error in matching corresponding points and reduces the effectiveness of dense 3D reconstruction. However, in the dense 3D reconstruction method based on the MAPM proposed in this study, the triangular mesh calculated from matched support points as a priori information provides an accurate initial depth value for dense 3D reconstruction. Compared with pixel-block-based dense reconstruction methods, such as BM and SGBM, this method improves the robustness of the 3D point cloud. In addition, the 3D point clouds reconstructed by the ORB–MAPM algorithms have some point cloud holes and missing edge point clouds. Compared to this algorithm, the QORB–MAPM algorithm in this study retains more abundant material piles information, and the reconstructed 3D material piles point cloud model is more accurate.

The local features of the morphology of the material piles vary dramatically while the operation task proceeds in the actual application of earthwork, which raises expectations for the real-time reconstruction of the material piles and earthwork volume estimation. Therefore, we consider 5 groups of images before shoveling as an example to compare the average calculation time of different methods for the 3D reconstruction of material piles, as shown in Table 2. Figure 12 and Table 2 show that while the 3D material piles reconstructed by the BM algorithm include significant noise points, their computation time is the shortest. In contrast, the SGBM algorithm not only performs poorly in terms of 3D point cloud reconstruction, but also takes the longest time to complete. It is challenging to use these two methods in actual earthwork. Compared with the traditional ORB–MAPM method, the calculation time of the QORB–MAPM method in this study was slightly increased, but the reconstructed 3D point cloud was more accurate, and the overall performance was better.

The 3D point cloud model of the material piles before and after shovel loading is not consistent because of the movement and changes in the loader's position and attitude during the operation process. Therefore, the ICP algorithm is used to register the 3D point cloud model of the material pile surface before and after the shovel loading. Before the point cloud model registration, the dense point cloud was first down-sampled using the voxel-mesh method, and then the point cloud was roughly segmented at the boundary of the shovel loading area, extending outward to a certain range, as shown in Figure 13(a). This reduces the computational burden in the point cloud registration process and increases the computational efficiency of the algorithm. The blue and red point clouds in the image represent the 3D point clouds on the surface of the material piles before and after shoveling, respectively. It can be clearly seen from the figure that, owing to the movement and change of the loader, there is a large difference in the position and attitude of the point cloud models built before and after shoveling. The point cloud registered using the ICP algorithm is shown in Figure 13(b). The figure clearly shows that the position and pose of the 3D point cloud on the surface of the material piles before and after shoveling, registered by the ICP algorithm, are highly consistent. Finally, the point cloud model was finely segmented after registration according to the boundary of the shoveling area to retain the point cloud of the shoveling area, as shown in Figure 13(c).

To verify the effectiveness of the proposed method in estimating the shoveling volume, the 3D point cloud of the material piles before shoveling was used as the reference point cloud model in this experiment, and the 3D point cloud of the material piles after each shoveling was used as the registration point cloud model in 8 shoveling tests. The point cloud model of the shoveling area was obtained through point cloud registration and segmentation. Finally, the Alpha-shape algorithm of Delaunay triangulation in the MATLAB toolbox was used to calculate the shoveling volume. The alpha-radius parameter values for each of the 8 shoveling tests during the experiment were set to 0.36, 0.36, 0.36, 0.32, 0.32, 0.38, 0.38, and 0.38, respectively. Each shoveling test consisted of 5 trials, and the shoveling volume was measured using the measuring tool shown in Figure 8. The segmented point cloud model and corresponding volume calculation results of the shoveling area after 8 shoveling tests are shown in Figure 14(a) and (b), respectively. The experimental values are listed in Table 3. It can be seen from the table that the maximum relative error and standard deviation of the estimated volume of the 8 shoveling tests are 10.19 \(\mathrm{\%}\) and 0.2395 \({\mathrm{m}}^{3}\), respectively. A relatively large error occurs after the first and second shoveling, which is due to the small amount of shoveling that makes the change of material piles in the shoveling area insignificant, resulting in inaccurate point cloud model segmentation. In general, the shovel loading volume estimation method proposed in this study is highly accurate and can meet the requirements of accurate estimation of shovel loading volume in practical earthwork applications.

The volume estimation methods based on BM (BM VE), SGBM (SGBM VE), and ORB–MAPM (ORB–MAPM VE), which have good performance and similar characteristics to the proposed method, were compared and verified to further verify the effectiveness of the volume estimation method based on QORB–MAPM (QORB–MAPM VE) in this study. 4 methods were used to estimate the volume of 8 shoveling tests, with 5 shoveling trials conducted for each test. The average of the 5 volume estimates was calculated and compared with the actual value (represented by the magenta solid line) measured by the measuring tool, as shown in Figure 15. The figure shows that as the shovel loading volume increases, the volume estimation values from the different methods follow the same trend of change as the actual values. This strongly confirms the reliability of the volume estimation method for the loader’s shovel load based on the 3D reconstruction of the material piles proposed in this study. The test results are presented in Table 4. The maximum relative error and standard deviation of the estimated results of the 4 methods are as follows: BM VE (20.53% and 0.2511 m³), SGBM VE (14.31% and 0.6526 m³), ORB–MAPM VE (17.07% and 0.4768 m³), and QORB–MAPM VE (10.19% and 0.2395 m³). Therefore, the QORB–MAPM VE volume estimation method proposed in this study has higher estimation accuracy.

In practical earthwork applications, the bucket fill factor, which is an important parameter for evaluating the performance and operating efficiency of earthwork machinery, is often estimated using the ratio of the volume of materials in a single bucket to the rated volume of the bucket. Using the 8 shoveling test images from above, one can indirectly estimate the volume of materials loaded into the bucket in a single shovel by constructing a three-dimensional point cloud model of the material piles before and after a single shovel. The volume of 8 shoveling tests was estimated throughout the test using 4 different methods, and 5 trials were performed for each shoveling test. As illustrated in Figure 16, the average value of the 5 volume estimations was calculated and compared with the actual value (represented by the magenta solid line) measured by the measuring tool. It can be seen from the figure that the estimation results of the different methods are quite different. The volume estimation results of the QORB–MAPM VE method in this study were consistent with the actual values. The detailed results are presented in Table 5. It can be seen from the table that the maximum relative error and standard deviation of the estimated results of the 4 methods are: BM VE (61.42 \(\mathrm{\%}\) and 0.2515 \({\mathrm{m}}^{3}\)), SGBM VE (43.68 \(\mathrm{\%}\) and 0.6463 \({\mathrm{m}}^{3}\)), ORB–MAPM VE (33.82 \(\mathrm{\%}\) and 0.4959 \({\mathrm{m}}^{3}\)) and QORB–MAPM VE (15.08 \(\mathrm{\%}\) and 0.2705 \({\mathrm{m}}^{3}\)) proposed in this study. Therefore, the QORB–MAPM VE volume estimation method in this study also has a higher estimation accuracy in bucket fill factor estimation and has important engineering application value.

The average calculation times of the different methods from the 3 main stages of 3D reconstruction of material piles, point cloud registration, and Alpha-shape algorithm volume calculation are compared, as shown in Table 6, and calculate the total time consumption. It can be observed from the table that the 3D reconstruction of material piles and point cloud registration account for most of the total calculation time, and point cloud registration takes the longest time. The SGBM VE method requires the highest computation time, whereas the ORB–MAPM VE and QORB–MAPM VE methods used in this study require the least computation time. It is worth noting that in the earthwork operation process, loaders, excavators, scrapers, and other earthwork machinery usually have a relatively determined operation mode. Taking loaders as an example, completing an operation cycle includes three main operational stages: shoveling, transportation, and unloading. According to many practical experiences and test statistics, it usually takes 40–50 s to complete an operational cycle. Therefore, the BM VE ORB–MAPM VE and the QORB–MAPM VE method in this study can satisfy the requirements of shovel loading volume estimation in real-time operation. In summary, considering the volume estimation accuracy and total calculation time, the QORB–MAPM VE method in this study has wider application prospects in earthwork shovel loading volume estimation.