Smart monitoring of road pavement deformations from UAV images by using machine learning

The study was conducted in Mansoura City, Egypt. Figure 1 displays orthomosaic image of unnamed aerial vehicle (UAV). Table 1 displays the parameters of the unnamed aerial vehicle (UAV) image information.

This article presents an innovative technique for extracting surface cracks from unmanned aerial vehicle (UAV) images using machine learning techniques. Figure 2 depicts a flowchart. First, a UAV flight had been launched in order to effectively capture the intricate details of the study area. Subsequently, the aerial photographs acquired during the flight were subjected to image processing techniques using specialized software, resulting in the production of photogrammetric outcomes. The initial processing stage of image processing entails image alignment, while the subsequent step involves the generation of a dense point cloud, encompassing both point cloud and mesh. The final stages of this process include the production of a digital surface model (DSM) and an orthomosaic, which includes both the DSM and the digital terrain model (DTM). Image attributes are then created, and the UAV image and attribute are merged into one file. Cracks are detected and extracted from unmanned aerial vehicle (UAV) images using machine learning techniques. Then, an assessment of the accuracy of the results was made (Table 2).

The use of unmanned aerial vehicles (UAVs) for generating photogrammetric data has emerged as a widely preferred technology in contemporary times. Consequently, a diverse array of software applications had been constructed to facilitate flight planning, observation, and interaction between the controller and the unmanned aerial vehicle (UAV). Moreover, it should be noted that automatic flight software operates independently of human intervention, with the exception of situations pertaining to safety and security considerations. Operation plans are formulated by carefully choosing the specific region for which a map will be generated. Subsequently, the alignment of overlap and flight altitudes is established in conjunction with the requirement for high-resolution data. To enhance the precision of image alignment and obtain a more comprehensive point cloud, it is advisable to incorporate oblique images captured alongside the nadir (vertical) photo. The flying height is a crucial component in drone terrain mapping as it directly affects the reliability and precision of the generated DEM models and orthomosaic images. The flight height in this study was set at 100 m, a height deemed highly appropriate for the purpose of the study. As the appropriate height gives an appropriate pixel value, in our case a pixel of 2 by 2 cm was obtained.

The quality of pavement surface photographs is typically diminished due to several factors, including variations in lighting circumstances such as sunny or cloudy weather, the presence of random grainy textures, uneven lighting, irregular shadows, pavement markings, watermarks, tire marks, oil stains, and so on.

These parameters greatly influence the detection of cracks through image processing. The primary objective of picture preprocessing is to mitigate or minimize the adverse impacts of many elements, hence enhancing the overall efficacy of image processing. The study used Gaussian function-based spatial filtering and top-hat transform to preprocess the pavement photographs gathered.

The Pix4DMapper programmer, which is available at www.​pix4d.​com, processed the UAV images obtained during the research period. In the research, the obtained UAV photographs were processed with the Pix4DMapper programmer (available at www.​pix4d.​com) to produce an orthomosaic, also known as a single large 3D image. During the photogrammetric calculation, the captured imagery and the obtained frames are worked together to produce a digital orthographic image. The structure from motion (SFM) method begins with the identification of tie points in imagery via picture matching, which is often carried out automatically by point-obtaining and comparing algorithms. This is how the procedure got its name. After that, the block bundle self-calibration process is carried out, initially to determine both the exterior and interior orientation parameters from the pictures and subsequently to determine the three-dimensional surface coordinates of the tie points. After that, a dense matching method is used in order to produce a digital elevation model of the region. The digital orthomosaic was created by employing a method known as backward projection.

Crack extraction utilizing UAV photographs has the advantages of being high-resolution, maneuverable, efficient, and cheap to operate. However, due to the intricacy of background details, some crack extraction techniques based on UAV photographs will always include some incorrect pixels. The number of photographs increases after image cutting; however, this is a necessary step since it simplifies the background detail of a UAV picture before crack image extraction and re-splicing. It seems logical to selectively identify cracks in the sub-images that actually have cracks. Therefore, machine learning is used to classify the sub-images into two groups. The sequential stages involved in the methodology are depicted in Fig. 2. Initially, an unmanned aerial vehicle (UAV) image is divided into smaller images in ENVI (Environment for visualizing Images). Secondly, the features of the sub-image are extracted for individual pixels, which are used as input information for the decision or classification trees. A collection of 24 potential attributes was chosen. Due to the construction of texture formulae obtained from the grey-level co-occurrence matrix GLCM [14], a significant correlation exists among several of these equations [15]. Based on the findings of this research, it was determined that out of the total of 24 attributes, only 8 exhibited no correlation with each other. Consequently, these uncorrelated attributes were deemed suitable for utilization in the process of classification. Features are derived from the grey-level co-occurrence matrix GLCM. The attributes encompass those that were calculated from the grey-level co-occurrence matrix GLCM, as described by [16]. A 3 × 3-pixel window was employed for the construction of the grey-level co-occurrence matrix (GLCM) and subsequent texture calculation. According to [17], the use of large window sizes often leads to a decrease in the producer’s accuracy. This accuracy is measured by the ratio of pixels that were correctly classified in a specific class to the overall number of pixels that need to be classified in that class. The reason for this decrease is primarily attributed to the impact of between-class differences on the borders of pixels. These edge pixels, characterized by high texture values, tend to be misclassified and placed in the incorrect category. Moreover, in the case where a window possesses dimensions of M × M, it is worth noting that a strip with a width of (M−1)/2 pixels surrounding the picture is going to stay vacant. The conventional approach for addressing this matter involves the use of the nearest texture estimation to populate the edge pixels. According to [18], the issue of edge effects can be challenging in the field of classification.

The shaded cells indicate attributes that are not correlated and can be utilized for classification purposes. Therefore, DSM subset images and the attributes generated from them will improve the classification process in the second step when combined into one classifier. Figure 3a–h depicts the resulting attributes derived from a sample of subset images that contain cracks in the UAV image. According to [19], it is anticipated that incorporating texture attributes will help minimize misclassifications that may arise due to similarities between different features.

There are several factors that contribute to the degradation of pavement surface photographs, including variations in brightness (e.g., bright or cloudy), the presence of randomly grainy texture, uneven lighting, unsteady shadows, asphalt markings, watermarks, and other similar factors. The aforementioned factors exert a substantial influence on the identification and detection of cracks through the utilization of image processing techniques. Image post-classification smoothing involves mitigating or minimizing the adverse impacts of these factors, thereby enhancing the overall efficacy of image processing. The present study employs Gaussian filtering, median filtering, and standard deviation filtering as post-classification smoothing techniques for the acquired pavement images.where n and m are pixel indexes, σ is the Gaussian filter standard deviation, and \({G}_{(n,m)}\) is the mask element value. Mask and original image are convolved to create filtered image. The detector’s noise sensitivity depends on σ values (sigma = 4’).

If the input image q belongs to the integer class, all of the output values will be returned as integers. In the event that the quantity of pixels within the neighborhood, denoted as M * N, is an even number, it is possible for certain median values to not be whole numbers. In these instances, the fractional components are disregarded. The handling of logical input is analogous in nature.

Edge detection is a fundamental technique in the field of image processing that aims to identify specific points within a digital image that exhibit discontinuities, specifically abrupt changes in the brightness of the image. The regions in an image where there is a significant variation in brightness are commonly referred to as the edges or boundaries of the image. This study employs edge detection as a means of initially identifying the potential area of crack edges. Subsequently, morphological operations are applied exclusively to the identified crack edge area, resulting in a substantial enhancement in detection efficiency. In the field of image processing, conventional methods for detecting edges encompass various algorithms such as Sobel, Prewitt, Gauss–Laplace (LoG), and Canny operators. In the context of image processing, edge detection is the process of identifying areas within an image where there is a sudden and significant change in grey-level values. The objective of this task is to compute intensity gradients and thereafter determine the regions within the image that exhibit the most pronounced intensity gradients. After that, non-maximum suppression is employed in order to reduce the density of edges. Our objective is to eliminate extraneous pixels that may not be considered part of an edge. Pixels are considered edges if their intensity gradient value surpasses a predetermined threshold. Pixels are classified as edges if the value of the density gradient is over a certain threshold. In the event that a pixel resides between the range of two thresholds, it should be deemed acceptable solely if it is in close proximity to a pixel that surpasses the upper threshold. The equations are displayed below:

The intensity gradient indicates the picture’s edge direction, which the Canny detector follows horizontally, vertically, and diagonally. G is the gradient quantity, is the gradient in the first direction, is the gradient in the second direction, a tan is the arctangent operator, and is the gradient direction. Equations 3 and 4 yield the gradient magnitude and direction; I is the image after applying smoothing filters; and BW is the Canny output.

Finally, the process concludes the identification of edges by effectively restraining all weak edges that are not interconnected with powerful edges. The operator known as Canny is known for its low error rate as it is designed to detect all edges accurately while minimizing the occurrence of false positive responses. Additionally, it has the capability to accurately identify the boundary in close proximity to the actual boundary. Hence, it can be argued that this method is among the most rigorously defined approaches that offer robust and dependable detection capabilities.

Mathematical morphology holds significant importance as a fundamental technique in the realm of a few levels of image processing [25]. The bottom and top hat transforms are generated in the morphology of the grayscale through the application of the process of subtracting one image from another, as well as closing and opening operations. The bottom and top-hat transforms serve similar purposes, with the key distinction lying in the object being considered. The top-hat transform is commonly employed to detect and enhance light-colored objects against a dusky background, while the bottom-hat transform is typically used to identify and enhance dark-colored objects against a light background. The concept of the top-hat operator is interpreted as the negation of its opening operation.

The top-hat operator of function I is mathematically interpreted as the difference between I and its opening operation.

The bottom-hat operator of function I is expressed as the result of subtracting I from the closing operation.

Let I represent the initial picture and b denote a structural component. The determination of size b primarily relies on the conversion ratio between pixels and real-world dimensions, as well as the typical dimensions of cracks. The standard dimension of b is 150 mm.

The operator in question uses a robust combination of erosion and dilation techniques. Upon the act of opening the box, the items contained within undergo a process of separation. The process of dilation results in an enlargement of an image, while erosion leads to a reduction in its size. The act of opening an image result in the refinement of its contour, the separation of narrow connections, and the removal of slender extensions in a comprehensive manner. The acronym AB denotes the sequential application of erosion and dilation operations in image processing. Specifically, AB signifies the process of expanding the boundaries of an image ‘A’ using a structural element B.

The combination of erosion and dilation results in the generation of a robust operator. Objects are brought into proximity when they are in a state of closure. Similar to the process of opening, closing operations in image processing have the effect of reducing irregularities in contour regions. Additionally, they serve to merge small discontinuities and elongated gaps, eliminate small voids, and fill in missing areas. The operation denoted as A·B represents a sequential process that involves dilatation and erosion. This process is observed when an image A undergoes closure through the application of a structural element B.

The process of achieving closure involves the gradual erosion of a dilated image. The act of closing serves to unite fragmented components and bridge any existing gaps within objects. After the closure of one or more dilations, a subsequent erosion operation takes place.

The underlying concept of a classification tree involves the iterative division of high-dimensional data into progressively smaller partitions. This process aims to enhance the purity of the partitions in relation to their class membership. The initial step involves the cultivation of a tree of excessive size, employing a norm. In order to produce the most efficient partitions for the tree. Typically, those expansive trees exhibit a high degree of compatibility with the training data set. However, their ability to generalize is limited, resulting in a low rate of accurate classification for novel patterns. The visual representation in Fig. 5 depicts a tree structure that appears visually complex, wherein a set of rules, such as “X6 < 0.0965483,” are employed to assign each pattern to one of the 15 terminal nodes for classification purposes. In order to ascertain the assignment of input patterns for a given surveillance, the classification tree commences at the highest node and proceeds to apply the corresponding rule. In the event that the dot adheres to the specified rule, the tree will proceed along the left path.

The proposal of Breiman [20] states that the suggested method entails iteratively removing branches from the excessively large tree in order to identify a hierarchical series of smaller subtrees. The optimal tree within this sequence is determined by evaluating the misclassification rate, which is estimated through either an independent test sample or cross-validation. This implies that a reduced subgroup of the aforementioned tree might yield minimal fault, as certain decision rules within the complete tree may have a detrimental rather than beneficial impact. The results of the tree test, as depicted in Fig. 6, were obtained through a ten-fold cross-validation approach. This involved partitioning the data into ten subsets, on the basis of only 10% of the information reserved for verification purposes Furthermore, the remaining 90% of the data is allocated for the purpose of training. Initially, the mistake made with the resubstitution, which represents the percentage of the initial observations that were incorrectly categorized by the different subsets of the first tree, was determined. Subsequently, cross-validation was employed to assess the accuracy of the estimation of fault for trees of different sizes. According to Fig. 6, it can be observed that the substitution error tends to exhibit an overly optimistic tendency. The relationship between tree size and error rate is inversely proportional, as evidenced by the decrease in error rate as the tree size increases. However, the results of cross-validation indicate that beyond a confirmed threshold, increasing the size of the tree actually leads to an increase in error rate. This conclusion is based on the selection of the tree exhibiting the lowest cross-validation error. Although the current approach may be deemed acceptable, it is advisable to opt for a less intricate tree structure if it yields comparable results. The employed criterion involves selecting the most straightforward tree that falls within a range of one standard error from the minimum.

It was derived by finding a threshold equal to the minimum cost plus one standard error and using that. Under this threshold, the optimal level is the one with the smallest tree. The cutoff value was calculated by adding one standard error to the minimum cost since best-level = 0 represents the unpruned tree. Next, the best level of the tree was used to prune it, and the anticipated misclassification cost was calculated. The edited tree of classification is shown in Fig. 7. The cost of misclassification is 1% for this set of cracked data.

The fundamental step of every edge detection algorithm involves the elimination of image blurriness or noise through the application of filtering algorithms. There exist multiple filtering algorithms, among which Gaussian filtering is widely regarded as the most optimal choice for various types of images. Following the application of the filtering algorithm, the subsequent step involves performing edge detection on the image utilizing the Canny edge detection technique. In this study, the Canny edge detection algorithm was employed to identify a diverse array of edges within images. The detection mechanism identifies edges by detecting the local maxima of the gradient of the function f (x, y). The calculation of the gradient involves the utilization of the derivative of a Gaussian filter, a median filter, and standard deviation filtering; the results are depicted in Fig. 8.

If the value of a pixel exceeds the high threshold, it is classified as an edge pixel. Conversely, if a pixel’s value is below the low threshold, it is not classified as an edge pixel. Through the analysis of various algorithms applied to diverse input data, it has been determined that the Canny edge detection algorithm consistently yields superior outcomes. The outcomes of applying the Canny edge detection algorithm to a sample of images, as shown in Fig. 9, support this assertion.

In the field of morphological image processing, the process of reconstructing the original image involves the use of dilation, erosion, opening, and closing operations for a finite number of iterations. In this study, a closed operation was used to detect cracks, and a filled image was used to detect patches and potholes, as shown in Fig. 10. The closed operation is responsible for merging narrow discontinuities, removing minor openings, and filling gaps in the contour. The process involves the systematic assignment of pixel values to the boundary region of an image. The operator in question is a potent computational tool that is derived from the fusion of the erosion and dilation operations.

The main focus of this work was the detection of cracks. To do this, the categorized image was converted into a closed image in order to segregate the cracks. The numerical values corresponding to cracks were transformed to a binary representation of one, and the numerical values corresponding to non-crack pixels were transformed to a binary representation of zero. Ultimately, the photographs were selectively cropped in order to exclusively display the presence of cracks. The outcome yielded a monochromatic depiction that effectively portrays the identified fractures and road depressions, devoid of extraneous visual elements or any absence of data. The study presents Fig. 11 as a representative example.

The evaluation of accuracy is regarded as a crucial concluding phase in the classification process. An accuracy assessment was conducted on the images derived from the classification tree, Canny edge detector, filled image, and morphologically close image in order to ascertain the adequacy of the image classification. The classified images from machine learning were compared with photographs obtained from unmanned aerial vehicles (UAVs). A total of one hundred randomly chosen points were utilized to classify the photographs within the designated study area. Subsequently, the accuracy rating’s reference column was completed based on the reference image data. The Cohen’s kappa coefficient was further examined in relation to the matrix’s shortcomings. The Kappa statistic demonstrates the effectiveness of the classification process in comparison with random values. The calculation of accuracy assessment is elucidated in Tables 3, 4, 5, 6, 7.

A comparison of the different stages that were performed on the image to extract cracks is shown in Fig. 12.

When compared to the reference data, the utilization of a classification tree resulted in the detection of cracks with well-defined edges with an overall accuracy rate of 86%, as shown in Table 3.

When applying pruning classification, the overall accuracy will rise to 90%, and the kappa coefficient will become 0.8, as shown in Table 4.

By analyzing different algorithms applied to various input data, it has been determined that the Canny edge detection algorithm consistently yields superior results. This assertion is supported by the results obtained from the accuracy assessment, as shown in Table 5, where the overall accuracy is 93% and the kappa coefficient is 0.8597.

In the end, the cracks and potholes were determined using the morphological process (closed and filled image), and it proved its efficiency in evaluating the accuracy, as the total accuracy was 96% for cracks and 94% for potholes, and the Kappa coefficient was 0.9195 for cracks and 0.8790 for potholes, as shown in Table 6 and 7.