nach oben

Complex & Intelligent Systems

Erschienen in:

Open Access 27.06.2022 | Original Article

Improved PSPNet-based water shoreline detection in complex inland river scenarios

verfasst von: Yuheng Yin, Yangang Guo, Liwei Deng, Borong Chai

Erschienen in: Complex & Intelligent Systems | Ausgabe 1/2023

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Patentsuche

Aus

Abstract

The water shoreline is essential for unmanned surface vessels (USVs) to navigate autonomously. Many existing traditional water shoreline detections approaches not only fail to overcome the effects of water reflections, image inversions, and other factors but are also unsuitable for water shoreline detection in a variety of weather conditions and in complex inland river scenarios. Therefore, we propose a water shoreline detection approach based on an enhanced Pyramid Scene Parsing Network (PSPNet). We introduce a migration learning approach to the PSPNet feature backbone extraction network Resnet50 to improve training efficiency and add a Convolutional Block Attention Module (CBAM) attention mechanism module to improve the robustness of training. In addition, the pyramid pooling module adds the branch of the atrous convolution module. Finally, the waterfront segmentation map is processed by the Canny edge detection method, which detects the water shorelines. For the network's training and validation, we use the USVInland dataset, the world's first urban inland driverless dataset. The experimental results show that the segmentation accuracy MIou of this paper is 96.87% and Accuracy is 98.41, which are higher than some mainstream algorithms. It is capable of detecting water shorelines accurately in a variety of interior river situations.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Introduction

With the development of autonomous driving technology, the autonomous driving scenario is gradually expanding from the road to the water, and Unmanned Surface Vehicles (USVs) and their related applications have become one of the research hotspots in recent years [1, 2]. Increasingly prominent due to their ability to perform hazardous and time-consuming tasks [3], the strong demand from commercial, scientific, and environmental communities has accelerated the development of USVs applications such as hydrographic surveying and mapping, marine resource exploration, water quality monitoring, and floating waste removal [4‐7]. Compared to marine USVs, inland waterway USVs are more closely related to human life and have great potential value, which makes them central to the construction of autonomous transportation systems for inland waterways [8].

For USVs in inland river scenarios to accomplish the established tasks without human intervention, it is necessary to deal with the perception of the environment by unmanned vessels in the near-shore environment, especially the perception and understanding of waterline information [9, 10]. The environment perception capability of the USVs in the inland river scenario is a key technology for the unmanned vessel to complete various tasks [11, 12], for example, the river patrol of the USVs in the inland river scenario, the automatic berthing, and docking of the USVs, the automatic mapping of the USVs, etc. First, the location and orientation of the water shoreline need to be determined to complete the set tasks under the premise of ensuring the safety of the unmanned boat's navigation. In addition, for some areas without obvious structure and lacking obvious water shoreline, the detection of the river surface can provide information about the drivable area, thus can improve the active safety function of the unmanned boat navigation system. Water shoreline detection helps to analyze and judge the specific navigation information of inland rivers, thus assisting and ensuring the safe navigation behavior of unmanned boats [13‐15].

The current research on water shoreline detection methods is mainly divided into the following categories: the edge detection-based methods, such as the water shoreline detection method based on the Canny edge detection operator and the Hough transform proposed by Zeng et al. [16], and the fractional dimensional method of fusing multiple features to identify the shoreline proposed by Shao et al. [17]. They argue that the waterfront demarcation line traverses the entire image and that the waterfront distant boundary approximates a horizontal straight line when the observation point is close to the water, and propose an edge extraction algorithm that combines fractional dimensional features and shape location for edge extraction of waterfront images, followed by fitting the waterfront demarcation line by the median method; the methods based on Spatial characteristics. The water shoreline detection method based on logistic regression and sample polynomial fitting proposed by Kröhnert [18] has obtained a better fitting effect on the water shoreline of nearly straight lines and has better real-time performance, but it is not suitable for irregular water shoreline detection. Peng et al. [19] analyzed the features of HSV color space under different illumination and used the features with higher land saturation for image segmentation to extract riverbank contour lines, but the method has low robustness and the segmentation effect is not satisfactory in cases such as the similar color of waterbanks or the influence of illumination. Sun et al. [20] developed a superpixel-based conditional random field model to segment the land and sea regions; the methods based on area growth, such as the water shoreline detection based on the area growth algorithm proposed by Zheng et al. [21], which achieves automatic growth point selection by constructing a least squares problem and obtains a better detection effect, but the time consumption of the area growth algorithm is large, and the real-time performance of water shoreline detection using the area growth algorithm is low; the methods based on deep learning, such as Steccanella et al. [22] proposed Convolutional Neural Network (CNN) for water surface area segmentation, such algorithms also have the problem of poor real-time performance. Shen et al. [23] proposed an improved algorithm based on DeepLab v3 + for semantic segmentation of shoreline images to extract the water surface region, and when combined with the traditional edge detection algorithm to detect the water shoreline. Erdem et al. [24] propose a majority voting method based on different deep learning structures to automatically acquire water shorelines Methods of deep learning are highly adaptable to a different times and shoreline scenes and can overcome the interference of uncontrollable factors such as reflection and ripple in shoreline scenes on image segmentation, and achieve accurate detection of water shoreline with fast extraction speed [25, 26].

Traditional detection methods for inland river scenes with varying weather and a complex water shoreline need adjusting parameters in different situations, which is not applicable, and the semantic segmentation method is used to train images of waterbanks in inland river scenes with different inland river scenes, enabling the model to learn waterbank segmentation in different inland river scenes and thus solve applicable problems. Due to the existing mainstream algorithms, there is a lack of information acquisition for irregular water shoreline details, resulting in the segmentation accuracy needs to be improved, we improve PSPNet to improve the network's robustness and segmentation accuracy to improve the accuracy of waterfront segmentation in inland river scenes. The following are our major contributions:

(1)

Applying migration learning to PSPNet networks using ResNet50 weights on the ImageNet dataset to improve training efficiency.

(2)

The CBAM attention mechanism is added to the backbone feature extraction network ResNet50 to improve the robustness of the network.

(3)

We have added a branch of the atrous convolution module to the pyramid pooling module to combine pooling and null convolution to improve the acquisition of global information from the network.

The rest of the paper is organized in this way. The second section introduces the PSPNet structure, the third section explains the methodology of the paper, and the fourth section conducts experiments and shows the results. The fifth section concludes the paper.

Principle and method

PSPNet network model

The algorithm in this paper is based on the Pyramid Scene Parsing Network (PSPNet) [27], which is a Fully Convolutional Network (FNC) [28]-based network, but differs significantly from the traditional FCN in that PSPNet considers contextual and local information to make predictions based on the traditional FCN. To enhance the extraction of contextual information and local features, PSPNet proposes a pyramid pooling module, which uses pooling operations at different scales to obtain image contextual and local information and realize the extraction of multi-scale features, which solves the problem of scale diversity in image segmentation to a certain extent and improves the image segmentation effect compared with traditional FCN. The PSPNet network structure is composed of two main parts, the first module is the CNN [29] backbone network for obtaining feature maps, and the second module is the pyramid pooling module. These two modules are described separately below.

PSPNet was proposed in 2017 by literature, the entire network consists of four parts: input image, feature map, pyramid pooling module and final prediction. lt for the input image in part 1, we use a pre-trained ResNet model with an extended network strategy to extract the feature maps, as shown in part 2. For the above feature maps, we use the pyramid pooling module to obtain contextual information, where the pyramid pooling module is divided into four levels with pooling kernel sizes of 1 × 1, 2 × 2, 3 × 3 and 6 × 6, which can eventually be fused into global features. Then, in the final part of module, we connect the fused global features to the original feature map. Finally, the final prediction map is generated in part 4 by a layer of convolution. The specific implementation of feature map connects is shown in Eq. (1):

$$ R = \sum\limits_{i = 1}^{C} {(M_{i} + N_{i} ) \otimes K_{i} } , $$

(1)

where $C$ denotes the total number of channels, $M_{i}$ and denotes the feature maps of feature map and feature map at the ith channel, respectively, $K_{i}$ is the convolution kernel at the ith channel in the convolution kernel and denotes the convolution operation. R is the output result of a single convolution after feature map connect (Fig. 1).

Feature extraction network

ResNet [30] is short for residual network and this network structure is usually used as a backbone network The core idea of ResNet is to add a shortcut branch to the ordinary convolutional neural network, short-circuiting the original convolutional layers and allowing the training process to skip a part of the convolutional layers, such a design can reduce a certain number of parameters, and the comparison between the residual network and the general network structure is shown in Fig. 2. This design can reduce the number of parameters.

The problem of gradient disappearance can be effectively solved by the residual structure, as can be demonstrated by Eq. (2).

$$ {\text{Loss}} = F(X,W) $$

(2)

$$ \frac{{\partial {\text{Loss}}}}{\partial X} = \frac{\partial F(X,W)}{{\partial X}} $$

(3)

$$ \begin{aligned} & {\text{Loss}}_{n + 1} = F_{n} (X_{n} ,W_{n} ),\\ & {\text{Loss}}_{n} = F_{n - 1} (X_{n - 1} ,W_{n - 1} ), \cdot \cdot \cdot {\text{Loss}}_{2} = F_{1} (X_{1} ,W_{1} )\end{aligned} $$

(4)

$$ \frac{{\partial {\text{Loss}}}}{{\partial X_{i} }} = \frac{{\partial F_{n} (X_{n} ,W_{n} )}}{{\partial X_{n} }} \times \cdots \times \frac{{\partial F_{n} (X_{i + 1} ,W_{i + 1} )}}{{\partial X_{i} }} $$

(5)

$$ \frac{{\partial X_{i + 1} }}{{\partial X_{i} }} = \frac{{\partial X_{i} + \partial F(X_{i} ,W_{i} )}}{{\partial X_{i} }} = 1 + \frac{{\partial F(X_{i + 1} ,W_{i + 1} )}}{{\partial X_{i} }}. $$

(6)

The equation is the loss function of the network, and the Eq. (3) is its back-propagation gradient value. The same principle is extended to the multilayer neural network with the loss function of the network as Eq. (4), where, n is the number of layers of the neural network, and finally, the gradient of the ith layer can be introduced according to the chain rule as the Eq. (5). Therefore, it can be seen that the gradient of the previous layer of the network becomes smaller and smaller as the error is passed back. ResNet changes the output layer H(x) = F(x) to H(x) = F(x) + x, which is changed from Eqs. (5) to (6), so the gradient will not disappear even if the network structure is very deep.

The residual module shown in Fig. 2b is called Bottleneck, and is explained in terms of the 50 layers used in this paper. The main part of the network architecture of ResNet50 is that it goes through 4 large convolutional groups, which contain 4 Bottleneck modules, 3, 4, 6, and 3, respectively. The convolutional groups are pooled globally, then fully connected in 1000 dimensions, and finally, the classification scores are output by Softmax. The specific structure is shown in Table 1.

Table 1

Resnet50 network structure

Layer name	Output size	50-layer
conv1	112 × 112	7 × 7, 64, stride 2
conv_2	56 × 56	3 × 3, max pool, stride 2 $\left[ \begin{gathered} 1 \times 1,64 \hfill \\ 3 \times 3,64 \hfill \\ 1 \times 1,256 \hfill \\ \end{gathered} \right] \times 3$
conv_3	28 × 28	$\left[ \begin{gathered} 1 \times 1,128 \hfill \\ 3 \times 3,128 \hfill \\ 1 \times 1,512 \hfill \\ \end{gathered} \right] \times 4$
conv_4	14 × 14	$\left[ \begin{gathered} 1 \times 1,256 \hfill \\ 3 \times 3,256 \hfill \\ 1 \times 1,1024 \hfill \\ \end{gathered} \right] \times 6$
conv_5	7 × 7	$\left[ \begin{gathered} 1 \times 1,512 \hfill \\ 3 \times 3,512 \hfill \\ 1 \times 1,2048 \hfill \\ \end{gathered} \right] \times 3$
	1 × 1	Average pool 1000-dfe, softmax
FLOPs 7.6 × 10⁹

Pyramid pooling module

In the deep neural network, the size of the receptive field can roughly represent the extent to which we use contextual information. Although the receptive field of ResNet is larger than the input image in theory, it can be found from experience that the receptive field of CNN is far smaller than the theoretical size, especially in deep networks, which makes many networks not fully integrate important global scene information. The pyramid pooling module can extract features through pooling cores of different sizes to obtain more global scene information.

The pyramid pooling module is composed of four levels of pooling, which are 1 × 1, 2 × 2, 3 × 3, and 6 × 6. 1 × 1 pool extracts the shallowest features, and processes the feature map completely. 2 × 2, 3 × 3 and 6 × 6 are the sub-region of the other three levels, which uses the pooling of different scales to process the feature map. The features of different scales are obtained by grading and pooling. Due to the different pool cores in the pool process, the output feature scales are also different. Finally, the features extracted from different pool cores can be fused into global features. In the last part of the pyramid pooling module, we connect the fused global features with the original feature map.

The improved pyramid scene analysis network

Implementation of CBAM attention mechanism

Convolutional Block Attention Module (CBAM) represents the attention mechanism module of the convolutional module, which is an attention mechanism module combining spatial and channel. The overall structure after adding the CBAM module is shown in Fig. 3. What can be seen is that the output of the convolutional layer will first pass through a channel attention module to get the weighted result and then will pass through a spatial attention module to finally weight the result. Compared with focusing only on the channel, the attention mechanism can achieve better results.

The channel attention module takes the input feature map, goes through global max pooling and global average pooling based on width and height, respectively, and then goes through MLP, respectively. The channel attention feature map and the input feature map are multiplied elementwise to generate the input features required by the Spatial attention module. The specific steps are shown in Fig. 4.

The Channel Attention Module compresses the feature map in the spatial dimension into a one-dimensional vector, which it then manipulates. Not only does the compression in the spatial dimension take into account Average Pooling, but it also takes into account Max Pooling. To create a channel attention map, use Average Pooling and Max Pooling to aggregate the spatial information of feature maps, transfer them to a shared network, compress the spatial dimensions of the input feature maps, and sum and merge them element by element. For a single graph, channel attention is concerned with which pieces on the graph are important. When doing gradient back-propagation calculations, mean pooling provides feedback for every pixel point on the feature map, whereas maximum pooling provides feedback for gradients only when the response is highest in the feature map. The following is an equation to describe the channel attention mechanism.

$$ \begin{aligned} M_{{\text{c}}} (F) & = \sigma (MLP({\text{AvgPool(}}F)) + MLP({\text{MaxPool}}(F))) \\ & = \sigma (W_{{1}} (W_{0} (F_{{{\text{avg}}}}^{{\text{c}}} )) + W_{1} (W_{0} (F_{{{\text{max}}}}^{{\text{c}}} ))), \\ \end{aligned} $$

(7)

where MLP denotes multilayer perceptron, $\sigma$ denotes the sigmoid function, $W_{0}$ and $W_{1}$ denotes the weight matrix in the multilayer perceptron, and $F_{{{\text{avg}}}}^{{\text{C}}}$ and $F_{{{\text{max}}}}^{{\text{C}}}$ denotes the average set feature and maximum set feature in the channel attention module.

The feature map output from the Channel attention module is used as the input feature map of the spatial attention module, as shown in Fig. 5. We execute global max pooling and global average pooling based on the channel first, and then we concat these two outcomes based on the channel. The sigmoid generates the spatial attention feature, which is then multiplied by the module's input feature to produce the end feature.

In the same way, the Spatial Attention Module compresses the channels and pools the mean value and the maximum value in the channel dimension, respectively. The AvgPool operation is to extract the average value on the channels, and the number of times it is extracted is also the height multiplied by the width; then, the previously extracted feature maps (the number of channels is 1) are combined to obtain a two-channel feature map. The following is an equation to describe the spatial attention mechanism.

$$ \begin{aligned} M_{{\text{s}}} (F)& = \sigma (f^{7 \times 7} ([{\text{AvgPool}}(F);{\text{MaxPool}}(F)])) \\ & = \sigma (f^{7 \times 7} ([F_{{{\text{avg}}}}^{{\text{s}}} ;F_{{{\text{max}}}}^{{\text{s}}} ])), \\ \end{aligned} $$

(8)

where $f^{7 \times 7}$ denotes the convolution operation with a filter size of 7 × 7, and $F_{{{\text{avg}}}}^{{\text{S}}}$ and $F_{{{\text{max}}}}^{{\text{S}}}$ denote the average pool feature and the maximum pool feature in the spatial attention module, respectively.

Combining the CBAM attention mechanism with the ResNet50 network for end-to-end training, using the mathematical principle of attention mechanism to calculate the region of interest can be understood as a larger value of the weight of the region of interest, then correspondingly after the attention mechanism, the weight accounts for a larger proportion of the region of interest, highlighting the region of interest. By adding the CBAM attention mechanism to the first and last layer of the Resnet50 network, the network can pay more attention to the useful information and discard the useless information. As Fig. 6 shows the training loss function curve before and after the addition of the CBAM attention mechanism, it can be seen that the CBAM attention mechanism effectively reduces the oscillation when the curve converges.

Improved pyramid pooling module with atrous convolution module

In the inland river scene, the surrounding environment of the riverbank is complex and changeable, and the water shoreline is extremely irregular. To segment the water area more accurately, there are high requirements for the extraction of the global information of the scene. Based on the original pyramid pooling module, we add an atrous convolution module branch to extract the global information more comprehensively by expanding the receptive field.

The atrous convolution is obtained by filling 0 with ordinary convolution so that the convolution can obtain a larger receptive field and more image feature information without changing the parameter size. More contextual information can be obtained by increasing the receptive field of the separation layer or fusing the deep and shallow features, which is easy to save the detailed information of the target as shown in Fig. 7a, but the three-layer convolution is an expansion convolution with r = 2. It can be seen that only 75% of the receptive field of the orange pixel on the top layer participates in the actual calculation. Using the hybrid dilated convolution, a certain number of layers are formed into a group, and then each group uses the continuously increasing hole rate, as shown in Fig. 7b. The combination of three-atrous rates of r = 1, r = 2, and r = 3 is adopted, which not only expands the receptive field, which is the same as the three r = 2 but also obtains information from a wider range of pixels. By designing a sawtooth wave with a void rate sequence $[r_{1} ,...,r_{i} ,...,r_{n} ]$, the aim of sensory field coverage of the whole map is achieved, with a small rate extracting local information and a large rate extracting long distance information, which can obtain wider area information and improve information utilization while keeping the size of the receiving field constant. And Eq. (9) for calculating the rate is defined as follows:

$$ M_{i} = \max \left[ {M_{i + 1} - 2r_{i} ,M_{i + 1} - 2(M_{i + 1} - r_{i} ),r_{i} } \right]. $$

(9)

With $M_{n} = r_{n}$, the design objective is to have $M_{2} \le K$. We adopt the idea of Fig. 7b and for our dilated convolution kernel size K = 3, we design the dilation rate as a [1, 2, 5, 1, 2, 5] sawtooth structure. By fusing the convolutional feature mapping of each dilation rate, the local and global information in the image is obtained by fusion, which facilitates the preservation of image details and enriches the feature information of the image while avoiding grid effects.

By adding a branch of the atrous convolution module to the pyramid pooling module of PSPNet, the global features extracted from the shallow features of the feature backbone network are extracted by one more hollow convolution module, and the features of its expanded field of perception are used to have better feature understanding for complex image information, and finally, the output of the atrous convolution module, the shallow feature map and the output of the pyramid pooling module are spatially connected and merged to make the current features include both global and local features, as shown in Fig. 8. Finally, the final prediction results are obtained by convolution and upsampling operations. The network's ability to gather global information about the image and make better judgments about complicated boundaries is improved when the pyramid pooling module and the atrous convolution module are combined.

The water shoreline detection

In the complex environment of the inland river, the water surface can be effectively segmented out using the improved semantic segmentation model. Based on segmenting out the water surface area, the edges of the water surface area are extracted by combining the classical edge detection Canny algorithm [31], and the extracted edge lines are superimposed onto the original image after expansion [32] processing to facilitate observation. The results are shown in Fig. 9. The experimental results show that a simple Canny edge detection operator can quickly and effectively detect a clear and complete water shoreline in the segmented image.

Experimental results and analysis

Introduction to the dataset

Our training uses the USVInland dataset [33], which is a multi-sensor dataset for USVs in inland waterways. The dataset is available online at https://www.orca-tech.cn/datasets. USVInland is collected across a 26 km trajectory in a variety of realistic scenarios in inland waterways using various modalities including lidar, stereo cameras, millimeter-wave radar, GPS and inertial measurement units (IMUs) and contains data collected in real inland river scenarios under different weather or lighting conditions to cover real-world driving scenarios. The dataset has a total of 700 images of waterways in various river landscapes of different widths and lengths. Furthermore, unlike most datasets used for seawater segmentation, where waterlines are mainly straight water antennas, the USVInland dataset covers different inland river landscapes with different waterline structures. As shown in Fig. 10, the water bank images of inland river scenes under different weather conditions in the data set are shown.

Evaluation based on improved semantic segmentation algorithm

Semantic segmentation algorithm training

In this dataset of 700 images containing multiple condition acquisitions, we set the images to a uniform size of 320 × 640 and trained them using the labels provided by the dataset. We selected 500 images of similar numbers for each scenes. Using 400 of these were used for training the improved semantic segmentation model, and the remaining 100 images were used as a validation set to test the accuracy of the algorithm.

The training networks in this paper were all implemented using the PyTorch deep learning framework in Python. The test device used was an RTX2080ti with 11 GB of video memory, Python 3.6, and PyTorch 1.4. A total of 200 epochs are trained for the network. The improved PSPNet model has the parameter FLOPS of 68.03G in time complexity and the parameter Param of 64.80 M in space complexity. It takes about 1.5 h to complete the whole training.

We use the idea of transfer learning [34] in training and fine-tune the weight of Resnet50 on the Imagenet dataset to greatly improve the efficiency of training. Therefore, we freeze the network in the first 25 epochs can speed up the training efficiency and prevent the weight from being damaged. In the freezing stage, the backbone of the model is frozen, and the feature extraction network does not change. The visual memory used is tiny, and the network is merely fine-tuned. The network is then unfrozen after 175 epochs. The backbone of the model is not frozen in the unfreezing step, and the feature extraction network will change. The video memory is fully utilized, and all network parameters will change.

For the evaluation of semantic segmentation. In classification and diagnostic tests, ROC and the area under the ROC curve (AUC) describe how an adjustable threshold causes changes in two types of errors: false positives and false negatives. However, the ROC curve and AUC are only partially meaningful when used with unbalanced data [35]. In this paper, the confusion matrix approach is used to evaluate the model, and the overall PA and MIoU are selected as the evaluation metrics for pixel-level semantic segmentation. PA calculates the ratio of correct pixels to the total number of pixels in the prediction, and the IoU calculations for each category of MIoU are summed and averaged to obtain a global evaluation.

$$ {\text{PA}} = \frac{{{\text{TP}} + {\text{TN}}}}{{{\text{TP}} + {\text{FN}} + {\text{FP}} + {\text{TN}}}} $$

(10)

$$ {\text{IoU}} = \frac{{{\text{TP}}}}{{{\text{FP}} + {\text{FN}} + {\text{TP}}}} $$

(11)

$$ {\text{MIoU}} = \frac{1}{k + 1}\sum\limits_{i = 0}^{k} {\frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}} + {\text{FP}}}}} , $$

(12)

where TP is the number of positive classes predicted as positive classes; FN is the number of positive classes predicted as negative classes; FP is the number of negative classes predicted as positive classes; TN is the number of negative classes predicted as negative classes.

Analysis of experimental results

Even though based on semantic segmentation models, watershed area detection is not only unlimited to areas with straight boundaries but is also very applicable. However, due to the high variability of different inland water environments, it is difficult to build robust segmentation models for water environments. Furthermore, unlike the lane detection task for autonomous vehicles, the similarity in appearance and reflection of objects on river banks makes accurate segmentation of waters difficult. The use of some mainstream semantic segmentation algorithms can suffer from poor segmentation of reflection regions, recognition of walls as water regions, and inaccurate segmentation of irregular water boundaries.

The improved PSPNet extracts shallow features through ResNet50 with the CBAM attention mechanism, then extracts global information through two branches of multi-scale pooling and multi-scale convolution, and then connects the shallow feature maps with the feature maps obtained from the two branches, so that the current features include both global and local features. Finally, the final prediction results are obtained by convolution and upsampling operations. The improved pyramidal scene analysis network can effectively solve the problem of not segmenting the reflection area and segmenting the walls.

The new model adds a branch of the atrous convolution module, which not only increases the segmentation accuracy of the original model but also enhances the solution to the segmentation problem of irregularly shaped water boundaries, which can effectively improve the accuracy of water bank segmentation in complex inland river scenes. A comparison of the specific shoreline segmentation results is shown in Fig. 11. We can see that when the water shoreline is relatively regular, the current semantic segmentation network can also segment the waterfront more accurately. However, it does not perform well in irregular water shorelines. The improved semantic segmentation model not only maintains its accuracy in regular water shoreline detection, but also extracts detailed information well in irregular water shoreline scenarios, which is closer to Ground Truth, effectively increasing the segmentation effect.

Meanwhile, the segmentation index of our improved semantic segmentation network is 96.87% for MIoU and 98.49 for PA, which is higher than some mainstream algorithms. It shows that our algorithm has a good segmentation effect in complex inland river scenes. Some improvements to the algorithm can increase the information acquisition ability of the network for water bank images and obtain high segmentation accuracy at the same time. The results are shown in Table 2.

Table 2

Comparison of different model indicators

Model	Background_IoU	Water_IoU	MIoU	PA
DeepLabv3 +	96.09%	93.47%	94.78%	97.01
UNet	96.32%	94.21%	95.27%	97.84
PSPNet	96.87%	95.02%	95.94%	98.01
Ours	97.59%	96.14%	96.87%	98.49

Results and analysis of water shoreline detection under different weather conditions

In the complex inland scene environment, when the coastline is extremely irregular or there is interference on the water surface, the use of traditional methods may fail, while the traditional coastline detection method uses some features of water surface and ground for image segmentation. In the face of different scenes, it needs to adjust the reference, which does not apply to complex inland river scenes and changeable weather environments. We use the improved semantic segmentation model, which has good applicability and improves the accuracy of water bank segmentation. Based on the water bank segmentation results, we use the Canny edge detection algorithm to extract the water bank line.

We extracted the waterfront segmentation map using the Canny edge detection algorithm to extract the waterfront contours. To further demonstrate the applicability of the algorithm in this paper, as shown in Fig. 12, we compared the water shorelines extracted by different networks with the reference standard water shorelines separately to be able to see the error with the standard reference water shorelines more intuitively. Where the green line indicates the reference standard water shoreline and the red line indicates the water shoreline extracted from the experimental results, with the overlap in yellow. We use the root mean square error (RMSE) to quantitatively evaluate the extraction error of the water shoreline. the RMSE is calculated as shown in equation (13):

$$ {\text{RMSE}} = \sqrt {\frac{{\sum\nolimits_{i = 1}^{n} {(A_{i} - B_{i} )^{2} } }}{n}} , $$

(13)

where A is the pixel sampling matrix of the actual extracted water shoreline obtained by uniform sampling n times, and B is the water shoreline extracted as a result of the experiment.

We can see that the algorithm in this paper extracts the water shoreline with the least error from the reference water shoreline, which is closer to the real water shoreline location. In Table 3, we can see that our algorithm has the lowest RMSE value, but as the water shoreline becomes more complex, the RMSE value increases.

Table 3

RMSE values for different methods

Method	Image1	Image2	Image3	Image4
Deeplabv3 +	33.86	39.53	33.37	38.39
UNet	31.74	36.44	31.18	38.15
PSPNet	29.40	31.42	35.27	37.80
Ous	26.38	30.82	31.71	36.48

As shown in Fig. 13a–f, we show the results of water shoreline detection using the algorithm in this paper under different weather conditions and different inland river scenes. We can see that our algorithm can segment the shoreline in a variety of weather conditions and in inland river scenes of varying complexity, and that when combined with the Canny edge detection algorithm and then superimposed on the original image by the expansion algorithm, we can intuitively see the position of the water shoreline.

Conclusion

In this paper, we propose a water shoreline detection algorithm based on the improved PSPNet algorithm. To improve the training efficiency, robustness, and accuracy of the algorithm, we first introduce the idea of migration learning based on the PSPNet algorithm, then we add the CBAM attention mechanism to the backbone feature extraction network ResNet50 to improve the algorithm's robustness, and finally, we add the atrous convolution module branch to the pyramid pooling module to combine pooling and atrous convolution. The use of the pyramid pooling module and atrous convolution module together considerably improves the accuracy of waterfront segmentation and the gathering of global information. Finally, the Canny edge detection algorithm extracts the shoreline segmentation map from the water shorelines. The algorithm is more suitable for complex inland river scenes in the Inland dataset under various weather conditions. It can overcome the interference of reflections, water waves, and riverbank obstacles in the inland scene and accurately achieve the segmentation of the waterbank image to detect the water shoreline. For USVs operating in inland river settings, the technology could be useful for visual navigation. In terms of the detection results, the detection effect is still low for the particularly difficult to identify reflection part and the part where bright light beams to the waterfront junction due to the diversity of the inland river scene. The detection accuracy can be improved by pre-processing the waterfront image to remove environmental interference before running the detection algorithm. In terms of approach, although semantic segmentation-based methods greatly enhance the applicability of the detection, they are too demanding in terms of data annotation, requiring not only large amounts of image data but also these images need to provide accurate tagging information down to the pixel level, so weakly supervised semantic segmentation-based approaches will be a direction of development.

Declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Vorheriger Artikel Architecture entropy sampling-based evolutionary neural architecture search and its application in osteoporosis diagnosis

Nächster Artikel An agent-based modeling framework for the design of a dynamic closed-loop supply chain network

Hou R, Tang Z, Wan B, Bingcheng Y, Ren T, Wu G (2020) Development status and trend of intelligent technology of surface unmanned vehicle. China Shipbuilding 61:211–220

Barrera C, Padron I, Luis F, Llinas O (2021) Trends and challenges in unmanned surface vehicles (Usv): from survey to shipping. TransNav Int J Mar Navig Saf Sea Transport 15:135–142. https://doi.org/10.12716/1001.15.01.13CrossRef

Han J, Cho Y, Kim J (2019) Coastal SLAM with marine radar for USV operation in GPS-restricted situations. IEEE J Oceanic Eng 44:300–309. https://doi.org/10.1109/JOE.2018.2883887CrossRef

Pastore T, Djapic V (2010) Improving autonomy and control of autonomous surface vehicles in port protection and mine countermeasure scenarios. J Field Robot 27:903–914. https://doi.org/10.1002/rob.20353CrossRef

Peng Y, Yang Y, Cui J, Li X, Pu H, Gu J, Xie S, Luo J (2017) Development of the USV ‘JingHai-I’and sea trials in the Southern Yellow Sea. Ocean Eng 131:186–196. https://doi.org/10.1016/j.oceaneng.2016.09.001CrossRef

Ruangpayoongsak N, Sumroengrit J, Leanglum M (2017) A floating waste scooper robot on water surface. In: 2017 17th international conference on control, automation and systems (ICCAS), pp 1543–1548. https://doi.org/10.23919/ICCAS.2017.8204234

Madeo D, Pozzebon A, Mocenni C, Bertoni D (2020) A low-cost unmanned surface vehicle for pervasive water quality monitoring. IEEE Trans Instrum Meas 69:1433–1444. https://doi.org/10.1109/TIM.2019.2963515CrossRef

Wang W, Gheneti B, Mateos LA, Duarte F, Ratti C, Rus D (2019) Roboat: an autonomous surface vehicle for urban waterways. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 6340–6347. https://doi.org/10.1109/IROS40897.2019.8968131

Wanghui Z, Xiong Z (2020) Water-shore-line detection for complex inland river background. J Phys Conf Ser. https://doi.org/10.1088/1742-6596/1486/5/052017CrossRef

10.

Zou X, Xiao C, Zhan W, Zhou C, Xiu S, Yuan H (2020) A novel water-shore-line detection method for USV autonomous navigation. Sensors 20:1682. https://doi.org/10.3390/s20061682CrossRef

11.

Anmin Z, Jian Z, Hao Z (2021) Environmental sensing technology and application development of surface unmanned vehicle. Tech review 39:106–116

12.

Liu Y, Wang J, Shi Y, He Z, Liu F, Kong W, He Y (2022) Unmanned airboat technology and applications in environment and agriculture. Comput Electron Agric 197:106920. https://doi.org/10.1016/j.compag.2022.106920CrossRef

13.

Specht C, Świtalski E, Specht M (2017) Application of an autonomous/unmanned survey vessel (ASV/USV) in bathymetric measurements. Polish Maritime Res 24:36–44. https://doi.org/10.1515/pomr-2017-0088CrossRef

14.

Wey Y (2017) Automatic waterline detection for USV with complicated shadow background. Sci Sin Technol 48:101–109. https://doi.org/10.1360/N092017-00217CrossRef

15.

Zha Q, Huang Y (2022) Research on positioning and navigation of USV based on Lidar. In: Proceedings of 2021 Chinese intelligent systems conference, pp 700–709. https://doi.org/10.1007/978-981-16-6328-4_71

16.

Zeng W, Wan L, Zhang T, Xu Y (2012) Fast detection of sea boundary based on visible light image of sea surface. J Opt 32:90–97. https://doi.org/10.3788/AOS201232.0111001CrossRef

17.

Shao C, Cheng G, Tang K, Han J, Yang Z (2005) Recognition of water bank boundary in color image of water bank by fusing fractal dimension features. Photoelectr Eng 32:78–81

18.

Kröhnert M (2016) Automatic waterline extraction from smartphone images. Int Archiv Photogramm Remote Sens Spat Inf Sci 41:857. https://doi.org/10.5194/isprsarchives-XLI-B5-857-2016CrossRef

19.

Peng M, Wang J, Wen X, Cong X (2018) Detection of water shoreline based on the characteristics of water surface image in HSV space. Chinese J Image Graph 23:526–533. https://doi.org/10.11834/jig.170498CrossRef

20.

Sun B, Li S, Xie J (2019) Sea-land segmentation for harbour images with superpixel CRF. In: IGARSS 2019–2019 IEEE international geoscience and remote sensing symposium, pp 3899–3902. https://doi.org/10.1109/IGARSS.2019.8899001

21.

Zheng Y, Xiao Y, Cun C, Xiaong K, Zhang H, Liu R (2020) Research on shoreline extraction method based on improved regional growth. Computer Appl Res 37:1876–1881. https://doi.org/10.19734/j.issn.1001-3695.2018.12.0917CrossRef

22.

Steccanella L, Bloisi D, Blum J, Farinelli A (2018) Deep learning waterline detection for low-cost autonomous boats. In: International conference on intelligent autonomous systems, pp 613–625.https://doi.org/10.1007/978-3-030-01370-7_48

23.

Shen J, Tao Q, Xiao Z (2019) Shoreline detection algorithm combined with improved deeplab V3 + network, pp 2174–2182. https://doi.org/10.11834/jig.190051

24.

Erdem F, Bayram B, Bakirman T, Bayrak OC, Akpinar B (2021) An ensemble deep learning based shoreline segmentation approach (WaterNet) from Landsat 8 OLI images. Adv Space Res 67:964–974. https://doi.org/10.1016/j.asr.2020.10.043CrossRef

25.

von Braun M-S, Frenzel P, Kading C; Fuchs M(2020) Utilizing mask R-CNN for waterline detection in canoe sprint video analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 876–877. https://doi.org/10.1109/CVPRW50498.2020.00446

26.

McLeay AJ, McGhie A, Briscoe D, Bi Y, Xue B, Vennell R, Zhang M (2021) Deep convolutional neural networks with transfer learning for waterline detection in mussel farms. In: 2021 IEEE symposium series on computational intelligence (SSCI), pp 1–8. https://doi.org/10.1109/SSCI50451.2021.9659987

27.

Zhao H, Shi J, Qi X, Wang X, Jia J(2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890. https://doi.org/10.48550/arXiv.1612.01105

28.

Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965

29.

Wu J (2017) Introduction to convolutional neural networks. National Key Lab for Novel Software Technology, Nanjing University, China, p 495

30.

He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

31.

Xu Z, Baojie X, Guoxin W (2017) Canny edge detection based on Open CV. In: 2017 13th IEEE international conference on electronic measurement & instruments (ICEMI), pp 53–56. https://doi.org/10.1109/ICEMI.2017.8265710

32.

Deng L, Guo Y, Chai B (2021) Defect detection on a wind turbine blade based on digital image processing. Processes 9:1452. https://doi.org/10.3390/pr9081452CrossRef

33.

Cheng Y, Jiang M, Zhu J, Liu Y (2021) Are we ready for unmanned surface vehicles in inland waterways? The usvinland multisensor dataset and benchmark. IEEE Robot Autom Lett 6:3964–3970. https://doi.org/10.1109/LRA.2021.3067271CrossRef

34.

Mukti IZ, Biswas D (2019) Transfer learning based plant diseases detection using ResNet50. In: 2019 4th international conference on electrical information and communication technology (EICT), pp 1–6. https://doi.org/10.1109/EICT48899.2019.9068805

35.

Carrington AM, Fieguth PW, Qazi H, Holzinger A, Chen HH, Mayr F, Manuel DG (2020) A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Med Inform Decis Mak 20:1–12. https://doi.org/10.1186/s12911-019-1014-6CrossRef

Titel: Improved PSPNet-based water shoreline detection in complex inland river scenarios
verfasst von: Yuheng Yin
Yangang Guo
Liwei Deng
Borong Chai
Publikationsdatum: 27.06.2022
Verlag: Springer International Publishing
Erschienen in: Complex & Intelligent Systems / Ausgabe 1/2023
Print ISSN: 2199-4536
Elektronische ISSN: 2198-6053
DOI: https://doi.org/10.1007/s40747-022-00793-8

Springer Professional

Improved PSPNet-based water shoreline detection in complex inland river scenarios

Abstract

Publisher's Note

Introduction

Principle and method

PSPNet network model

Feature extraction network

Pyramid pooling module

The improved pyramid scene analysis network

Implementation of CBAM attention mechanism

Improved pyramid pooling module with atrous convolution module

The water shoreline detection

Experimental results and analysis

Introduction to the dataset

Evaluation based on improved semantic segmentation algorithm

Semantic segmentation algorithm training

Analysis of experimental results

Results and analysis of water shoreline detection under different weather conditions

Conclusion

Declarations

Conflict of interest

Publisher's Note

Premium Partner

Springer Professional

Abstract

Publisher's Note

Introduction

Principle and method

PSPNet network model

Feature extraction network

Pyramid pooling module

The improved pyramid scene analysis network

Implementation of CBAM attention mechanism

Improved pyramid pooling module with atrous convolution module

The water shoreline detection

Experimental results and analysis

Introduction to the dataset

Evaluation based on improved semantic segmentation algorithm

Semantic segmentation algorithm training

Analysis of experimental results

Results and analysis of water shoreline detection under different weather conditions

Conclusion

Declarations

Conflict of interest

Publisher's Note

Weitere Artikel der Ausgabe 1/2023

A novel density deviation multi-peaks automatic clustering algorithm

GRAN: graph recurrent attention network for pedestrian orientation classification

A coin selection strategy based on the greedy and genetic algorithm

A robust target tracking algorithm based on spatial regularization and adaptive updating model

A deterministic and nature-inspired algorithm for the fuzzy multi-objective path optimization problem

Intelligent depression detection with asynchronous federated optimization

Premium Partner