Detection of Damaged Stop Lines on Public Roads by Focusing on Piece Distribution of Paired Edges

In our previous study [1], we developed a system for localization on a digital map, in which an intelligent vehicle detected stop lines and guardrails as landmarks for localization. However, in our previous study, the experiments were conducted on a private test course that had only well-maintained stop lines. The implemented function for detecting stop lines was elementary, and it could not detect damaged stop lines on actual public roads. Thus, in this study, we aimed to develop an improved system for the detection of stop lines as an enhanced component for our localization system. Figure 3 shows a conceptual schematic of a localization system based on stop-line detection that is supported by a digital map. In this study, we aimed to develop a stop-line detection system as a first part, on the assumption of the support from a digital map as a later part.

When designing stop-line detection systems, it is necessary to reduce the number of missed and false detections and increase the number of positive ones. In this study, positive detection means that the system outputs the final detection results by using the detection process in the case when a stop line actually exists, and missed detection means that the system does not output the final detection results even though a stop line actually exists. In contrast, false detection means that the system outputs the final detection results even in the case that a stop line does not exist. Mistaking other road markings such as crosswalks for stop lines is an example of false detections. In general, a tradeoff between missed and false detections exists. For example, a horizontal long straight line, which can be extracted by the processes of edge detection and Hough transformation, is one of the characteristics of stop lines. Thus, the parameter of the assumed line length used in Hough transformation plays a key role in the tradeoff. If we assign a large value to the parameter, which indicates a long line, the number of missed detections increases because damaged stop lines may be divided. However, if we assign a small value, which indicates a short line, the number of false detections increases because the system also considers similar edges such as other road markings and patchy surfaces of repaired roads. Figures 4 and 5 show examples of patchy surfaces of repaired roads. Thus, first, determining a suitable balance of false and missed detections according to the purpose and usage is important. Considering this aspect, our system is deemed to be used as a basic component for localization. Furthermore, a digital map can exclude false detections when stop-line data are not registered. Thus, in this study, reducing the number of missed detections is more important than reducing the number of false detections. Considering this aspect, as many stop lines as possible must be detected to increase the chance of localization. Specifically, we focus on the pieces of faint features of stop lines to reduce the number of missed detections, although focusing on these features may provoke false detections.

However, the digital map cannot exclude a false detection near a true stop line although it can exclude a false detection at a distance from a true stop line. Because other road markings are usually present around the stop lines, mistaking them for the true stop line must be avoided. For example, in Japan, some crossings have bicycle crossing lanes beside a crosswalk, although the Japanese National Police Agency recently began removing the bicycle crossing lanes. Figure 6 shows an example of a bicycle crossing lane near a stop line. Although the bicycle crossing lane is represented as a dashed line in some countries, it is a continuous line in Japan. Because the shape characteristics of bicycle crossing lanes are almost the same as those of stop lines, countermeasures are necessary to prevent false detections. An approach to overcome this problem is to exclude candidates within a certain area after the first detection of a stop line because bicycle crossing lanes, if they exist, are located behind the stop lines.

In addition, the bumper of a preceding vehicle is also an object that the detection system may mistake for a stop line around a true stop line. This is because the system aims to detect stop lines by focusing on pieces of faint features. The approach for addressing this problem is to exclude false detection by using the camera-based detection results of a preceding vehicle. Because recent ADAS can often detect a preceding vehicle, we believe that this information is a practical supplement for detecting stop lines.

A stop-line detection system was developed using the basic concept discussed in Section 2.1. Figure 7 shows a flowchart of the stop-line detection algorithm. The process flow is roughly divided into four steps from the preprocess to vehicle removal, as shown on the left side of Fig. 7. Regarding the main part of the detection, the algorithm mainly focuses on the piece distribution of the paired horizontal edges, which consist of positive and negative edges. Figure 8 shows the conceptual schematic of positive and negative edges. In this study, positive edges represent the boundary from the dark pixels of the upper side to the bright ones of the bottom side. In contrast, negative edges represent the boundary from the bright pixels of the upper side to the dark ones of the bottom side. Thus, the upper sides of the road marks exhibit positive edges, whereas the lower sides exhibit negative edges. The algorithm detects the stop lines by using several steps to extract these features. Although this system contains some components used in existing approaches, such as inverse perspective mapping, horizontal edge extraction, and Hough transformation, some new components were introduced for excluding false detections, for example, edge trinarization, edge pair verification, positional confirmation, and preceding vehicle removal. The details of each step are described in the following sections. In the implementation, we used the Robot Operating System (ROS) and OpenCV libraries.

The first step is the preprocess, which involves obtaining the camera input and performing inverse perspective mapping, as shown in Fig. 7. Figure 2 shows an example of a camera input image. The proposed system involves a monocular camera (GS3-U3-15S5C-C, produced by FLIR) installed inside the windshield of an experimental vehicle. Because the system aims to detect various traffic elements such as road markings, other traffic participants, and traffic lights using only a single camera, a color camera with a resolution of 1280 × 960 is used. However, because the following process needs only grayscale information, we convert the color input image to a grayscale image. Then, we perform inverse perspective mapping and resize the image to 640 × 480. Figure 9 shows an example of the inverse perspective mapped image.

The second step starts with horizontal edge extraction. A Sobel filter is used to extract the horizontal edges from the inverse perspective mapped images. Figure 10 shows an example of the horizontal edge images. The grey areas do not contain any edges. In contrast, the black and white lines represent horizontal edges. Specifically, the black lines represent negative edges whereas the white lines represent positive edges.

To determine the positive and negative edges more clearly, we trinarize the horizontal edge images. Figure 11 shows a conceptual schematic of the trinarization of an edge image. The leftmost image in Fig. 11 shows a part of the horizontal edge image, and the red line indicates the cross-section line. The left graph indicates the scaled edge value along the cross-section line. The negative and positive values correspond to the negative and positive edges, respectively. Subsequently, we trinarize the local extrema, the absolute values of which are larger than a certain value. The right graph indicates the result of the trinarization process, and the rightmost image represents the corresponding part of the trinarized image. The trinarization process is repeated for all vertical cross-section lines over the image. Figure 12 shows an example of a trinarized horizontal edge image. The black and white lines respectively indicate the trinarized negative and positive edges.

Although the trinarized horizontal edge image contains the edges of actual stop lines, it also contains the edges of other road marks such as parts of crosswalks. To exclude the edges of non-targets, the positional relation between the positive and negative edges is observed. For all vertical cross-section lines, the pixels are scanned from the bottom side. If a pixel of negative edges is located under a pixel of positive edges within a certain distance, the negative pixel is considered to be a part of the verified pairs.

Because we assume that the paint of stop lines on the road is damaged, the width between both edges may not be ideal. Thus, we use a relatively relaxed threshold value for the width confirmation, although this threshold value cannot exclude all non-target edges. This verification process is repeated for all vertical cross-section lines over the image. Figure 13 shows an example of the verified paired edges, in which the white line indicates the verified paired edges. As shown in Fig. 13, the edges of the stop lines are adequately extracted. However, because the size of the damaged road marks of parts of a crosswalk is accidentally nearly the same as that of actual stop lines, the image of verified paired edges shown in Fig. 13 still includes the edges of non-targets.

To exclude the remaining edges of non-targets, line segments with lengths larger than a certain threshold are selected by using a probabilistic Hough transformation. Figure 14 shows an example of the extracted line segments, which indicates that the edges of non-targets are adequately removed. Figure 15 shows an example of a sequentially processed images. Subsequently, we determine the number of pixels of the extracted line segments for each height, and measure the peak height of the areas that contain the maximum number of pixels. Subsequently, the line segments distributed around the peak height are grouped, and both ends of the stop line are determined by considering the leftmost and rightmost pixels in the group. If the number of extracted pixels around the peak height is larger than a certain value, and if the length of the candidate is larger than a certain value, the candidate is considered valid. Using the results obtained from these processes, we confirm the shape of a candidate of a stop line for a certain frame. In addition, to reduce the number of unexpected false detections, we track the detected results of a few continuous frames. A candidate of a stop line is determined using these processes. Figure 16 shows an example of the final detection result. The green line indicates the detection result that the proposed system automatically draws in the image of the camera input.

The third process involves the removal of the bicycle crossing lane. As discussed in Section 2.1, some large crossings have bicycle crossing lanes. Because the shape characteristics of these lanes are similar to those of stop lines, they cannot be discriminated in the main detection process. To overcome this limitation, we log the data of the estimated driving distance calculated using the time-series velocity, which is obtained from the Control Area Network (CAN) when the vehicle approaches a stop line. Then, to prevent false detections, the output of the main detection process is ignored for a certain distance from the detected stop line. Figure 17 shows the conceptual schematic of the ignored area for preventing false detections pertaining to the bicycle crossing lane. The distance parameter to ignore the candidates was 35 m, considering the size of the intersections. Although we used a constant value in this study, in future studies, it is desirable for this parameter to be variable by referring to the digital map.

The fourth process involves the removal of vehicles. To detect a preceding vehicle, we use YOLO [15], which is an open-source software for vehicle detection. Using the image of the camera input, YOLO outputs a bounding box that contains the area of the preceding vehicle. If the detected candidate of a stop line passes through the bounding box, we consider the candidate as a false detection and the final detection results are not output.

Figure 18 shows an example of vehicle removal. The left image is a typical example in which the main detection process easily makes a mistake because the color of the bumper is relatively bright, and the edge is relatively straight. The blue line in the right image shows the temporary false detection result, and the red rectangle shows the bounding box detected by the vehicle detection module. In this case, because the line candidate is inside the bounding box, the result is considered a false detection.

In some existing studies regarding computer vision, open benchmark data sets were used for evaluating the proposed methods. However, such data sets contain driving data for European countries where some road markings are different from those in Japan. Because the proposed method is developed considering the traffic situation in Japan, such benchmark data sets could not be used. Thus, to evaluate the performance of the proposed system, we collected driving data, which contained front monocular camera images at 15 Hz and the estimated driving distance at approximately 100 Hz, for actual public roads. As a first step, the roads near the university were set as the evaluation course. Specifically, two evaluation courses were prepared: a clockwise course and a counterclockwise course. Figure 19 shows an aerial photographic map of the evaluation course. This figure is based on a map image published by the Geospatial Information Authority of Japan [16]. Figures 1, 2, 4, 5, and 6 correspond to these courses. Because these courses consist of municipal roads, the surface of the roads was not maintained as high-standard roads are, such as highways. The total length of the course was approximately 3 km.

Although this course contained many stop lines, the evaluation targets were 12 stop lines which the experimental vehicle drove over in a straight manner. Therefore, for example, the stop lines on both sides of the crossings were not evaluation targets. We collected the image data for this course using a monocular camera and the data of estimated driving distance. The sensor data for both courses was collected three times to confirm the reproducibility of detecting stop lines to a certain degree, although the number of trials was limited. The weather was slightly cloudy; thus, only a few shadows of surrounding road construction were present. After the collection, we evaluated the offline performance of the proposed system.

We evaluated the computational speed of the proposed system corresponding to the stop-line detection by measuring the average calculation speed across 50 frames. The detection process of stop lines, which does not include the detection process of a preceding vehicle, took 0.021 s (47.6 Hz) on average when using a desktop PC (3.0-GHz CPU under the condition of hyperthreading). Because the frame rate of the camera in the proposed system was 15 fps, real-time detection of a stop line could be realized. In this manner, the processing speed of the proposed system was confirmed to be practical.

To evaluate the detection performance of the proposed method, we compared the detection results obtained using the proposed method with those obtained using conventional methods. The characteristic aspect of the proposed system was the reduction of missed and false detections by employing the processes of edge trinarization, edge pair verification, positional confirmation, and preceding vehicle removal. Thus, for comparison, conventional methods that did not include the abovementioned processes were considered. Specifically, the conventional methods only included the processes of inverse perspective mapping, horizontal edge extraction of negative edges, and Hough transformation. In addition, to discuss the effect of assuming an ideal shape of stop lines, two settings were considered for the comparison methods, specifically, conventional methods 1 and 2. In the process of Hough transformation, the parameter of minimum line length was considered. In our inverse perspective mapped images, the length of a single lane was larger than 110 pixels. Because our proposed method aimed to select as many small line segments as possible, the parameter of minimum line length for the proposed method was set as 5 pixels, which represented a substantially short length. For conventional method 1, the same value as that for the proposed system was assigned. However, for conventional method 2, a value of 66 pixels was assigned, which corresponded to 0.6 times the minimum lane width. Thus, conventional method 2 assumed a relatively ideal shape of stop lines whereas conventional method 1 did not involve such an assumption.

Table 1 presents a summary of the detection results. Three types of detection results: positive, missed, and false were considered. To account for the false detection, we checked the objects that the systems mistook for stop lines, and these objects were classified into five categories: crosswalks, other road markings, patchy road surfaces, bicycle crossing lanes, and preceding vehicles. In terms of the positive detection, the proposed method and conventional method 1 could detect all the stop lines whereas conventional method 2 could detect only approximately half of the stop lines. Because the proposed method and conventional method 1 did not assume the ideal shape of stop lines, they could detect stop lines even in cases involving damaged paint. However, conventional method 1 generated several false detections; because conventional method 1 detected small edges without paired edge verification, it mistook crosswalks, other road markings, and patchy road surfaces of repaired road for stop lines. In this regard, the proposed method could prevent false detections owing to paired edge verification, whereas conventional method 2 could prevent false detections owing to the assumptions of ideal shape. Considering the bicycle crossing lane, the proposed method could prevent false detections owing to the confirmation of estimated driving distance, whereas both the conventional methods led to false detections. In terms of the preceding vehicle, the proposed method could prevent false detections owing to supplemental information being obtained from the detection module of the preceding vehicle, whereas both conventional methods led to false detections. In summary, conventional method 1, which did not assume the ideal shape of stop lines and did not include additional modules for preventing false detections, caused many false detections whereas conventional method 2, which assumed an ideal shape without additional modules, caused many missed detections. In contrast, the proposed method, which did not assume an ideal shape and included additional modules for preventing false detections, could prevent missed and false detections. The following subsections present the details of some characteristic cases.

Figure 20 shows an example of a damaged stop line, which the proposed system could detect correctly. Although the paint on the road was damaged, as shown in Fig. 20, the proposed system detected only the target stop line. Figure 21 shows the midstream images of each process. Figures 21-A, 21-B, 21-C, and 21-D show the inverse perspective mapped image, trinarized horizontal edge image, verified paired edge image, and line segments extracted by the Hough transformation, respectively. As shown in Fig. 21-A, several road markings are present. In addition, because the paint of the crosswalk is relatively damaged, many positive and negative edges are extracted, as shown in Fig. 21-B. However, owing to the process of edge pair verification, a large portion of the extracted edges is removed, as shown in Fig. 21-C. Furthermore, the remaining small edges are also removed by the process of probabilistic Hough transformation, as shown in Fig. 21-D. In addition, the proposed system removed the remaining edges of non-targets by confirming pixel distribution. As a result, the proposed system could output a suitable final detection result without false detections, as shown in Fig. 20.

Figures 22 and 23 show examples of a bicycle crossing lane near a stop line. Although the paint of both the stop line and bicycle crossing lane exhibits certain damage, the proposed method could detect them as candidates of stop lines. Thus, our proposed system referred to the estimated driving distance data. First, when the vehicle approached the stop line, the proposed system logged the value of the estimated driving distance. For the situations shown in Figs. 22 and 23, the estimated driving distances were 3733.6 m and 3741.8 m, respectively. Because the distance from the first candidate of the stop line shown in Fig. 22 to the second candidate shown in Fig. 23 was smaller than the assigned threshold parameter, the proposed system considered the second candidate as a false one and prevented a false detection.

Figure 24 shows an example of the prevention of false detection pertaining to a preceding vehicle. Similar to in Fig. 18, the blue line represents the temporary false detection result, and the red rectangle shows the bounding box detected by the vehicle detection module. Figure 25 shows the midstream images. As shown in Fig. 25-A, the rear part of the preceding vehicle in this case is similar to the actual stop line, from the perspectives of color and shape. Thus, the considered processes could not eliminate the false detection, as shown in Figs. 25-B, 25-C, and 25-D, and our proposed system considered the line segments as candidates of the stop line. However, because the candidate is inside the bounding box of the detected preceding vehicle, our proposed system finally considered this candidate as a false one and prevented a false detection.