Provident vehicle detection at night for advanced driver assistance systems

First, it must be noted that current camera-based perception models used to detect vehicles at night are object detectors. As the most reliable information source, headlamps of other vehicles are used to detect the position of vehicles. Consequently, headlamps are used as “objects” from which succeeding systems can infer the location of vehicles present in the image (e. g., Alcantarilla et al., 2011). While being a robust reference, the restriction on headlamps limits the performance of such systems, since the earliest point in time they can perceive a vehicle is when they have direct sight to the vehicle (see “object becomes visible” in Fig. 2 and the middle image in Fig. 1). As already mentioned in Sect. 1, this differs from how humans estimate whether and where a vehicle is oncoming because humans can react to light artifacts like the light reflections on the guardrail in Fig. 1. Thus, the question is why light artifacts are not naturally being detected or tried to being detected by current vehicle detection systems considering the apparent discrepancy between humans and systems regarding this task (see the time gap between the human provident and camera-based object detection in Fig. 2). We can only speculate why this is the case but expect that one reason is the object detection paradigm: The algorithms detect objects that match an object definition. For example, if the object detector is a bounding box regressor, it must be possible to specify the object boundaries to define bounding boxes. In vehicle detection at night, it is possible to apply this strategy for headlamps (direct light instances) as they cause intensity singularities in the image (extraordinary high-intensity peaks) with sharp gradients. However, for light reflections (indirect light instances), this strategy is not appropriate since they often illuminate almost homogeneously larger areas with small gradients, and their intensity varies heavily depending on their strength and other global light sources. Therefore, indirect light instances cannot be treated as objects without further thoughts due to their unclear object boundaries, causing annotation difficulties. Building on the work of Saralajew et al., 2021, we tackle this challenge by using keypoint annotations to derive a suitable object detector for detecting all sorts of light artifacts.

As already mentioned in Sect. 3, Oldenziel et al., 2020 presented the results of a test group study that investigated the detection latency of an in-production computer vision system and humans—the results are summarized in Fig. 2. In particular, they observed that even if, in theory, detection systems are able to detect vehicles directly after direct sight, on average, they have a system-related latency. This latency is caused by a small delay until the object paradigm is fulfilled, by the image processing steps of the detection pipeline (as described in Sect. 1), and by the time needed to perform the plausibility check. In summary, the following steps cause the latency in a vehicle detection system—which is also qualitatively visualized in Fig. 2: This latency is frequently discussed in combination with glare-free high beam systems because of the possible caused glare (e. g., Helmer, 2020, Chapter 3.3.1): Hummel, 2009, Chapter 5.4.3 reported that the plausibility check causes a latency between two and four frames, where a frame has a processing time of \(45\,{\textrm{ms}}\), and measured a delay from capturing the image until the high beam adapted of \(\left( 196\pm 21\right) \!{\hbox {ms}}\).⁵ Totzauer, 2013, Chapter 3.2.4 reported a similar result of \(\left( 288\pm 17.4\right) \!{\hbox {ms}}\) total system delay. Moreover, López et al., 2008 reported a validation time of two or three frames for oncoming vehicles. Unfortunately, they do not specify the camera’s frame rate, so these results cannot be converted into seconds. However, they state that the overall processing delay is less than \(200\,{\textrm{ms}}\). The approach of Alcantarilla et al., 2011 validates detected vehicles for five frames at a frame rate of 30, leading to a minimum delay of around \(167\,{\textrm{ms}}\). Summarily, current camera-based systems for detecting vehicles at night are expected to have an inherent detection delay caused by the mandatory vehicle validation procedure and the assumption that vehicles are characterized solely by headlights (as discussed in the previous Sect. 4.1). Consequently, if such a vehicle detection system is used to deploy a glare-free high beam system, it has to be expected that oncoming drivers are exposed to high beam light patterns when they appear in direct sight. Whether these glare moments are critical has not yet been conclusively clarified (Helmer, 2020).

Not only object detection systems have an internal latency, but also humans: reaction time. In Fig. 2, the human reaction times during the test group study are illustrated. As Oldenziel et al., 2020 showed, the camera-based vehicle detection is approximately 200 ms faster in a fair setting (allowed detection after direct sight) than its human counterpart. However, human detection almost reaches the minimal possible detection time, acting only approximately 800 ms after the first indication of oncoming vehicles (compare with the reaction times for braking maneuvers Green, 2000).

In Sect. 5, we present a detection system that is able to detect light artifacts. Even if this system reacts to light artifacts, it still has the inherent system-related latency. Therefore, depending on the scenario, it does not necessarily detect oncoming vehicles before direct sight but shifts as much as possible of the inherent latency before the moment of direct sight and, thus, detects oncoming vehicles faster than current systems do.

The first element in the vehicle detection pipeline is the object detector. The task here is to detect both direct and indirect light instances within the camera image. The feasibility of the light artifact detection was shown by Oldenziel et al., 2020 through multiple practical examples. The general setup for such a detector can be divided into the following sub-tasks: This pipeline is used in many state-of-the-art systems as well as in machine learning object detectors (e. g., Ren et al., 2015). The usual approach for such a system would be to use an NN-based system in an end-to-end manner. As already said, we rejected this approach because of its inevitable obscure nature and computational load. Instead, the method proposed by Saralajew et al., 2021 with a tailored region proposal generation algorithm and an NN classifier is used. The resulting overall pipeline is depicted in Fig. 3. In the following paragraphs, each module of the object detector is described in more detail.

First, a dynamic thresholding procedure is performed to retrieve intensity regions of interest from the image. Bounding box proposals are then inferred through a blob detection in the generated binary image (1: above threshold; 0: below threshold). The classification is performed using a small Convolutional NN (CNN). The results are bounding box representations of light artifacts within the image.

Image preparation—The raw image is filtered to reduce the amount of noise present. First, the image size is decreased to half (640\(\times \)480 pixels) by a bilinear interpolation in order to suppress small noisy image regions (it also reduces the computational complexity of the later steps). Second, noise is further removed by applying a Gaussian blur over the entire image. This smooths out edges and removes high-frequency noise like salt-and-pepper noise. The effects of the filtering are depicted in Fig. 3b.

Dynamic thresholding—Due to the low intensities for light reflections and glares, a global thresholding strategy is not suitable to retrieve interesting regions from the raw image. In contrast, all considered artifacts share the common feature of a higher intensity relative to their surroundings (Saralajew et al., 2021). This can be used to perform dynamic thresholding on the image. Therefore, a pixel-wise threshold is calculated to retrieve interesting regions.

The criterion for the dynamic threshold T(x, y) at pixel (x, y) is defined as the following:with \(\mu (x,y)\) being the local mean intensity—calculated over a fixed-sized window w around the pixel (x, y)—and \(\varDelta (x,y)=I(x,y)-\mu (x,y)\) being the deviation of the pixel intensity I(x, y) from the local mean. The sensitivity of this threshold can be adjusted using the factor \(\kappa \in \mathbb {R}\). Equation 1 is adapted from Singh et al., 2011, who originally developed this technique to binarize documents. Comparisons with other threshold techniques showed that this method yields high-quality results. Also, by using the integral image to compute the local means, this method can be efficiently implemented (Singh et al., 2011). The threshold T(x, y) is calculated for every pixel in the filtered image and used to infer a binary image B(x, y) withAn example of such a binary image is shown in Fig. 3c.

Blob detector—The binary image contains multiple, unconnected regions generated by the thresholding step. For ease of use and further handling, these regions are compressed into bounding boxes. This is achieved by applying a standard blob detection routine to find connected areas and allowing gaps of the size d—measured with respect to the \(L_{\infty }\) distance.⁶ After the bounding boxes have been computed, they are filtered by removing bounding boxes where the mean absolute deviation of the included intensity values is smaller than a threshold s.

Classification—The bounding boxes still contain many false positives because, simply put, all bright areas of the image are detected. This allows for a high recall of interesting regions but also yields a low precision and therefore reduces the quality of following modules (e. g., the glare-free high beam functionality). Therefore, a shallow NN is added to classify each of the proposals (this strategy is similar to a Faster-RCNN architecture). For that, to provide context information for each bounding box, an enlarged region around each proposal bounding box is passed through a CNN. The network classifies the proposal to be either true positive or false positive (see also Sect. 6.2 for a formal definition of true and false positive), leading to a binary classification problem. Here, a proposal is considered true positive if it coincides with a light artifact (direct or indirect) of oncoming vehicles. Therefore, false-positive proposals are all remaining regions. The reason for posing this as a binary classification problem is explained in more detail in Sect. 6.1. As the network architecture is equivalent to Saralajew et al., 2021, we will not discuss the classifier architecture in detail. For more information, see Saralajew et al., 2021 or the publicly available implementation in the corresponding GitHub repository.

The approach used to detect light artifacts was chosen to allow for a suitable implementation in a production car, where only limited computational resources are available. While many detection and recognition systems designed for the automotive context rely heavily on parallel computing (e. g., through GPUs, TPUs), such hardware is not yet implemented in most production cars, limiting the practical usage of these systems. Even if more and more computational power is available in the upcoming years, resources will always be limited as the number of functions increases as well. Therefore, two of the major requirements for the detection system are to be computationally efficient and to not rely too heavily on additional hardware. The simple operations used to build the proposal generation are a result of these requirements. The classification is still performed on a GPU but is still efficient enough to be implemented on a production car’s hardware with only minor adjustments. As shown in literature, end-to-end learned systems outperform conventional methods like the proposed one. This is also partly true for our case when computational resources are unlimited. However, this fact changes with the constraint of limited computational resources, as shown by the evaluation in Sect. 6.2.

In real-world driving scenarios, it is often not sufficient to just provide the spatial information of the detected object in the two-dimensional image space. Only knowing where the object of interest is located in the environment enables the vehicle to react appropriately—for example, for performing emergency brakes or adjusting the adaptive high beams. Therefore, it is necessary to compute an estimate for the three-dimensional position of detected light artifacts.

As discussed in the related work in Sect. 3, in the literature, there are several methods described to perform the distance estimation⁷ to locate objects. However, the special use case of nighttime images captured by a monocular grayscale camera adds clear restrictions. The general problem is that the images are relatively dark and low-textured (e. g., see Fig. 1), which complicates the application of state-of-the-art depth estimation methods. Also, light reflections can be considered non-rigid, arbitrarily deforming objects over time. In addition, the overall goal of running the method in real-time places a constraint on computational complexity, and the goal of using it for a real-world use case requires a certain range.

Figure 1 presents a summary of possible applicable distance estimation methods. Due to the aforementioned constraints, the applicability of depth estimation from video and structure from motion is not possible. Additionally, monocular, single image depth estimation approaches have a too high computational complexity and cannot provide the required range, so they cannot be used for the studied approach as well.⁸ Hence, the only applicable state-of-the-art method is the object localization through ground plane approaches. Note that this method assumes that an object is located on the ground plane, which is not necessarily true for light artifacts (e. g., a light artifact on a guardrail). In previous work about vehicle detection where the vehicle’s headlamps are detected (e. g., Alcantarilla et al., 2011), the fulfillment of this assumption is achieved by assuming a fixed, known height of the headlamps and by shifting the ground plane by this height.

To overcome the limitations of the methods listed in Fig. 1, we also evaluated a rather unconventional method for estimating the distance of light artifacts by fusing the position of the object in the image with Predictive Street Data (PSD). The PSD protocol contains information about the road geometry ahead of the vehicle (see Fig. 4) based on map data and GPS and is used, for instance, for advanced navigation functionalities or adaptive cruise control. With this data, the road lying ahead can be projected into the vehicle coordinate system (see Fig. 5), giving a three-dimensional representation of the road geometry. For our implementation, the road ahead described by the PSD is defined as a set of n points \(\mathcal {P}=\left\{ P_{0},P_{1},P_{2},\dots ,P_{n}\right\} \), where \(P_{i}\) is a point in the vehicle coordinate system lying on the road ahead described by the PSD. A point was sampled for every meter. At the same time, knowing the intrinsic and extrinsic camera calibration,⁹ a point \(\left( x,y\right) \) in the image can be associated with a camera ray \(\mathbf {\textbf{x}}\left( t\right) \) in the vehicle coordinate system (e. g., see Hartley & Zisserman, 2004). Assuming that a detected light artifact always lies on or at least close to the road, the ray \(\mathbf {\textbf{x}}\left( t\right) \) and the road ahead described by \(\mathcal {P}\) are used to search for the closest point \(P_{i}\in \mathcal {P}\) with respect to \(\mathbf {\textbf{x}}\left( t\right) \), which is an intersection between the road ahead and the ray in the best case. This point is then considered as the object position in the vehicle coordinate system. The distance between a point \(P_{i}\) and the ray \(\mathbf {\textbf{x}}\left( t\right) \) can be calculated bywhere \(\mathbf {\textbf{n}}\) and \(\mathbf {\textbf{a}}\) are the parameter vectors of the ray \(\mathbf {\textbf{x}}\left( t\right) =\mathbf {\textbf{a}}+t\mathbf {\textbf{n}}\), and where \(\mathbf {\textbf{p}}_{i}\) is the canonical vector representation of the point \(P_{i}\). With this, the closest point of the road ahead is given bywhich determines the position of the object in the vehicle coordinate system—detected in the image at position \(\left( x,y\right) \) (Table 1).

Based on the PSD data and the available ground plane approaches, the following four methods are analyzed in the experiments:

The object detection and distance estimation are frame-based computations and, therefore, can be unstable with respect to the temporal context (e. g., if a vehicle gets occluded). To improve the detection stability, object tracking algorithms are usedIn the literature, the dominating algorithms for this are \(\alpha \)-\(\beta \) filters (e. g., Pham & Yoo, 2020, López et al., 2008) and Kalman filters (e. g., Alcantarilla et al., 2011, Teoh & Bräunl, 2011), whereas the former is a derivative of the latter. Due to the computational efficiency of the \(\alpha \)-\(\beta \) filter and because good estimates for the noise covariance matrices to instantiate a Kalman filter are not known (especially for light artifacts), the proposed tracker is mainly composed of \(\alpha \)-\(\beta \) filters:Between different frames, the object matching is performed by computing the intersection-over-union between the tracked objects and the detected objects and assigning the detected objects to the tracked objects with the highest intersection-over-union. To handle noise in the detections with respect to the bounding box size, the bounding box size of the detected objects is slightly increased before the intersection over union is computed.

If an object is occluded (not detected in the last frame), the prediction of the \(\alpha \)-\(\beta \) filter is used to forecast the position of the object for a maximal number of three frames before it is removed from the list of tracked objects. Additionally, to increase the precision of the vehicle detection system, an object is only output when it is already detected for a minimal number of five frames and if the estimated confidence is greater than a threshold of 0.5—thus, the tracker also operates as a plausibility checker, which is a common strategy as discussed in Sect. 4.2. Finally, to lower the number of tracked objects, the tracker only considers objects with a confidence value greater than 0.1.

PVDN dataset: For the evaluation of the object detector performance, the detection times, and run-times, the PVDN dataset (Saralajew et al., 2021) is used. This dataset contains 59 746 grayscale images with a resolution of 1280\(\times \)960 pixels where all light artifacts—both direct (e. g., headlamps) and indirect (e. g., light reflections on guardrails)—of oncoming vehicles are annotated via keypoints. The underlying sequences of the images are recordings of test drives on rural roads with a single oncoming vehicle or multiple oncoming vehicles. Several images in the dataset include artificial light sources like street lamps that increase the difficulty of detecting light artifacts correctly. As the authors of the dataset argue, the keypoint annotations allow for an objective annotation by placing the keypoint on the intensity maximum of each light artifact. Also, from this, an automatic generation of bounding boxes is possible, which becomes useful because most of the state-of-the-art object detectors rely on bounding box annotations. Since those bounding boxes are inferred automatically, it may happen that one bounding box covers both direct and indirect instances at the same time. This is why the task of detecting bounding boxes on the dataset is currently framed only as a binary classification problem, namely whether the bounding box covers a relevant light artifact caused by an oncoming vehicle (either direct or indirect) or not.

The images are frames of recorded video sequences so that the temporal relationships within the images of a sequence are preserved. Each scene is recorded with 18 Hz, either with a short exposure (day cycle, darker images) or long exposure (night cycle, brighter images). For the experiments in this work, the day cycle data is used as the shorter exposure results in a stronger contrast between the background and light artifacts. Within the PVDN dataset, each illumination cycle is split into a train, a validation, and a test dataset to enable the development, evaluation, and testing of algorithms. Most importantly, the sequences of the dataset contain tags that mark the timestamps whereThose tags were collected during the annotation process and the test group study, which was performed when the dataset was recorded.

Distance evaluation data: Since the PVDN dataset does not contain depth data, an additional small dataset for the evaluation of the distance estimation methods was recorded. The dataset consists of 24 scenes with 438 images in total (181 direct and 257 indirect light instances). Each scene contains five consecutive image frames to later allow for time series analyses. An exemplary sample is shown in Fig. 6. The light instances (both direct and indirect ones) were annotated manually with bounding boxes. The ground truth depth data was captured using a Hesai LiDAR sensor and the same camera system that was used to record the PVDN dataset. The single ground truth depth value for each light instance was calculated using the median of all available depth measurements within a respective bounding box.

Test car and software framework: A test car is used as the platform for deploying the pipeline for a real use case. It has to be noted that the test car was also used for recording the PVDN and distance evaluation dataset. Consequently, the same image input specification as for the PVDN dataset hold. Furthermore, the test car has a glare-free high beam system (Böke et al., 2015, Knöchel-mann et al., 2019), which is used in the experiment for visual demonstrations and for deploying a provident glare-free high beam system. Each headlight consists of 84 LEDs, and each is almost illuminating another solid angle (within a headlight) and can be dimmed independently to all the other LEDs. Therefore, if an oncoming vehicle is detected and the glare-free high beam system is activated, the individual LEDs where the vehicle is located can be turned off such that the overall headlight system stays in “high beam mode” without blinding the oncoming vehicle—the vehicle moves in a black corridor. This corridor can be perceived by the driver, making it possible to assess the light artifact detection quality in a real environment.

To perform the experiments, an additional compute platform is used consisting ofThe implementations on this platform are done using Python and C++ in the Melodic distribution of the Robot Operating System (ROS).¹² The ROS network consists of the following nodes connected sequentially to each other: All nodes run single-threaded on a CPU except the shallow NN, which is executed on the GPU. With running most of the algorithm parts single-threaded on a CPU, we want to underline the transferability of the algorithms to a series platform with hardware acceleration.

The object detector described in Sect. 5.1 is conceptually equivalent to the detector proposed by Saralajew et al., 2021 and consists of a rule-based proposal generation algorithm and an NN-based classifier. Originally, the parameters of the proposal generation algorithm were selected by a random search using the PVDN dataset. To improve this selection, a hyperparameter search for the proposal generation algorithm is performed using the tree-structured Parzen estimator approach (Bergstra et al., 2011). This approach belongs to the family of sequential model-based optimization approaches and is a standard algorithm for hyperparameter optimization. Before defining the objective function, the bounding box quality score and the events to define the F-score, recall, and precision is introduced (Saralajew et al., 2021): The goal of the bounding box quality score is to define a measure to assess the quality of a bounding box prediction algorithm by using ground truth keypoints. Because each light artifact is annotated by exactly one keypoint and each bounding box should span exactly one light artifact, in the best case,To formalize this concept, the following events for keypoints and bounding boxes are introduced:By using these events, the F-score, precision, and recall can be computed. Additionally, to quantify the quality of true-positive bounding boxes, the following quantities are calculated:To convert these numbers into values in the range \(\left[ 0,1\right] \), where one means best performance and zero worst, the reciprocal value is taken. Finally, the values are averaged across the dataset to obtain measures for the performance of a detector:where \(N_{B}\) is the total number of true-positive bounding boxes, and \(N_{K}\) is the total number of keypoints covered by bounding boxes. These two measures quantify the uniqueness of predicted bounding boxes that are true positives with respect to how many keypoints are contained within a bounding box and how many bounding boxes cover the same keypoint. For example, in an image with several keypoints, \(q_{K}\) is low if a large bounding box spans over the whole image. The overall bounding box quality is determined by \(q=q_{K}\cdot q_{B}\), where a value of one indicates best performance and zero worst.

Using the definition of the bounding box quality q based on the introduced events, the objective function \(h\left( \theta \right) \) of the hyperparameter optimization iswhere \(\theta \) is a hyperparameter configuration. This objective function encourages bounding box generators (proposal generation algorithms) where each keypoint is uniquely assigned a bounding box. The specific search space configuration can be found in Table 2.

We optimized the hyperparameters on the official PVDN training set, selected the best parameters based on the objective function value on the validation set, and reported the results on the test set. These optimized hyperparameters were then used to generate an optimized set of bounding box annotations for the PVDN dataset: bounding boxes that cover a keypoint (a light artifact) were kept as ground truth bounding boxes. With these optimized bounding box annotations, the classifier was trained to distinguish between bounding boxes that contain light artifacts and bounding boxes that do not. Similar to the hyperparameter optimization, all three dataset splits of the PVDN dataset were used accordingly to train the proposal classifier. The classifier was trained for 300 epochs with an initial learning rate of 0.001, batch size of 64, weight decay of 0.01, and binary cross-entropy. Moreover, the Adam optimizer (Kingma & Ba, 2015) was used, and images were augmented with horizontal flips, rotations, crops, and gamma corrections while training. The confidence threshold for a valid classification of a light artifact was set to 0.5. To foster public use and to ensure reproduction, the whole pipeline implemented in Python with the deep learning framework PyTorch¹³ is publicly available.¹⁴ In order to make our custom detection approach comparable to state-of-the-art end-to-end object detection algorithms, we also report the performance of both YoloV5s and YoloV5x.

Table 2 shows the results of the hyperparameter optimization. Using these parameters to generate the bounding box annotations, the optimized proposal generator achieves the results reported in Table 3 (see values in parentheses). The optimized proposal generation algorithm increases the bounding box quality q from 50 % to 70 % and increases the F-score from 84 % to 93 %. Therefore, the optimized proposal generation algorithm shows a clear improvement of the automatically inferred bounding box annotations compared to the baseline. However, the performance of the optimized region proposal algorithm is still not optimal because not each performance value is 100 %. The scores which are at 100 % must be at this level due to the construction principle of the bounding boxes: Each ground truth bounding box in the generated annotations is a valid bounding box so that the precision must be 100 %; It is likely that each ground truth bounding box only covers one keypoint due to the construction of the bounding boxes by non-maximum suppression so that the quality \(q_{B}\) must be close to 100 %.

In summary, for embedded and safety-critical purposes, our custom approach is to be chosen before YOLOv5. Because it has significantly fewer parameters and thus requires less memory, and has high optimization potentials for embedded hardware. Also, the proposed optimized object detector consisting of a rule-based proposal generation algorithm and a shallow NN-classifier is more transparent than the deep NN-architectures of YOLOv5. Therefore, the approach is easier to validate and verify for safety-critical applications since the model behavior can be better understood and interpreted. Finally, we conclude that the optimized and trained light artifact detector sets a new benchmark for the PVDN dataset.

This section presents the evaluation of the proposed methods of Sect. 5.2 on the dataset described in Sect. 6.1. First, the general performance of each method is evaluated on the available data. For that, each light artifact marked by a bounding box is transformed to a single pixel by taking the center of the bounding box. Performance results are shown in Fig. 7. It becomes clear that with a median relative error of 0.12 for direct and \(-0.32\) for indirect light instances, the Ground Plane approach (GP), which only considers the area ahead as a plane, outperforms approaches PSD-3D, PSD-3D+, and PSD-2D, which try to estimate the road geometry using the PSD. A negative error means that the estimated distance is less than the ground truth distance. An in-depth analysis of the PSD shows that the positioning of the vehicle on the road described by the PSD is often too inaccurate to give a precise enough representation of the exact road geometry ahead, which is needed in order for the PSD approaches to work. Especially in curves, an accurate positioning of the vehicle on the road segment is absolutely mandatory since even a slight deviation can cause a considerable discrepancy between the actual road geometry ahead and the one described by the PSD at a specific time step.

When looking at the performance of the GP method, a performance deficit between direct and indirect light instances becomes clear. There are several possible reasons for this. First, the direct light instances are always located further away from the ego-vehicle than the indirect ones. The direct instances in the underlying data have an average distance of 83 m, whereas indirect instances are, on average, 63 m away. Therefore, inaccuracies influence the relative error more for the indirect instances. Second, indirect light instances often span over a large area (e. g., on the street), where the acquisition of a single ground truth distance value is difficult as the beginning of the annotated area has a different distance value than the end. This can lead to partly inaccurate ground truth values. Third, all of the mentioned methods strongly depend on the quality of the intrinsic and extrinsic camera calibration. Thus, unknown inaccuracies in the calibration can also affect the result. Finally, the assumption of the environment ahead being a plane could often be inaccurate. If the assumption were true, the expected result for indirect light instances on the road would be nearly perfect compared to the ground truth, whereas all light instances located above the road (e. g., headlights or light reflections on guardrails) would give a too far away distance, as the camera ray would intersect with the plane behind the actual light artifact. However, the results show that the direct light instances are nearly perfect (only a little overshooting distance estimation of approximately 12 %), whereas the distance estimation for indirect instances falls too short. This indicates that the data often contains scenes where the ground plane assumption does not hold.

To investigate whether the distance estimation can be improved by considering the distance estimate of each pixel within a bounding box, several heuristics are analyzed. As the methods using the PSD already did not show satisfying results in the first experiment, the following evaluations are only done for the GP method. To retrieve the final distance estimation from all distance values within a bounding box, five simple approaches were compared with each other: The results are shown in Fig. 8. Interestingly, the five approaches do hardly show any improvements. Only the approach of taking the maximum distance value within a bounding box improves the distance estimation for indirect light instances, which makes sense considering that the original estimation for indirect light instances was often too short.

Since the annotation format of the dataset was chosen so that the correspondence of light instances across multiple frames can be determined, in a final experiment, the distance estimation for the GP method is attempted to be stabilized by considering a series of consecutive distance estimations for the same instance. The idea is that with this, possible outliers can be filtered. For that, the two approaches of taking the median or the mean of a series of distance estimations are compared. One series consists of five consecutive images. The results can be seen in Fig. 9 and do not show a significant improvement or stabilization of the distance estimations. The relative errors show an offset by roughly the same positive amount, which is a reasonable behavior since the predictions from previous time steps, where the instances were still further away, increase the final estimated distance. Note that this approach requires a tracking of detected objects across multiple images.

To summarize, the high positioning inaccuracy of the ego-vehicle on the road described by the PSD results in a highly inaccurate distance estimation of light artifacts. However, the core idea itself is promising, as it models the road environment ahead in its actual shape. Yet, the current positioning inaccuracies make the data unusable for this task. This limitation could be tackled by using high-definition maps and visual odometry to improve localization in the future. The alternative approach of modeling the world ahead as a simple plane and finding the intersection with the camera ray, however, shows satisfying results. Another advantage is that this approach does not require any sensor data except for the camera input and its calibration and also comes at a very low computational cost since calculating the intersection of a line with a plane requires only a few floating-point operations. Still, when an object is not located directly on the ground (e. g., light reflections on guardrails), the method becomes inaccurate, too. For future improvements, approaches should be analyzed that try to estimate the surface of the environment ahead in order to account for curvatures of the road surface and thus return a better approximation than the simple plane (e. g., Alcantarilla et al., 2011). However, for the system presented in this paper, the accuracy of this method is considered to be sufficient, and, therefore, the GP approach is used as the distance estimation module in our system.

The goal of this experiment is to evaluate the time benefit of the proposed system in terms of a provident detection of oncoming vehicles with respect to human performance and an in-production computer vision system for vehicle detection at night. For this purpose, evaluations are performed on the test and validation dataset (to increase the database) of the PVDN dataset (see Sect. 6.1). The sequences from the dataset are processed by the implemented ROS network, and the detection times are computed on the image frame level of the dataset. This means that the number of frames is counted between the first indirect sight annotation is available (object detectable by human annotator) and the respective system recognizing the object in the image. By knowing that the frame rate of the camera is 18 Hz, the counted number of frames can be converted into seconds. Note that these computed times are without the system-specific processing time (see Sect. 6.5) and that the images in the dataset include a tag that specifies when the in-production system detected the oncoming vehicle. Additionally, note that even if the dataset contains images with multiple oncoming vehicles, only one can become visible first because the images are recorded on a two-lane road.¹⁵ Therefore, the number of frames is counted with respect to this first vehicle by using the available keypoint semantics (keypoints in the dataset are associated with the vehicles). In this context, the frame number when the single frame or tracking-based detection recognizes an oncoming vehicle is determined by the first bounding box that includes a keypoint associated with the first vehicle.

Figure 10 shows the results in the form of a box plot on the respective 39 sequences of the PVDN dataset. In 18 sequences, the in-production computer vision system did not detect a vehicle and thus has fewer measurement samples. It can be seen that the proposed system based on the tracker detects oncoming vehicles, on average, 1.6 s faster than the in-production computer vision system and is as fast as a human on average. The first detection based on a single frame is, on average, 2.1 s before the detection of the in-production computer vision system and 0.5 s before the detection of a human. The delay between the single frame detection and the tracker is caused by the plausibility phase of the tracker: an object has to be detected for at least five frames before it is sent as output. Five frames correspond to approximately 277 ms, and this is the minimal delay inherently caused by the plausibility checker (compare with Fig. 2). Therefore, it is not surprising that the time difference between the single frame detection and the tracker is, on average, 500 ms. However, overall, the results clearly show the considerable time benefit that can be achieved by such a sensing system.

To demonstrate the usefulness of the provident detection information for ADAS functionalities, we integrated the proposed detection system into the test car and used the provident detection information to control the adaptive headlights. By doing so, a provident glare-free high beam functionality is realized. The results of this experiment provide useful information about the applicability in real use cases:The glare-free high beam functionality is suitable to visualize the detection results, as the headlights can be considered as projectors that visualize the detected objects by turning off the respective pixels. Therefore, any serious inaccuracy in the system becomes immediately visible, and thus the integration serves as a proof of concept whether the object localization uncertainties are in such a range that they still provide useful information for later systems. As already said, for this experiment, the proposed detection pipeline is integrated into the test car, and test drives are performed on public rural roads at night. To ensure that other drivers are not put at risk, the light artifact detection pipeline is integrated in such a way that the detection results of our pipeline are muted as soon as the in-production system detects a vehicle. Thus, the integration of the proposed pipeline just bridges the time difference between the provident detection and the in-production system. To have a unique light artifact detection output, only the detected object (after tracking) with the highest intensity value is sent to the glare-free high beam module. Because the system is tested on two-lane roads, this ensures that the detected light artifact with the highest intensity always converges to the vehicle’s headlamps since, after direct sight to the vehicle, the intensity maximum is in a bounding box of the headlamps. This concept circumvented the need

Before the system is tested on public roads, the real-time capabilities of the pipeline are analyzed by measuring the computation times on the test car’s hardware (see Sect. 6.1). In this context, the run-times are determined by the elapsed time (ROS time) from receiving the node input to publishing the output. For instance, the run-time of the proposal classifier is the time from receiving the input in the form of the bounding boxes and the image until the classification of all bounding boxes is determined and published. Since the camera captures images with 18 Hz, the requirement is that the entire pipeline has an execution time faster than 18 Hz.

Figure 11 presents the run-time analysis in the form of a box plot. The measurements were performed on the 7 030 images (test and validation dataset) of the PVDN dataset. The average run-time of the complete pipeline (entire ROS network) for one image is on average 0.044 s so that the real-time requirement is fulfilled. However, it must be noted that the run-time of the pipeline is not constant. For example, the run-time is strongly affected by the number of bounding boxes created by the proposal generator. Moreover, with an increasing number of components (after the dynamic thresholding step) during the proposal generation, the run-time of the bounding box creation (inside the proposal generator) increases as well. Overall, the real-time requirement is fulfilled for 90 % of the analyzed images (see Fig. 11), and, therefore, the system can be deployed in the test car.¹⁶ It must be noted that the run-times of the proposal classifier and the proposal generator are different from the run-times reported in Fig. 6.2. This is because the evaluation here is conducted on the test and validation dataset and that the ROS framework causes a non-neglectable overhead due to data transformations to publish and process messages.

Figure 12 shows an example scene of the test drives on rural roads at night. This scene illustrates very well the accuracy and time benefit of the proposed system in a real environment in terms of a provident vehicle detection compared to the in-production computer vision system. In Fig. 12a, the first light artifact of the oncoming vehicle can be seen. After 0.5 s, the first detection is made by the proposed system based on a single frame, as shown in Fig. 12b. Then, after 2.6 s, the tracker has validated the object and output’s it correctly to the glare-free high beam module, see Fig. 12c. Based on the result of the tracker, the end of the road is dimmed proactively (a black gap can be seen in the white box) to avoid blinding the oncoming driver. The in-production system detects the oncoming vehicle after 3.8 s when it is fully visible and after a significant latency, see Fig. 12d. Therefore, the in-production system would have caused a short glare for the oncoming driver. In this scene, there is a total time benefit of 1.2 s of the proposed system.

This experiment provides a useful visualization interface of the detection results in the real world. It also shows that despite the localization uncertainties of the proposed detector and distance estimator, the information can be used to realize a provident glare-free high beam system. However, there are two points that need to be clarified: First, the reason for the size of the dimmed area is due to the low resolution of the headlights (see Sect. 6.1), and, for safety reasons, detected objects are always increased by a safety margin inside the glare-free high beam module. Moreover, the left tendency might be caused by the inaccuracies of the distance estimation. Second, the reason for the late detection of the tracker (in this case, 2.1 s after the first detection based on a single frame) is because the vehicle disappears behind the trees several times, which makes it difficult for the tracker to continuously track the vehicle over multiple frames.