Uncertainty Quantification for Object Detection: Output- and Gradient-Based Approaches

Various methods for uncertainty or confidence estimation exist. While confidence scores are directly produced by a DNN, oftentimes other uncertainty-related quantities, such as softmax margin or entropy, can be generated from the network output. In variational inference (e.g., MC dropout, deep ensembles, or batch norm uncertainty), for a single network input, several predictions are made. Their variance or standard deviation is then interpreted as a measure of prediction uncertainty. Oftentimes, a common strategy for different kinds of uncertainty is missing and comparison between methods is highly context-dependent. Moreover, different methods to estimate uncertainty may produce mutually redundant information which, for lack of comparability, cannot be established experimentally. We propose uncertainty aggregation methods as unifying strategy for different tasks.

Meta classification: In classification tasks, an input is categorized as one class out of a finite number of classes. Evaluation usually takes a binary value of either true or false. Meta classification can then be understood as the task of predicting the label true or false from uncertainty metrics. To this end, a lightweight classification model (e.g., logistic regression, a shallow neural network, or a gradient boosting classifier) is fitted to the training dataset, where uncertainty metrics can be computed and the actual, true label for each sample can be established. This can be regarded as learning a binary decision rule based on uncertainty information. On an evaluation dataset, such a meta classification model can be evaluated in terms of classification metrics like the area under the receiver operating characteristic \( AuROC \) or the area under precision-recall curve \( AuPR \) [DG06]. Different sources of uncertainty information, thus, serve as co-variables for meta classification models. Their performance can be regarded as a unified evaluation of confidence information. Network-intrinsic confidence scores naturally fit into this framework and can be either directly evaluated as any meta classification model or also serve as a single co-variable for such a model. In particular, different sources can be combined by fitting a meta classification model on the combined set of uncertainty metrics. This allows for the investigation of mutual redundancy between uncertainty information. Meta classification can be applied to any setting where a prediction can be marked as true or false and has served as a baseline for image classification [HG17], but also the tasks of semantic segmentation (“MetaSeg”) [RCH+20] and object detection (“MetaDetect”) [SKR20] can be subject to meta classification. In semantic segmentation, segments can be classified as true or false based on their intersection-over-union (\( IoU \)) with the ground truth being above a certain threshold. Similarly, bounding boxes are ordinarily declared as true or false predictions based on their maximum \( IoU \) with any ground truth box (see Sect. 3.2).

Meta fusion: Obtaining well-performing meta classifiers allows for their incorporation in the form of decision rules (replacing the standard score threshold) into the original prediction pipeline. Doing so is especially useful in settings where the DNN prediction is based on positive (foreground/detection) and negative (background) predictions, such as semantic segmentation and object detection. Implementing meta classifiers into such a framework is called meta fusion, whereby the improvement of the detection of false predictions over the DNN confidence is supplemented by increased prediction sensitivity of the DNN (see Fig. 2). Greater sensitivity tends to reduce false negative (FN) predictions and increase true (TP) and false positive (FP) predictions. The meta classifier’s task is then to identify the additional FPs as such and filter them from the prediction which leads to a net increase in prediction performance of the combination DNN with the meta classifier over the standalone DNN baseline. Such an increase can be evaluated utilizing metrics usually employed to assess the performance of a DNN in the respective setting, e.g., mean \( IoU \) in semantic segmentation [CRH+19] or mean average precision (\( mAP \)) in object detection [RRSG21].

Calibration: Confidence estimates with high predictive power (as measured in meta classification) have useful applications, e.g., in meta fusion where detection accuracy can be improved by investing uncertainty information. In addition to this advantage, there is another aspect to the probabilistic estimation of whether the corresponding prediction is correct or not. Confidence calibration tries to gauge the accuracy of the frequentistic interpretation, for example, from 100 examples each with a confidence of about \(\tfrac{1}{4}\), statistically we expect about 25 of those examples to be true predictions.

This is often investigated by dividing the confidence range into bins of equal width and computing the accuracy of predictions conditioned to the respective bin (see Fig. 3). For calibrated confidences, the resulting distribution should then be close to the ideally calibrated diagonal. Meta classification is closely related to post-processing calibration methods in that a map is learned on a validation dataset which yields confidences with improved calibration. In particular, meta classification with the confidence score as the only co-variable is (up to the explicit optimization objective) isotonic regression as introduced in [ZE02].

Meta regression: In a regression task, a continuous variable (e.g., in \(\mathbb {R}\)) is estimated given an input. By the same logic as in meta classification, whenever the quality of the prediction of a DNN can be expressed as a real number, we may fit a simple regression model (e.g., linear regression, shallow neural network, or a gradient boosting regression model) on a training dataset where both uncertainty metrics and the actual prediction quality can be computed. This is called a meta regression model which maps uncertainty information to an (e.g., \(\mathbb {R}\)-valued) estimation of the prediction quality of the DNN (see Fig. 4). The relationship between actual and predicted quality can, again, be evaluated on a dedicated dataset in terms of the coefficient of determination \(R^2\), a well-established quality metric for regression models. Meta regression shares many features as an evaluation protocol as meta classification, perhaps with the exception that network confidence scores fit less directly into the meta regression logic. However, a meta regression model based on such a score still follows the same design. We note that fitting non-linear regression models (e.g., a neural network or a gradient boosting as opposed to a linear regression) on one variable tends to not significantly improve meta regression performance. The \( IoU \) or modifications thereof in semantic segmentation [RCH+20] or object detection [SKR20] are suitable quality measures to be used in meta regression.

We develop output- and gradient-based uncertainty metrics for the task of 2D bounding box (“object”) detection on camera images \(\mathbf {x} \in \mathbb {I}^{H \times W \times C}\), with \(\mathbb {I} = [0, 1]\). A DNN tailored to this task usually produces a fixed number \(N_{\mathrm {out}} \in \mathbb {N}\) of output boxeswhere \(\boldsymbol{\theta } \in \mathbb {R}^{N_{\mathrm {DNN}}}\) with \(N_{\mathrm {DNN}} \in \mathbb {N}\) denotes the number of weights in the respective DNN architecture and \(N_\mathrm {C} \in \mathbb {N}\) is the fixed number of classes (the set of classes is denoted by \(\mathcal {N}_\mathrm {C} = \{1, \ldots , N_\mathrm {C}\}\)) to be learned. Each output box \(\mathfrak {O}^j(\mathbf {x}; \boldsymbol{\theta }) = (\hat{\xi }^j, \hat{s}^j, \hat{p}^j) \in \mathbb {R}^4 \times \mathcal {S} \times \mathbb {I}^{N_\mathrm {C}}\) for \(j \in \mathcal {N}_{\mathrm {out}} = \{1, \ldots , N_{\mathrm {out}}\}\) is encoded by

The predicted class of \(\mathfrak {O}^j(\mathbf {x}; \boldsymbol{\theta })\) is determined to be \(\hat{\kappa }^j = \mathrm {arg\, max}_{k \in \mathcal {N}_\mathrm {C}} \, \hat{p}_k^j\). The \(N_{\mathrm {out}}\) boxes subsequently undergo filtering mechanisms resulting in \(|\mathcal {N}_{\mathbf {x}}| = \hat{N}_{\mathbf {x}}\) detected instances \(\mathcal {N}_{\mathbf {x}} \subseteq \mathcal {N}_{\mathrm {out}}\)Commonly, the much smaller number \(\hat{N}_{\mathbf {x}} \ll N_{\mathrm {out}}\) of predicted boxes is the result of two mechanisms. First, score thresholding, i.e., discarding any \(\mathfrak {O}^j(\mathbf {x}; \boldsymbol{\theta })\) with \(s^j < \varepsilon _s\) for some fixed threshold \(\varepsilon _s > 0\), is performed as an initial distinction between “foreground” and “background” boxes. Afterward, the non-maximum suppression (NMS) algorithm is used to reduce boxes with the same class and significant mutual overlap to only one box. This way, for boxes that are likely indicating the same object in \(\mathbf {x}\), there is only one representative in \(\hat{y}\). By “overlap”, we usually mean the intersection-over-union (\( IoU \)) between two bounding boxes \(\hat{\boldsymbol{\xi }}^j\) and \(\hat{\boldsymbol{\xi }}^k\) which is defined asi.e., the ratio of their area of intersection and their joint area. The \( IoU \) between two boxes is always in \([0, 1]\), where 0 means no overlap and 1 means the boxes have identical localization. One also rates the localization quality of a prediction in terms of the maximum \( IoU \) it has with any ground truth box on \(\mathbf {x}\) which has the same class.

The NMS algorithm is based on what we call “candidate boxes”. An output instance \(\mathfrak {O}^k(\mathbf {x}; \boldsymbol{\theta })\) is a candidate box for another output instance \(\mathfrak {O}^j(\mathbf {x}; \boldsymbol{\theta })\) if it

We then denote bythe set of candidate boxes for \(\mathfrak {O}^j(\mathbf {x}; \boldsymbol{\theta })\). In the NMS algorithm, all output boxes \(\mathfrak {O}(\mathbf {x}; \boldsymbol{\theta })\) are sorted according to their score in descending order. Iteratively, the box \(\mathfrak {O}^j(\mathbf {x}; \boldsymbol{\theta })\) with the highest score is selected as a prediction and \(\mathrm {cand}[\mathfrak {O}^j(\mathbf {x}; \boldsymbol{\theta })]\) is removed from the ranking of output boxes. This procedure is repeated until there are no boxes left. Note that NMS usually follows score thresholding so this stage may be reached quickly, depending on \(\varepsilon _s\).

DNN object detection frameworks are usually trained in a supervised manner on images with corresponding labels or ground truth. The latter usually consist of a number \(N_{\mathbf {x}}\) of ground truth instances \(\overline{\mathbf {y}} = (\overline{\mathbf {y}}^1, \ldots , \overline{\mathbf {y}}^{N_{\mathbf {x}}}) \in (\mathbb {R}^4 \times \mathcal {N}_\mathrm {C})^{N_{\mathbf {x}}}\) consisting of similar data as output boxes which we denote by the bar (\(\overline{\cdot }\)). In particular, each ground truth instance \(\overline{\mathbf {y}}^j\) has a specified localization \(\overline{\boldsymbol{\xi }}^j = (\overline{c}^j_{\min }, \overline{r}^j_{\min }, \overline{c}^j_{\max }, \overline{r}^j_{\max }) \in \mathbb {R}^4\) and category \(\kappa ^j \in \mathcal {N}_\mathrm {C}\). From the network output \(\mathfrak {O}(\mathbf {x}; \boldsymbol{\theta })\) on an image \(\mathbf {x}\) and the ground truth \(\overline{\mathbf {y}}\), a loss function \(J(\mathfrak {O}(\mathbf {x}; \boldsymbol{\theta }), \overline{\mathbf {y}})\) is computed and iteratively minimized over \(\boldsymbol{\theta }\) by stochastic gradient descent or some of its variants. In most object detection frameworks, \(J\) splits up additively intowith \(J_{\boldsymbol{\xi }}\) punishing localization mistakes, \(J_s\) punishing incorrect score assignments to boxes (incorrect boxes with high score and correct boxes with low score), and \(J_{\mathbf {p}}\) punishing an incorrect class probability distribution, respectively. In standard gradient descent optimization, the weights \(\boldsymbol{\theta }\) are then updated by the following rule:where \(\eta > 0\) is the either fixed or variable learning rate parameter. The learning gradient \(\mathbf {g}(\mathbf {x}, \boldsymbol{\theta }, \overline{\mathbf {y}}) := \nabla _{\boldsymbol{\theta }} J (\mathfrak {O}(\mathbf {x}; \boldsymbol{\theta }), \overline{\mathbf {y}})\) will play a central role for the gradient-based uncertainty metrics which will be introduced in Sect. 3.4.

In this section, we construct uncertainty metrics for every \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\in \hat{\mathbf {y}}(\boldsymbol{x}, \boldsymbol{\theta })\). We do so in two stages, first by introducing the general metrics that can be obtained from the object detection pipeline. Second, we extend this by additional metrics that can be computed when using MC dropout. We consider a predicted bounding box \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\in \hat{\mathbf {y}}(\boldsymbol{x}, \boldsymbol{\theta })\) and its corresponding filtered candidate boxes \(\mathfrak {O}^k(\mathbf {x}; \boldsymbol{\theta })\in \mathrm {cand}[\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })]\) that were discarded by the NMS.

The number of corresponding candidate boxes \(\mathfrak {O}^k(\mathbf {x}; \boldsymbol{\theta })\in \mathrm {cand}[\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })]\) filtered by the NMS intuitively gives rise to the likelihood of observing a true positive. The more candidate boxes \(\mathfrak {O}^k(\mathbf {x}; \boldsymbol{\theta })\) belong to \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\), the more likely it seems that \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\) is a true positive. We denote by \(N^{(j)}\) the number of candidate boxes \(\mathfrak {O}^k(\mathbf {x}; \boldsymbol{\theta })\) belonging to \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\), but suppressed by NMS. We increment this number by 1 and also count in \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\).

For a given image \(\boldsymbol{x}\), we have the set of predicted bounding boxes \(\hat{\mathbf {y}}(\boldsymbol{x}, \boldsymbol{\theta })\) and the ground truth \(\overline{\mathbf {y}}\). As we want to calculate values that represent the quality of the prediction of the neural network, we first have to define uncertainty metrics for the predicted bounding boxes in \(\hat{\mathbf {y}}(\boldsymbol{x}, \boldsymbol{\theta })\). For each \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\in \hat{\mathbf {y}}(\boldsymbol{x}, \boldsymbol{\theta })\), we define the following quantities:

Altogether, this results in \(47+N_\mathrm {C}\) uncertainty metrics which can be aggregated further in a meta classifier or a meta regression model.

We now elaborate on how to calculate uncertainty metrics for every \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\in \hat{\mathbf {y}}(\boldsymbol{x}, \boldsymbol{\theta })\) when using MC dropout. To this end, we consider the bounding box \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\in \hat{\mathbf {y}}(\boldsymbol{x}, \boldsymbol{\theta })\) that was predicted without dropout and then we observe under dropout K times the output of the same anchor box that produced \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\) and denote them by \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })_1,...,\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })_K\). For these \(K+1\) boxes \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta }),\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })_1,...,\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })_K\), we calculate the standard deviation for the localization variables, the objectness score, and class probabilities. This is done for every \(\mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\in \hat{\mathbf {y}}(\boldsymbol{x}, \boldsymbol{\theta })\) and results in \(4 + 1 + N_\mathrm {C}\) additional dropout uncertainty metrics which we denote by \(\mathrm {std}_\mathrm {MC}(\cdot )\) for each of the respective \(4 + 1 + N_\mathrm {C}\) box features. Executing this procedure for all available test images, we end up with a structured dataset. Each row represents exactly one predicted bounding box and the columns are given by the registered metrics. After defining a training/test splitting of this dataset, we learn meta classification (\( IoU >0.5\) vs. \( IoU \le 0.5\)) and meta regression (quantitative \( IoU \) prediction) from the training part of this data. All the presented metrics, except for the true \( IoU \), can be computed without the knowledge of the ground truth. We now want to explore to which extent they are suited for the tasks of meta classification and meta regression.

In the setting provided in Sect. 3.2, a DNN learns from a data point \((\mathbf {x}, \overline{\mathbf {y}}) \in \mathbb {I}^{H \times W \times C} \times (\mathbb {R}^4 \times \mathcal {N}_{\mathrm {C}})^{N_{\mathbf {x}}}\) by computing the gradient \(\mathbf {g}(\mathbf {x}, \boldsymbol{\theta }, \overline{\mathbf {y}})\). The latter is subsequently used to iteratively adjust the current weights \(\boldsymbol{\theta }\), for example, by the formula given in (6). As such, the quantity \(\mathbf {g}(\mathbf {x}, \boldsymbol{\theta }, \overline{\mathbf {y}})\) can be interpreted as learning stress on \(\boldsymbol{\theta }\) induced by the training data \((\mathbf {x}, \overline{\mathbf {y}})\). We explain next how a related quantity gives rise to confidence information.

Some intuition about gradient uncertainty: Generally, the loss function \(J\) measures the “closeness” of output boxes to the ground truth \(\overline{\mathbf {y}}\) and \(\mathbf {g}(\mathbf {x}, \boldsymbol{\theta }, \overline{\mathbf {y}})\) expresses the induced change in \(\boldsymbol{\theta }\) for any deviation from the ground truth. If we assume that \(\hat{\mathbf {y}}(\mathbf {x}; \boldsymbol{\theta })\) is close to \(\overline{\mathbf {y}}\), only little change in \(\boldsymbol{\theta }\) will be required in an update step. We, therefore, expect \(\mathbf {g}(\mathbf {x}, \boldsymbol{\theta }, \overline{\mathbf {y}})\) to be of comparably small magnitude. A DNN which is already “well-trained” in the ordinary sense is expected to express high confidence when its prediction is correct for all practical purposes. Conversely, if the network output deviates from the ground truth significantly, learning on \((\mathbf {x}, \overline{\mathbf {y}})\) induces a larger adjustment in \(\boldsymbol{\theta }\), leading to a gradient of larger magnitude. In this situation, the confidence in this prediction should be small.

The gradient \(\mathbf {g}(\mathbf {x}, \boldsymbol{\theta }, \overline{\mathbf {y}})\) is not realizable as a measure of uncertainty or confidence, since it uses the ground truth \(\overline{\mathbf {y}}\), which is not accessible during inference time. An approach to circumvent this shortcoming was presented in [ORG18], in that the ground truth \(\overline{\mathbf {y}}\) was replaced by the prediction of the DNN \(\hat{\mathbf {y}}\) on \(\mathbf {x}\). We format the prediction so that it has the same format as the ground truth. Particularly, in the initial application to image classification, the softmax output of the DNN was collapsed by an \(\mathrm {arg\,max}\) to the predicted class before insertion into \(J\). With respect to bounding box localization, there is no adjustment required. Additionally, when computingwe disregard the dependency of \(\hat{\mathbf {y}}\) on \(\boldsymbol{\theta }\), thereby sticking to the design motivation from above. The quantity in (8) represents the weight adjustment induced in the DNN when learning that its prediction on \(\mathbf {x}\) is correct. Uncertainty or confidence information is distilled from \(\mathbf {g}(\mathbf {x}, \boldsymbol{\theta }, \hat{\mathbf {y}})\) by taking it to a scalar. To this end, it is natural to employ norms, e.g., \(|\!| \cdot |\!|_1\) or \(|\!| \cdot |\!|_2\), but we also use other maps such as minimal and maximal entry as well as the arithmetic mean and standard deviation over the gradient’s entries (see (9)). Gradient uncertainty metrics allow for some additional flexibility. Regarding (8), we see that if \(J\) splits into additive terms (such as in the object detection setting, see (5)), gradient metrics allow the extraction of gradient uncertainty from each individual contribution. Additionally, we can restrict the variables for which we compute partial derivatives. One might, for example, be interested in computing the gradient \(\mathbf {g}(\mathbf {x}, \boldsymbol{\theta }_{\ell }, \hat{\mathbf {y}})\) for the weights \(\boldsymbol{\theta }_\ell \) from only one network layer \(\ell \). In principle, this also allows for computing uncertainty metrics for individual convolutional filters or even individual weights, thus, also offering a way of localizing uncertainty within the DNN architecture.Application to instance-based settings: The approach outlined above requires some adaptation in order to produce meaningful results for settings in which the network output \(\mathfrak {O}(\mathbf {x}; \boldsymbol{\theta })\) and the ground truth \(\overline{\mathbf {y}}\) consist of several distinct instances, e.g., as in object detection. In such a situation, each component \(\mathfrak {O}^j(\mathbf {x}; \boldsymbol{\theta })\) needs to be assigned individual confidence metrics. If we want to estimate the confidence for \(\hat{\mathbf {y}}^j := \mathfrak {B}^j(\mathbf {x}; \boldsymbol{\theta })\), we may regard \(\hat{\mathbf {y}}^j\) as the ground truth replacement entering the second slot of \(J\). Then, the first argument of \(J\) needs to be adjusted according to \(\hat{\mathbf {y}}^j\). This is necessary because the loss \(J(\mathfrak {O}(\mathbf {x}; \boldsymbol{\theta }), \hat{\mathbf {y}})\) expresses the mistakes made by all the output boxes, i.e., the prediction of instances which actually appear in the ground truth \(\overline{\mathbf {y}}\) but which have nothing to do with \(\hat{\mathbf {y}}^j\) would be punished. The corresponding gradient would, therefore, misleadingly adjust \(\boldsymbol{\theta }\) toward forgetting to see such ground truth instances. Thus, it is a natural idea to identify \(\mathrm {cand}[\hat{\mathbf {y}}^j]\) in the network output and only enter those boxes into \(J\) which likely indicate one and the same instance on \(\mathbf {x}\). We define the corresponding gradientwhich serves as the basis of gradient uncertainty in instance-based settings. The flexibility mentioned in the previous paragraph carries over to this definition and will be exploited in our experiments.

We investigate three different object detection datasets in the Pascal VOC [EVGW+15], the MS COCO [LMB+14], and the KITTI vision benchmark [GLSU15]. Pascal VOC \(\mathcal {D}^\mathrm {VOC12}\) and MS COCO \(\mathcal {D}^\mathrm {COCO17}\) are vision datasets containing images of a wide range of mostly everyday scenarios with 20 and 80 annotated classes, respectively. They both possess training and dedicated evaluation splits for testing and constitute established benchmarks in object detection. In order to test our methods in driving scenarios, we also investigate the KITTI dataset \(\mathcal {D}^\mathrm {KITTI}\) which shows different German urban environments and comes with annotated training images. From the annotated training set, we take 2,000 randomly chosen images for evaluation. An overview of the sizes of the respective training and evaluation datasets can be found in Table 1.

The experiments we present here have been carried out based on a YOLOv3 [RF18] re-implementation in PyTorch. We have trained our model from scratch on each of the three training splits under dropout with a probability 0.5 between the last and the second-to-last convolution layer in each of the regression heads. As meta classification and meta regression models, we employ the gradient boosting models in [CG16] with standard settings.

For the evaluation of our meta classification models, we use cross-validated area-under-curve metrics (\( AuROC \) and \( AuPR \) [DG06]) instead of accuracy, since the former measure the classification quality independently of any threshold, whereas accuracy only reflects the decisions made for the classification probability \(0.5\). For cross-validation, we use 10 individual image-wise splits of uncertainty metrics for the training of meta classifiers and the respective complementary split for evaluation. This particular procedure is employed whenever cross-validation is used in the following experiments. Especially in the VOC and KITTI dataset, \( AuROC \) values are mostly over \(0.9\), so we regard \( AuPR \) as a second classification metric. Object detection quality in meta fusion experiments will on the one hand be evaluated in terms of cross-validated mean average precision (\( mAP \)) [EVGW+15]. However, as the authors of [RF18] recognized \( mAP \) as computed in the VOC challenge is insensitive to FPs, which is why, on the other hand, we compute the cross-validated mean \(F_1\) score (\( mF _1\)) as well. For the precision-recall curves of each class, we compute individual \(F_1\) scores which are sensitive to FPs and we average them over all classes as a complementary metric to \( mAP \). Confidence calibration is usually evaluated in terms of expected or maximum calibration error (\( MCE \)) [NCH15], computed over bins of width 0.1. As the expected calibration error is sensitive to the box count per bin, we instead regard in addition to the \( MCE \) the average calibration error (\( ACE \)) [NZV18]. Meta regression quality is evaluated by the usual coefficient of determination \(R^2\).

Our object detection models receive a dropout layer between the second-to-last and the last layers in each of the three YOLOv3 detection heads. Dropout is active during training with a rate of 0.5 and also during MC inference, where we take standard deviations over 30 dropout samples for each of the \(4 + 1 + N_\mathrm {C}\) instance features of all output boxes. The MetaDetect metrics introduced in Sect. 3.3 are computed from a score threshold of \(\varepsilon _s = 0.0\), as it has been found in [SKR20] that lower thresholds lead to better performance in meta classification and meta regression.

Gradient uncertainty metrics were computed for the weights in the second-to-last (“\(T - 1\)”) and the last (“T”) convolutional layers in the YOLOv3 architecture at the same candidate score threshold as for the MetaDetect metrics. As there are three detection heads, corresponding to a \(76 \times 76\) (“S”), a \(38 \times 38\) (“M”), and a \(19 \times 19\) (“L”) cell grid, we also compute gradient uncertainty metrics individually for each detection head. We use this distinction in our notation and, for example, indicate the set of parameters from the last layer (\(T\)) of the detection head producing the \(76 \times 76\) cell grid (\(\mathrm {S}\)) by \(\boldsymbol{\theta }(T, \mathrm {S})\). Moreover, as indicated in Sect. 3.4, we also exploit the split of the loss function in (5). Each of the computed \(2 \times 3 \times 3 = 18\) gradients per box results in the 6 uncertainty metrics presented in (9) giving a total of 108 gradient uncertainty metrics per bounding box. Due to the resulting computational expense, we only compute gradient metrics for output boxes with score values \(\hat{s} \ge 10^{-4}\) and regard only those boxes for all of the following experiments.

Starting from pre-trained models, we compute the DNN output, as well as MC dropout, MetaDetect, and gradient uncertainty metrics on each evaluation dataset (see Table 1). For each output box, we also compute the maximum \( IoU \) with the respective ground truth such that the underlying relation can be leveraged. Before fitting any model, we perform NMS on the output boxes which constitutes the relevant case for our meta fusion investigation. In the latter, we are primarily interested in finding true boxes which are usually discarded due to their low assigned score. We compare the performance of different sets of uncertainty metrics in different disciplines. In the following, “Score” denotes the network’s confidence score, “MC” stands for MC dropout, and “MD” the MetaDetect uncertainty metrics as discussed in Sect. 3.3. Moreover, “G” denotes the full set of gradient uncertainty metrics from Sect. 3.4 and the labels “MD+MC”, “G+MC”, “MD+G”, and “MD+G+MC” stand for the respective unions of MC dropout, MetaDetect, and gradient uncertainty metrics.

Meta classification: We create 10 splits of the dataset of uncertainty metrics and computed \( IoU \) by randomly dividing it 10 times in half. We assign the TP label to those examples which have \( IoU > 0.5\) and the FP label otherwise. On each split, we fit a gradient boosting classifier to the respective set of uncertainty metrics (see Fig. 5) on one half of the data. The resulting model is used to predict a classification probability on the uncertainty metrics of the second half of the data and vice-versa. This results in meta classification predictions on all data points. We evaluate these in terms of the previously introduced area-under-curve metrics under usage of the TP/FP labels generated from the ground truth, where we obtain averages and standard deviations from the 10 splits shown in Fig. 5. Performance measured in either of the two area-under-curve metrics suggests that the models situated in the top-right separate TP from FP best. In all datasets, the meta classifiers MD, G, and combinations thereof outperform the network confidence by a large margin (upwards of 6 \( AuROC \) percent points (ppts) on \(\mathcal {D}_\mathrm {test}^\mathrm {VOC12}\), 2.5 on \(\mathcal {D}_\mathrm {val}^\mathrm {COCO17}\), and 1.5 on \(\mathcal {D}_\mathrm {eval}^\mathrm {KITTI}\)), with MC dropout being among the weakest meta classifiers. Note that for both \(\mathcal {D}_\mathrm {test}^\mathrm {VOC12}\) (top) and \(\mathcal {D}_\mathrm {val}^\mathrm {COCO17}\) (center), the standard deviation bars are occluded by the markers due to the size of the range. On \(\mathcal {D}_\mathrm {eval}^\mathrm {KITTI}\), we can see some of the error bars and the models forming a strict hierarchy across the two metrics with overlapping MD+MC and G+MC. Overall, the three sources of uncertainty MC, MD, and G show a significant degree of non-redundancy across the three datasets with mutual boosts of up to 7 \( AuPR \) ppts on \(\mathcal {D}_\mathrm {test}^\mathrm {VOC12}\), 4.5 on \(\mathcal {D}_\mathrm {val}^\mathrm {COCO17}\), and 1.1 on \(\mathcal {D}_\mathrm {eval}^\mathrm {KITTI}\).

Meta fusion: We use the 10 datasets of uncertainty metrics and resulting confidences as in meta classification in combination with the bounding box information as new post-NMS output of the network. Thresholding on the respective confidence on decision thresholds \(\varepsilon _\mathrm {dec} \in (0, 1)\) with a fixed step width of 0.01 reproduces the baseline object detection pipeline in the case of the score. The network prediction with assigned confidence can then be evaluated in terms of the aforementioned class-averaged performance measures for which we obtain averages and standard deviations from the 10 splits, except for the deterministic score baseline. In our comparison, we focus on the score baseline, MC dropout baseline, output- and gradient-based uncertainty metrics (MD and G), and the full model based on all available uncertainty metrics (G+MD+MC). We show the resulting curves of \( mAP \) over \( mF _1\) in Fig. 6 on the right, where we see that the maximum \( mF _1\) value is attained by the confidence score. In addition, we observe a sharp bend in the score curve resulting from a large count of samples with a score between 0 and 0.01, where the interpolation to the threshold \(\varepsilon _s = 0\) reaches to 0.912 \( mAP \) at a low 0.61 \( mF _1\) (outside the plotting range). In terms of \( mAP \), however, the score is surpassed by the meta fusion models G and in particular the meta classification models involving MD (maximum \( mAP \) close to 0.92 at around 0.81 \( mF _1\)). The raw MetaDetect confidences seem to slightly improve over G+MD+MC in terms of \( mAP \) by about 0.3 ppts., which may be due to overfitting of the gradient boosting model, but could also result from the altered confidence ranking which is initially performed when computing mean average precision. Meta fusion based on MC dropout uncertainty shows the worst performance of all four methods considered in this test.

Calibration: From the generated meta classification probabilities and the score, calibration errors can be computed indicating their calibration as binary classifiers and compared; see Fig. 7 with standard deviations indicated by error bars. In addition, we show the Platt-scaled confidence score as a reference for a post-processing calibration method. Naturally, \( ACE \) is smaller than \( MCE \). The meta classification models show consistently far-smaller calibration errors (maximum \( MCE \) of 0.11 on \(\mathcal {D}^\mathrm {KITTI}_\mathrm {eval}\)) than the confidence score (minimum \( MCE \) of 0.1 on \(\mathcal {D}^\mathrm {COCO17}_\mathrm {val}\)) with comparatively small miscalibration on the COCO validation dataset (center). We also observe that meta classifiers are also better calibrated than the Platt-scaled confidence score with the smallest margin on \(\mathcal {D}^\mathrm {KITTI}_\mathrm {eval}\). Note from the \( MCE \) that meta classification models show generally good calibration with a maximum \( ACE \) around 0.05 as indicated in Sect. 3 and by the example of Fig. 3.

Meta regression: By an analogous approach to meta classification, we can fit gradient boosting regression models in a cross-validated fashion to sets of uncertainty metrics and the true maximum \( IoU \) with the ground truth. The resulting predicted \( IoU \) values can be evaluated in terms of \(R^2\) which is illustrated in Fig. 6 on the left. Note once again that error bars for the standard deviation are covered by the markers. We observe an overall similar trend of the metrics to the meta classification performance in that combined models tend to perform better (boosts of up to 6 \(R^2\) ppts.) indicating mutual non-redundancy among the different sources of uncertainty. The models based on MD and G individually show significant improvements over the confidence score (over 5 \(R^2\) ppts. on all datasets) and a slight improvement over MC (up to 3 \(R^2\) ppts.) for all investigated datasets.

Parameter importance: We investigate which parameters are most important for meta classification in order to find the best performance with as few metrics as possible. We pursue a greedy approach and evaluate single-metric models at first in terms of \( AuROC \) and \( AuPR \) in the cross-validated manner used before. We then fix the best-performing single metric and investigate two-metric combinations and gather the resulting metrics ranking with their area-under-curve score in Table 2. In addition to the nested models, we also show the model using all metrics which is also depicted in Fig. 5. Note that the numbers represent cross-validated averages, where we choose not to show standard deviations as we are mainly interested in seeing which metrics combine to yield well-separating meta classifiers. On all datasets, we find that a total of 9 metrics suffices to come close to or reach the performance of the best model in question. The network confidence appears as the most informative single metric 5 out of 6 times. Another metric that leads to early good improvements is the related MC dropout standard deviation of the confidence score. Other noteworthy metrics are the left border coordinate \(\hat{c}_{\min }\) and gradient metrics for the class probability contribution \(J_p\). Overall, gradient metrics contribute a large amount of information to \(\hat{s}\) and \(\mathrm {std}_\mathrm {MC}(\hat{s})\) which is indicative to meta classification performance.