Uncertainty-aware Prediction Validator in Deep Learning Models for Cyber-physical System Data

2.1 Uncertainty Quantification in DL

Uncertainty quantification in DL models is an ongoing topic of intensive research. The main method for quantifying uncertainty in DL models is the use of the softmax variance and expected entropy over multiple models. The work by Gal [14] demonstrated that a neural network model with prediction-time-activated dropout is equal to a specific variational inference on a Bayesian neural network model. The prediction model uncertainty is approximated by averaging probabilistic feed-forward MC dropout sampling during the prediction time. This approach is highly effective in terms of practical implementation for large models and has proved useful in understanding the dynamics of the networks under different testing conditions [53].

In addition to MC dropout-based Bayesian Neural Network [36], an alternative uncertainty quantification method in DL is Deep Ensembles, a non-Bayesian method for uncertainty quantification [31]. In Deep Ensemble, several models are trained in parallel, and each of them is trained using random noise (with adversarial instances) in its subset part of the training dataset. The final output of the training phase is a set of independent classifiers, each of which has its own unique set of weights. The ensemble can predict the same input instance with different predictions using its individual models. Considering that Deep Ensemble requires several models, it is not suitable for single-model uncertainty quantification. Therefore, in this work, we opted for the prediction-time-activated dropout-based neural networks approach for single-model uncertainty quantification.

Methods in adversarial ML aim to create perturbed instances to evade (or fool) a DL model. Recently, Ma et al. [36] showed the correlation between uncertainty and model prediction performance. In their approach, to increase the DL models’ robustness, they applied adversarial ML attacks (e.g., Fast Gradient Sign Method, Basic Iterative Method, DeepFool, Jacobian-Based Saliency Map Attack, and Carlini-Wagner) to create highly uncertain and misclassified instances. Newly created adversarial instances are used as a new training dataset in the re-training phase to increase the robustness of the DL model. The iterative training with adversarial instances is like traditional adversarial training [20]. They used MNIST, Fashion MNIST, and CIFAR-10 image datasets in their experiments to evaluate the effectiveness of their approach. Though this method is similar to adversarial training, NIRVANA improves the prediction performance of DL models by using highly uncertain inputs instead of adversarial inputs.

Ghoshal and Tucker [18] studied how Bayesian Convolutional Neural Networks (CNNs) can quantify uncertainty in DL models to enhance the diagnostic performance of the human-machine alliance using a COVID-19 chest X-ray dataset. Another study reported in [45] uses uncertainty values and quality estimates to model predictive uncertainty estimation. Their main idea relies on the difference between the detection box and ground truth in images, and therefore is only applicable for object detection problems but not CPS data.

In our recent work [7], we proposed an uncertainty quantification metric for object detection models, named PURE. Experiments were performed on the KITTI, Stanford cars, Berkeley DeepDrive, and NEXET datasets, together with YoLo, SSD300, and SSD512 object detection models, to quantify their prediction uncertainty with PURE. PURE is an uncertainty quantification method in object detection models for autonomous driving; on the other hand, NIRVANA aims to assess the reliability of DL models’ prediction with another model and improve DL models’ predictions with generated uncertainty quantification results.

2.2 Uncertainty in Software Engineering

In the last decade, uncertainty has attracted significant attention in software engineering, due to the fact that uncertainty is gradually being recognized as an inevitable characteristic of complex systems such as CPSs, due to their increasingly unpredictable, open, and networked operating environments [16]. There exists a number of solutions (e.g., [2, 8, 60, 61]) in the literature for modeling or specifying known uncertainty on requirements, design models, and so forth. There also exist approaches in software engineering that focus on refining uncertainty measurements from historical data (e.g., [59]), reasoning with Bayesian inference (e.g., [52]), and testing software systems under uncertainty (e.g., [6, 35]). To compare with NIRVANA, these works are generic solutions and do not particularly focus on software system with DL models deployed.

Same as for conventional software systems, software systems with DL models employed also face quality and reliability issues. Not handling them properly has disastrous consequences. To this end, Zhang [62] proposed an approach to generate highly uncertain test inputs using adversarial ML attacks and improve prediction performance of DL models. More specifically, the approach first uses genetic algorithms to generate adversarial instances to effectively penetrate DL defense techniques, therefore uncovering hidden defects in the DL models. Second, it re-trains the DL models with adversarial inputs. Tuna et al. [11] showed that adversarial inputs generated by adversarial ML attacks can fool DL models with low prediction uncertainty values. These re-training approaches based on adversarial inputs are different from NIRVANA, as NIRVANA improves the prediction performance of DL models by re-training them with highly uncertain training inputs.

In autonomous driving, Michelmore et al. [39] applied variation ratios, entropy, and mutual information-based uncertainty quantification methods to discover the relation between uncertainty and wrong prediction of steering angles of autonomous cars. They found that wrong angle decisions of an autonomous car have significantly higher uncertainty values than correct decisions. Based on this observation, they defined a threshold to determine crash risks. This is different from NIRVANA, as NIRVANA employs an SVM-based prediction model (instead of a threshold) for prediction validation.

When particularly considering CPSs with DL models employed, it is critical to verify accurate interactions between a DL model and the physical operating environment of the CPS with the DL model employed and confirm its safety and correct operation [32]. Along this line, Dreossi et al. [10] proposed a toolkit, named VERIFAI, which focuses on the simulation-based safety analysis of systems enabled with artificial intelligence and ML components. VERIFAI especially explores challenges with implementing formal methods to understand ML models and analyze system behavior under environmental uncertainty. On the other hand, NIRVANA is not a formal method tool. Instead, it aims to predict wrong DL models’ predictions based on uncertainty values.

2.3 ML and CPS Testing

There is an increasing interest in ML and CPS testing, and new techniques and research have been conducted toward ML and CPS testing. A survey conducted by Sherin et al. [47] classifies testing methods in ML models. They showed a dramatic increase in the number of publications between 2015 and 2018. Most ML testing methods are black box (74%), and more than half (57%) of the proposed ML testing methods are about testing neural networks models.

Asadollah et al. [1] classified CPS testing into six categories: hardware testing, structural and computation testing, extra-functional properties testing, network testing, integration testing, and system testing. DL models are now becoming one of the main components of CPSs. Thus, the CPS testing should also include DL model testing. In another work [64], the authors stated that a CPS is a complex system with the following features: data driven, software defined, and self-government. They also said that uncertainty modeling for CPS testing is one of the challenges for future complex CPS testing methods.

Berend et al. [4] showed that DL models do not provide statistical guarantee and have limited capability in handling data that falls outside of the training dataset’s distribution (i.e., out-of-distribution (OOD) data). They conducted a large-scale study on the impact of data distribution on state-of-the-art DL testing techniques. Their results show that existing OOD detection techniques could distinguish OOD data from newly generated test cases and a distribution-aware dataset tends to be more effective in robustness enhancement. Our prediction validator model predicts wrong predictions of DL models employed for CPS data using uncertainty measurements, which is different from the aim of their study: enhancing model robustness with OOD datasets.

Zhang et al. [63] investigated the capability of different uncertainty metrics (including prediction confidence score and variation ratio of original prediction) in differentiating benign examples (BEs) and adversarial examples (AEs) of deep learning models. They further investigated uncertainty patterns of BEs and AEs generated by existing adversarial ML techniques and observed that these BEs and AEs follow certain uncertainty patterns and some patterns were largely missed. Based on these observations, they proposed a technique to automatically generate diverse adversarial data and concluded that uncommon data generated with their method is hard to be defended by existing defense techniques. To compare with this work, we identify wrong predictions with uncertainty measurements instead of generating adversarial data. In our recent study [12], we showed that uncertainty in predictions using inputs manipulated by adversarial ML attacks could be quite low and show similar patterns, just as in BEs.

Weiss and Tonella [56] proposed a fail-safe DL-based system with a supervisor that monitors the DL uncertainty for any given input at runtime such as ignoring predictions for high-uncertainty inputs and running a safe fallback process by stopping a self-driving car at the side of the street, for instance. The supervisor has the capability to discard predictions for highly uncertain inputs above a specific threshold. The threshold setting is critical since a high threshold will fail to reject many false predictions, while a low threshold will result in an excessive number of false warnings. The authors introduced a new metric for determining the appropriate threshold value to address this concern.

3.1 Challenges and Context

Developing CPSs is challenging as compared to classical software due to the unique characteristics of CPSs, such as the interdisciplinary nature (e.g., involving software, hardware, physics), close interactions with their operating environment including humans, and the use of communication mediums for communicating among their components [16]. In addition, CPSs face uncertainties both during their development and throughout their operation, e.g., due to intrinsic uncertainty of the DL models employed, inherently unpredictable environment, unpredictable human behaviors, and unreliable network communications among CPS components [35, 60]. Such uncertainties make CPSs’ development and operation even more challenging. Thus, to better understand uncertainties of CPS behaviors with the aim to enhance the dependability of the next generations of CPSs, develop new test cases, and so forth, data produced during their operation can be used for various analyses with DL models. To this end, the context of this work focuses on CPS data produced, e.g., during their operation.

The overall context of this article is shown in Figure 1. Since we focus on uncertainty in DL models employed on CPS labelled data, we expect that there is an Execution and Labelling System that interacts with the CPS of interest, obtains relevant CPS data, and labels them with a DL model. Examples of such labels include whether a security attack was detected by the CPS, and whether or not an execution of the CPS system failed. We would also like to acknowledge that such labeled data are commonly available and we use them in this article for the evaluation purpose. The Labeled Data is then the input for Uncertainty Quantification, which consequently produces another set of data collected for uncertainty metrics (i.e., Uncertainty Measurements). These data are then the input to another ML model, Prediction Validator, which makes predictions about wrong classifications of DL models employed for CPS data due to uncertainty. In our context, the motivation of having Prediction Validator is to predict the correctness of the DL models employed for CPS data’s output by using the model’s uncertainty values. In a typical DL model training, the training dataset \( \mathcal {D} \) inevitably cannot include all possible input instances. Thus, any model trained with CPS data has no chance to reach the goal of discovering the best parameters (i.e., weights in neurons) that can successfully map all possible inputs to CPS outputs. This makes an inequality between the ground truth y and the prediction of a model for CPS data \( \hat{y} \), called model uncertainty. One straightforward strategy to reduce the model uncertainty is to obtain more data, which is often not possible in practice. For example, Kendall and Gal [29] stated that given enough data, model uncertainty could be reduced to a negligible level. This is a hard task that cannot be achieved for any realistic applications. In this paper, our main aim is about using uncertainty quantification metrics to predict wrong predictions of DL models employed on CPS labeled data. We end up with a prediction validator model that indicates how much a given DL model predicts correctly and how much the actual value (i.e., ground truth) is deviated from the predictions.

Fig. 1.

View Figure

3.2 Running Example

To explain various concepts in our method, we will use the running example of the Robot dataset [13]. The dataset is collected from a robot that moves itself in a room with the help of 24 different sensors attached circularly around its waist. Based on the current set of sensor values, the robot decides its next move, which is classified into four labels: Move-Forward, Slight-Right-Turn, Sharp-Right-Turn, and Slight-Left-Turn. Table 1 shows the excerpt of the labeled dataset. The columns US1 to US24 represent the 24 sensors. The sensor values are continuous. The Label column shows the label associated with each set of sensor values, i.e., each row in the table. To validate our method, we will build a DL model to predict the direction of motion based on sensor readings by using the dataset. We split the dataset as training dataset to build the DL model and test dataset to evaluate the DL model. We aim to quantify the uncertainty in the decisions of the DL model about making the next move.

5.1 Prediction Validator

To predict wrong predictions of DL models employed for CPS data, we opted for an SVM-based detection model (i.e., prediction validator). We chose SVM since the scientific literature [28] showed that SVM’s prediction performance is generally better than multilayer perceptron (MLP). In the future, we will try with MLP-based classification models. The training dataset for the prediction validator is prediction outputs of the DL model employed for CPS data and their uncertainty (Figure 2). Wrong labeling information includes false-positive (FP) and false-negative (FN) labeled instances in the training dataset. True-positive (TP) and true-negative (TN) predictions of the DL model employed for CPS data are labeled as 0 in the prediction validator training dataset; FP and FN predictions are labeled as 1 in the dataset, as shown in Table 2.

Table 2.

View Table

Table 2. Example Training Datasets: The Table on the Top Shows the CPS Data and the MC Dropout Model’s Outputs; the Table Below Shows the Training Dataset for Prediction Validator

Based on our running example of the Robot dataset in Section 3.2, we created a DL model to predict the direction of the motion. The DL model reads data from the test dataset as input values and then predicts the label. The training dataset might not contain enough instances to learn all situations. Our MC dropout-based approach will quantify uncertainty of the DL model’s prediction with the entropy and softmax prediction probability variance. Table 3 shows the entropy and softmax prediction probability variance in the Entropy and Pred. Variance columns. These two values are associated with each row, i.e., one set of sensor values. The row numbers are the same as in Table 1; i.e., they identify the same set of sensor values. Higher entropy and prediction variance values indicate higher uncertainty associated with a prediction. For example, row 2 with the entropy value of 0.95 for the prediction Move-Foward has high uncertainty as compared with row 1 with the same prediction that has the entropy value of 0.56.

The prediction validator will validate the DL model’s predictions, based on uncertainty values. Validation results are illustrated in Table 4. Wrong predictions are highlighted in the grey rows.

Suppose we created another dataset using the DL model employed for CPS and its uncertainty outputs shown in Figure 3. There are two class labels in the new dataset: correct and wrong predictions. The class labels are separable in the Euclidean space using a region-based classifier such as RBF kernel-based SVM. Then, the new classifier’s output can be used to show whether the DL model employed for CPS data prediction for an input instance is likely correct or wrong. Let \( X \in \mathbb {R}^{m \times n} \) denote the CPS test dataset and \( \mathbf {y} \in \mathbb {R}^m \) denote the vector of corresponding labels for X; then the DL model employed for CPS data \( h: X \mapsto Y \) outputs an uncertainty dataset \( \mathcal {U} \in \mathbb {R}^{m \times k} \), where k is the number of uncertainty metrics (i.e., variance, entropy).

Fig. 3.

View Figure

Algorithm 1 shows the uncertainty dataset generation and prediction validator model building. The DL model employed for CPS data is built in lines 1–2. The uncertainty dataset is created in lines 3–6. The prediction validator model is built in line 7.

We designed our prediction validator for classification tasks. To extend it for regression tasks (i.e., continuous outputs), we need to adjust its implementation. For example, one can use the standard error of the regression to create a training dataset, which describes the average distance that predicted values fall from the ground-truth regression line. This metric can label wrong predictions for the training dataset for the prediction validator model. More specifically, if a continuous output is higher than the standard error, we can label it as wrong prediction. We will implement such a prediction validator in the future and evaluate its effectiveness with relevant case studies.

5.2 Dropout Ratio Selection

Considering that a comprehensive approach would increase uncertainty values of incorrectly labeled input instances (i.e., FN and FP) while reducing uncertainty values of correctly labeled input instances (i.e., TP and TN), the model’s prediction time dropout ratio, p, should be tuned for the base model to produce lower uncertainty values for true predictions and higher uncertainty values for false predictions. At the same time, this approach should maintain the prediction performance of the prediction model. Another parameter for estimating the best dropout ratio p value is model prediction performance. The \( F_1 \) value of the model is maximized to solve this optimization problem. The optimization problem’s output is the optimal solution for the given model parameters. The equation is as follows: (4) \( \begin{equation} \begin{aligned}\text{arg\,min}_p & \left(\frac{1}{m} \sum _{\mathbf {x}_i \in \mathcal {D}} \mathbb {I}\left(y_i =f_p(\mathbf {x}_i)\right)\mathcal {U}(f_p(\mathbf {x}_i))\right.\\ & \quad \ +\ \frac{1}{n} \sum _{\mathbf {x}_i \in \mathcal {D}} \mathbb {I}\left(y_i \ne f_p(\mathbf {x}_i)\right) \frac{1}{\mathcal {U}(f_p(\mathbf {x}_i))}\\ & \left.\quad \ +\ \frac{1}{F_1 ^{(f_p)}}\right), \end{aligned} \end{equation} \) where m is the number of correctly labeled examples, n is the number of incorrectly labeled examples in the unseen input space, \( \mathcal {U} \) is the uncertainty quantification metric (i.e., entropy or prediction variance), and \( F_1 ^{(f_p)} \) is the \( F_1 \) performance metric value of the model f. We used the greedy approach for the minimization of Equation (4) to estimate the best \( p \in [0,1] \) value for the optimal model classifier. The optimization problem is a simple minimization problem; there is only one variable, p, that justifies the sufficiency of using a greedy algorithm.

6.1 Experiment Design

We designed a series of experiments by uniformly varying 50 different p ratios from 0.01 to 0.90. Each experiment was executed 20 times, the results of which, corresponding to each p ratio, are averaged to smooth the plotting. We selected the best hyper-parameters for the DL models for the datasets we used in the experiments using a simple grid search, and it turned out that the best hyper-parameters are the Rmsprop optimizations with a learning rate of 0.01. Figure 4 shows the DL models for each dataset. The code is available from GitHub.²

Fig. 4.

View Figure

In the rest of the section, we first present the datasets we employed for the experiments in Section 6.1.1, followed by research questions (RQs) (Section 6.1.2) and evaluation metrics (Section 6.1.3).

6.1.1 Datasets.

In this work, we used three publicly available and one industrial CPS datasets: SWaT [19], Robot [13], KITTI [38], and Video conferencing [44] to demonstrate the efficiency of NIRVANA.

The SWaT testbed implements the process for secure water treatment. The testbed has four security attack scenarios: single-stage single-point, single-stage multi-point, multi-stage single-point, multi-stage multi-point. The testbed consists of 51 sensors and actuators and their values and states are logged every second. The labeled data in this testbed consists of whether an attack occurred or not. To obtain such labels, various attacks according to the four scenarios were introduced to the testbed. The data we used is for over 11 days and it consists of both normal and attack scenarios.

The Robot dataset is the same as our running example presented in Section 3.2; however, the running example was simplified so that it can be used to explain things easily.

In the Video conferencing dataset, the data was collected about whether two Video conferencing systems can establish calls with each other under different configurations of the systems. The two Video conferencing systems are industrial CPSs that are available to us through our previous research collaborations.³

The KITTI dataset contains a series of computer vision tasks built using an autonomous driving platform. We used the KITTI road-type classification task [15]. The original dataset was designed for object detection and segmentation. However, the dataset has six different categories: City, Residential, Road, Campus, Person, and Calibration.⁴ In our experiments, we classify road types of the KITTI dataset’s images and consequently selected these three relevant categories: Residential, City, and Road. The other three categories are not relevant since their images contain either no roads or only a few walking paths. Figure 5 shows the class label distribution of the classes. We selected 50 images for each class to build the DL road classification model. The training time is quite high because of the images. Therefore, we could only select 150 images due to the practical constraints on the computational infrastructure we have, though the original KITTI dataset contains a fairly large number of images.

Fig. 5.

View Figure

For the SWaT, Robot, and Video conferencing datasets, we developed feed-forward-based DL models for prediction. Since the KITTI dataset contains only autonomous car camera views (i.e., images), we developed a CNN-based DL model for KITTI to make predictions. In traditional image classification and object detection tasks, CNN-based DL models are most commonly used to solve problems that cannot be solved in feed-forward neural networks [27, 48].

Highly correlated features in a dataset are computationally expensive for the model building phase. To simplify it, we removed the highly correlated features from the SWaT, Robot, and Video conferencing datasets and trained each model using only low-correlated features to build more reliable and simple models. The resulting models are used in the evaluation phase to determine the prediction performance.

6.2 Experiment Execution

All the experiments were performed using Python scripts and ML libraries: Keras, Tensorflow, and Scikit-learn, on the following machine: 2.8 GHz Quad-Core Intel Core i7 with 16GB of RAM.

For each dataset, two models, with and without MC dropout, were trained to obtain prediction results. In the first model, the dropout is applied at the prediction time. The hyper-parameters such as the number of hidden layers and the number of neurons in the hidden layers, the activation function, the loss function, and the optimization method are the same for both models.

Each dataset has been divided into two parts: 33% test sets and 67% training sets. In the test sets, we randomly chose a subset of the datasets and use them to evaluate the prediction performance of the models. The number of epochs was set to 500, and the epoch size was fixed for all the models. The prediction time dropout values were set to 0.01 and increased to 0.90 with an increment of 0.02.

We used dropout layers at the prediction time only. If we traditionally used the dropout method in training time, the DL models’ neuron weights for each experiment (i.e., for different dropout values) could be different from each other’s model. Here, we aim to carry out uncertainty measurements with varying dropout values of the DL model, whose neuron weights are very close to each other. If we use training time dropouts, then our DL models’ weights can be very different in each experiment.

MC dropout models were used for each test input with \( T = 100 \) times for prediction of the test dataset. The predicted labels for the test set were calculated by averaging the output of the last layer’s softmax function over T iterations. The prediction uncertainty for each input in the test sets was quantified using Shannon entropy and prediction variance, respectively.

6.3 Results

6.3.1 Results for RQ0.

To answer this RQ, first, we obtain values of the prediction performance metrics (i.e., accuracy, precision, recall, and \( F_1 \)) of the base NN models and the prediction-time-activated dropout-based NN models, as shown in Figure 2. Table 5 presents the prediction performance for each dataset of the two types of models. As shown in the table, the accuracy of the prediction-time-activated dropout-based NN model is roughly equal to that of the baseline model’s prediction performance.

Table 6.

Dataset	Model	Acc.	Prec.	Recall	\( \mathbf {F_1} \)
SWaT	Prediction-time-activated dropout-based NN	0.97	0.99	0.78	0.87
	Baseline model	0.97	0.99	0.79	0.88
Robot	Prediction-time-activated dropout-based NN	0.98	0.98	0.98	0.98
	Baseline model	0.97	0.97	0.97	0.97
Video conferencing	Prediction-time-activated dropout-based NN	0.81	0.80	0.81	0.80
	Baseline model	0.81	0.80	0.81	0.80
KITTI	Prediction-time-activated dropout-based NN	0.94	0.93	0.93	0.93
	Baseline model	0.94	0.93	0.93	0.93

View Table

Table 6. Prediction Performance Metrics of the Models

Second, all the samples were ranked in descending order according to their uncertainty values to find highly uncertain instances. Third, we discarded the topmost uncertain examples ranging from 0.1% to 60% in the datasets to demonstrate the negative effect of model uncertainty on prediction performance by removing the most uncertain samples from the datasets. Figure 6 shows changes in the prediction performances (measured with \( F_1 \)) of the models with different dropout rates as a result of removing the most uncertain samples from each dataset. The left side of the figure shows the entropy-based uncertainty quantification results, whereas the right side of the figure shows the variance-based uncertainty quantification results. From the figure, we can observe that the prediction performance of the models increases along with discarding highly uncertain samples based on entropy-based uncertainty quantification. When looking at the results for variance-based uncertainty quantification, the trend is not as significant as for the entropy-based uncertainty quantification. In summary, the correlation between the entropy-based uncertainty quantification and the prediction performance is highly negatively correlated for most of the dropout rates and datasets, whereas no correlation between the variance-based uncertainty quantification and the prediction performance can be observed.

Fig. 6.

View Figure

Another way to show DL model performance is ROC plots. In our case, the dropout ratio is varying. Thus, we need to plot 50 different ROCs for each dataset. Consequently, this approach is not suitable to show the relationship between uncertainty and prediction performance.

Concluding Remarks for RQ0: The prediction performance of models increases along with discarding highly uncertain samples based on entropy-based uncertainty quantification, which is, however, not always the case for variance-based uncertainty quantification.

6.3.2 Results for RQ1.

We used entropy and prediction variance values of the DL models’ softmax probability outputs to compare each uncertainty quantification method to determine the most and least uncertain predictions on each dataset. Figures 7 to 10 show the two most uncertain test instances with entropy-based and softmax prediction probability variance-based uncertainty quantification for the four datasets. Each subfigure shows an input example with prediction probability (T times) histogram. The x-axis shows the softmax probability value of the class, and the y-axis shows its frequency.

Fig. 7.

View Figure

Fig. 8.

View Figure

From these figures, one can observe that softmax probability output distributions for each class show very similar patterns in the results. When looking at Figure 9(c), the Video conferencing system model’s prediction is 0, the true label is also 0 for the input example, and the uncertainty is very high (with the prediction variance being 0.15). Also, the class assignment probabilities for this input example are almost equal for each class label: 50.00% and 50.00%, indicating highly unreliable prediction. When looking at Figure 9(a), the Video conferencing system model’s prediction is wrong, exhibiting the average prediction probability for 0-class being 49.94 %, and 1-class being 50.01% with an uncertainty value being 1 when measured with entropy.

Fig. 9.

View Figure

In summary, the entropy and prediction softmax probability variance-based uncertainty quantification for the three datasets (i.e., SWaT, Robot, Video conferencing) show very similar patterns in the results in terms of uncertainty quantification and predictions’ distributions, as Figures 7 to 9 show that the DL models’ prediction conforms to the normal distribution. The softmax distribution of the predictions with high uncertainty generally shows a distribution with a midpoint of 0.5. Moreover, the average softmax prediction values of the class labels are also very close to each other. The only exception is for the KITTI dataset, as shown in Figure 10. We can see from the figure that, in general, the KITTI DL model’s predictions have low uncertainty, which is also supported from data reported in Table 5, where the precision, recall, and F1 prediction performance values of the KITTI DL model are higher than the other models.

Fig. 10.

View Figure

Concluding Remarks for RQ1: The entropy- and softmax probability variance-based methods perform similarly in terms of uncertainty quantification and predictions’ distributions, suggesting that highly uncertain instances’ predictions are likely wrong.

6.3.3 Results for RQ2.

We can predict if the labeling is FP or FN with another prediction model using entropy and variance uncertainty values. For this purpose, for each dataset, we created a dedicated SVM-based prediction validator model with the following parameter configurations: RBF kernel, Gamma value being 1,000, Regularization parameter (C) being 1,000. The parameter values were chosen to maximize the prediction validator accuracy based on grid search. The input data of these prediction models are the entropy and variance values of each instance in the training datasets; the labels are correct prediction (0) or wrong prediction (1). These models learn decision boundaries from the training datasets followed by predicting wrong predictions based on regions.

One important feature of a prediction validator model is that it can form decision boundaries that distinguish among classes in a dataset in the form of polynomial/linear functions and clusters [25]. In particular, a class’s boundary is defined as a set of clusters with all the instances having the same label. Figure 11 shows decision boundaries for positive and negative classes separated as clusters. The grey regions cover positive labeled instances (FN and FP predictions for the base NN model), while the red regions show negative labeled instances (TP and TN predictions for the base NN model). In other words, if the entropy and variance uncertainty values of a test instance are in the grey region of the decision boundary, the prediction validator model will classify this instance as a wrong prediction.

Fig. 11.

View Figure

As shown in Figure 11, there are six clusters (separated grey regions) for SWaT, around 20 clusters for Robot, 2 clusters for Video conferencing, and 1 cluster for KITTI. The prediction validator model for KITTI results in only one cluster, implying that the prediction validator models with the RBF kernel can easily distinguish positive and negative instances. Moreover, Table 6 shows the prediction performance results with three different kernel functions: RBF, Linear, and Polynomial. From the table, we observe that the RBF-based SVM model for each dataset achieved the highest classification results. More details about the comparisons will be discussed in Section 7.

Table 7.

Data	RBF				Linear				Polynomial
	Prec.	Rec.	\( \mathbf {F_1} \)	Acc.	Prec.	Recall	\( \mathbf {F_1} \)	Acc.	Prec.	Rec.	\( \mathbf {F_1} \)	Acc.
Robot	0.97	0.95	0.96	0.97	0.81	0.81	0.81	0.81	0.77	0.77	0.77	0.77
SWaT	0.94	0.90	0.92	0.95	0.84	0.84	0.83	0.84	0.79	0.70	0.71	0.78
Video	0.96	0.93	0.94	0.94	0.77	0.77	0.77	0.77	0.74	0.75	0.73	0.75
KITTI	0.97	0.97	0.97	0.97	0.95	0.95	0.95	0.95	0.88	0.85	0.85	0.85

View Table

Table 7. SVM Model Prediction Performance

Table 7 presents results of the comparison of the threshold-based wrong label prediction and SVM-based wrong label prediction. We aim to compare the prediction performance of the prediction validator and threshold-based wrong label prediction method. To make a fair comparison, we used the false-positive rate (FPR) metric to find the matching threshold value. In this way, we developed two different models that have the same FPR value. In the threshold-based wrong label prediction approach, if a prediction’s entropy-based uncertainty value is higher than a predefined value (i.e., threshold value), this prediction is accepted as wrong. Accordingly, we used the 500 different threshold values between the minimum and maximum uncertainty values. In the second step, we calculated the corresponding FPR and TPR values for each threshold value. Finally, the threshold with the closest FPR rate with the SVM FPR rate is selected. The matching threshold values are 0.763 for Robot, 0.373 for SWat, 0.587 for Video conferencing, and 0.910 for KITTI datasets. Though the FPR values are almost the same, the TPR values are higher with our SVM-based method. Therefore, NIRVANA gives better prediction performance than threshold-based wrong label predictions.

Table 8.

Data	Threshold (Entropy) Based			SVM Based
	Threshold	FPR	TPR	FPR	TPR
Robot	0.763	0.094634	0.752941	0.094718	0.88591
SWat	0.373	0.139068	0.566565	0.138698	0.613824
Video	0.587	0.105154	0.844735	0.105459	0.854712
KITTI	0.910	0.082192	0.571429	0.082278	0.95302

View Table

Table 8. Comparison against the Current Standard in Terms of Entropy-based Uncertainty Quantification

The training dataset used to train a DL model may not contain enough examples and may not cover all situations for a given task. In this case, the DL model’s prediction uncertainty value would be high. For this reason, unfamiliar input to the training set will be predicted as wrong by the prediction validator as it has high uncertainty in its prediction. Moreover, the DL model’s prediction will probably be incorrect. As can be seen in Table 6, the performance of the prediction validator model to detect incorrect predictions is quite high, with, for instance, \( F_1 \) scores between 0.92 and 0.97. As a result, the prediction validator model successfully extracts the relationship between uncertainty measurements and incorrect predictions.

Concluding Remarks for RQ2: In most cases, the prediction validator model can extract relationships between uncertainties and incorrect predictions with high accuracy (more than 0.94), recall (at least 0.90), precision (more than 0.97), and \( F_1 \) score (at least 0.92), thereby providing a valuable tool for uncertainty analyses related to incorrect predictions.

6.3.4 Results for RQ3.

For RQ3, we aim to find the best dropout ratios, p, using Equation (4) for each dataset. We conducted all experiments using the test datasets to find the best p values. Figure 12 shows the output (y-axis) of the equation with respect to p. According to the figures, the best p value is 0.13 for SWaT, 0.59 for Robot, 0.02 for Video conferencing, and 0.09 for KITTI datasets. We observed that the best p value selected for Robot is higher than for the other datasets. This is because, from the figure, we can see that the standard deviation of the outputs of Equation (4) with varying p values (i.e., the dark grey area) for Robot is large. One can also observe that the outputs increase until the p value reaches 0.59. Therefore, we think that choosing a lower value for p as the best dropout ratio will not make much difference in uncertainty quantification.

Fig. 12.

View Figure

Figure 13 shows the CPS data models’ entropy-based uncertainty quantification with the best p values for each dataset. The figure has the same shape for all four datasets. In each figure, the x-axis shows the model uncertainty value and the y-axis shows the density of data points. The green lines are the TP and TN values of each model. For instance, Figure 13(a) shows the uncertainty histogram of the SWaT dataset. Its sub-figure on the left shows the uncertainty of positive examples (true Attack labels) of predictions with TP and FP labels, respectively. The TP and TN labeled instances’ uncertainty histogram values are clustered in the lower values (the left side of the figures), which means low uncertainty on correctly labeled samples.

Fig. 13.

View Figure

On the other hand, the FP and FN labeled instances’ uncertainty histogram values are more scattered in the upper values (the right side of the figures), indicating high uncertainty in incorrectly labeled samples. This pattern can be observed for all four datasets. The only exception is on the TN predictions for the Video conferencing dataset. As shown in Figure 13(c), the uncertainty histogram values for the TN predictions (in the Connected class) with FN predictions are clustered in higher uncertainty values (the right side of the figure), indicating high uncertainty both in incorrect and in correctly labeled samples. The main reason is that the Video conferencing dataset’s instances with the negative labels have a similar pattern as instances with the positive labels. Wrong predictions for the KITTI dataset are only in the Road class. There are no wrong predictions in the other two classes. The Road class’s histogram plot shows a similar pattern as the other datasets’ prediction histogram plots; the uncertainty values of correct predictions are on the left side, while the uncertainty values of wrong predictions are on the right side of the figure.

In conclusion, the uncertainty values of the correct and incorrect predictions are clustered in different regions in the figures. The correct predictions’ uncertainty values are on the left side (i.e., low uncertainty), and the wrong predictions are on the right side (high uncertainty).

Concluding Remarks for RQ3: The selected best dropout ratio for each dataset (0.13 for SWaT, 0.59 for Robot, 0.02 for Video conferencing, and 0.09 for KITTI) successfully increases the wrongly labeled instances’ uncertainty values while reducing the correctly labeled instances’ uncertainty values.

6.3.5 Results for RQ4.

We have shown in RQ0 that samples with high uncertainty have a high rate of wrong predictions. Highly uncertain samples can be used for retraining to improve the DL models’ prediction performance. To achieve this, we iteratively retrained each DL model. At each retraining iteration, predictions were made using the test datasets, and we further ranked these predictions in descending order according to their uncertainty values. We used the most uncertain instances in the next iteration of retraining, and consequently improved the prediction performance of the DL models.

To converge each iteration, the number of new rows in retraining was selected using a decaying approach, as shown below: (5) \( \begin{equation} num\_of\_rows = \frac{ \sum _{\mathbf {x}_i \in \mathcal {D}}{\mathbb {I}\left(y_i \ne f_p(\mathbf {x}_i)\right)}}{1 + \alpha \cdot iter\_no}, \end{equation} \) where \( y_i \) is the actual output, \( f_p(\mathbf {x}_i) \) is the predicted output, and \( \alpha \) is the tuning parameter. The \( \alpha \) value is 1.0 for the SWaT, Robot, and Video conferencing datasets, and 0.2 for the KITTI dataset. The KITTI dataset’s number of instances is lower than the other datasets. Thus, we aimed for the iteration steps to take longer by choosing a lower \( \alpha \) value. In this way, we were able to have more \( num\_of\_instances \) values in each iteration step. Figure 14 shows the prediction performance improvement for each dataset. The SWat, Robot, and Video conferencing DL models’ iteration sizes are 30, while the KITTI DL model’s is 5 because, as shown in the figure, the KITTI DL model converges to 1.0 within five iterations.

Fig. 14.

View Figure

Table 8 summarizes the prediction performance improvement of each DL model. According to the table, all the DL models’ prediction performance increases after being retrained with highly uncertain training instances provided with our approach. For instance, the prediction performance improvements of the DL models in terms of accuracy are 2.17% for SWaT, 16.98% for Robot, 2.23% for Video conferencing, and 9.09% for KITTI.

Table 9.

	Accuracy		Precision		Recall		F1
	Before	After	Before	After	Before	After	Before	After
SWaT	0.96457912	0.98552189	0.96563238	0.98546561	0.96457912	0.98552189	0.96191914	0.985234
Robot	0.80122154	0.93725708	0.80191343	0.93729346	0.80122154	0.93725708	0.80088805	0.93724854
Video	0.97038835	0.99199029	0.97128911	0.99198228	0.97038835	0.99199029	0.9706586	0.99198553
KITTI	0.91666667	1.0	0.91841492	1.0	0.91666667	1.0	0.91652174	1.0

View Table

Table 9. Accuracy Improvement of the DL Models When Retrained with Most Uncertain Input Instances

Concluding Remarks for RQ4: Retraining DL models with highly uncertain training instances increases their prediction performance by 2% to 17% on all four metrics. These results suggest that the prediction performance of a DL model can be improved by retraining the model with uncertain instances.

6.4 Threats to Validity

A key external validity threat is related to the generalization of results [43]. In our experiments, we used only four CPS datasets, and we definitely need more case studies to generalize the results. Moreover, these datasets reflect different types of CPSs, including robotics, automation systems, and communication systems.

Our key construct validity threat is related to the selection of uncertainty quantification metrics that do not precisely quantify uncertainty. Nevertheless, note that these quantification metrics are from the literature [43] and have been applied to quantify uncertainty. In the future, we will conduct dedicated empirical studies to systematically investigate more uncertainty quantification metrics. Another threat is to use the SVM algorithm for the prediction validator model. One reason was that the SVM models are capable of finding nonlinear decision boundaries in input space [24]. Moreover, the SVM algorithm is capable of separating class instances using a hyperplane with kernel trick (i.e., imitating higher-dimensional input space). Nonetheless, in the future, we will conduct experiments to build a prediction validator to investigate more classification algorithms. The last threat is related to activating the dropout layers in the prediction time only. Because of this, our approach is not fully compatible with Gal’s approach in terms of quantifying uncertainty in DL models. Also, we developed our implementation without using any uncertainty quantification library. Being now aware that the uncertainty-wizard library [57] makes uncertainty quantification compatible with BNN models, we will carry out experiments by using BNN models and the uncertainty-wizard library in the future.

Our main conclusion validity threat is due to finding the best dropout ratio that is responsible for increasing uncertainty for the wrong labeling instances and decreasing the uncertainty for the correctly labeled instances. To mitigate this threat, we repeated each experiment 20 times for each dataset to reduce the probability that the results were obtained by chance. In a standard neural network training, all weights are initialized uniformly at random. In the second stage, using optimization, these weights are updated to fit the classification problem. Since the training started with a probabilistic approach, there is a possibility of facing the local minimum problem in the optimization. In order to eliminate the local minimum problem, we repeated the training 20 times to find the p value that gives the best result. In each repetition, the weights were initialized uniformly at random but with different values, such that if the optimization function failed to find the global minimum, in the next experiment, it is likely to find it as the weights have been initialized with different values.

7.1 Detection of Wrong Predictions

This study demonstrated that there is a strong relationship between DL models’ prediction accuracy and predictive uncertainty with various CPS datasets. In existing uncertainty-based prediction quality studies [17, 39], a model’s prediction reliability for a given instance is generally decided using only a threshold value. If a prediction’s uncertainty value is higher than a predefined threshold value, the system concludes that this prediction is likely wrong. Our study shows that this relationship is not simply a threshold or a linear correlation. We created uncertainty datasets for each DL model employed for CPS data using the models’ prediction uncertainty over the entire input CPS datasets. Labels of such an uncertainty dataset are binary classes of correct predictions (TP and TN) and wrong predictions (FP and FN). In this dataset, we observed that correct and incorrect predictions are clustered into specific regions. For this reason, we built an SVM model with RBF kernel that creates region-based decision boundaries in the uncertainty datasets. Thus, we increased the performance of the detection of wrong predictions as shown in Table 6.

In this study, the prediction validator estimates wrong predictions of classification models using uncertainty metrics. Apart from classifications, a regression model’s wrong predictions can be calculated in the same way by using error metrics such as mean absolute error, mean square error, and root mean square error. Therefore, the prediction validator approximation is suitable not only for classification but also for regression problems.

7.2 Applications in the Context of the Studied CPS Datasets

Below, we discuss such applications for each of the studied CPS datasets and the corresponding underlying CPS.

SWaT: The SWaT CPS case study is focused on the detection of security attacks. To this end, uncertainty datasets produced with NIRVANA can be used as training data or testing input to facilitate the development and testing of an uncertainty-aware DL-based attack detection system that can predict attacks in various uncertain situations such as due to sensor and actuator errors. The aim of such attacks is to deceive the system and lead it to anomalous behavior. When the decision of the DL model employed for CPS data is wrong, it makes the target system fail (e.g., wrong behavior of actuators). In the studies about “Adversarial machine learning” attacks such as FGSM [20], BIM [30], and DeepFool [40], the uncertainty of the DL models’ decisions as a result of the attacks is high [12]. For this reason, the use of uncertainty datasets in CPSs’ attack detection will increase the reliability of the system.

Video conferencing system: For a CPS to work correctly and efficiently, it is essential that its system parameters are properly configured during its operation. A video conferencing system running with incorrectly configured parameters is highly likely to fail. In this context, we foresee two possible applications. First, an uncertainty-aware DL-based component can be developed to predict call failures ahead of time due to uncertain configurations of the video conferencing system. Such a component, in addition, can produce quantified uncertainty associated with each prediction, which can be used by the system to make better decisions such as keeping the call connection but reducing the quality of videos of call participants automatically. Second, an uncertainty-aware method, by benefiting from uncertainty datasets, for testing videoconferencing systems under various configurations can be developed.

Robot: In the context of this system, we can possibly develop DL-based systems to predict the next move of the robot even in uncertain situations such as obstacles with quantified uncertainty for each move decision. If decisions made by an autonomous robot system involve a high level of uncertainty, such decisions are probably wrong. It is critical for autonomous robots to measure the quality of their decisions during navigation and be aware that their decisions may be wrong and might need to pursue human intervention. For example, we think it can be used for autonomous robots that can be used in extraterrestrial environments such as the Moon and Mars or deep ocean research.

KITTI: The KITTI case study is focused on the detection of road types from autonomous cars’ cameras. We can possibly develop DL-based systems to predict road types in uncertain situations such as obstacles with quantified uncertainty for each steering wheel angle decision. Detecting the prediction error made by the DL model employed for CPS data and warning the CPS operator are of vital importance in many cases to avoid fatal crashes. Examples of autonomous cars making wrong decisions include detecting lanes incorrectly and outputting wrong steering angle and braking commands. Such wrong decisions often have a high degree of associated uncertainty, if it is measured. Figure 11(d) shows the prediction validator’s decision boundaries based on entropy and variance uncertainty values. While the entropy value is high, the variance value is low in some instances. The opposite is also observed in some other instances, indicating that wrong predictions may not have high uncertainty as shown in Section 6.3.5. As we have shown in our experiments, it is necessary to use another prediction model to derive this uncertainty and prediction error relationship.