Securing 6G-enabled IoT/IoV networks by machine learning and data fusion

The captured network traffic packets are firstly transformed to conversations among devices. Main features include originator (source) and responder (destination) addresses, ports and payload size in bytes and number of packets per conversation. General statistical calculation methods are employed to characterize datasets, including mean, standard deviation, quantiles at 0, 25, 50, 75, 100 percentages, skewness [9], kurtosis [12, 15] and median. Later, the first-order and second-order differentials of the above statistical characteristics are investigated to find insight between performance of algorithms and dataset fusion.

As mentioned previously, this work uses machine learning to indicate non-neural network methods. XGBoost algorithm is currently one of the widely used representative machine learning methods and is used as a baseline here.

Tree-based algorithms have high compatibility and can effectively classify and predict instances with a variety of different types of data.

Bagging trains several sub-models, and each model is trained using a generated bootstrapped dataset. Bootstrapping is a resampling method which samples instances (usually with replacement) from the original dataset. Datasets created by bootstrapping are only weakly dependent and randomness of samples matters. Thus, bagging builds sub-models in a parallel way. Boosting, however, builds sub-models in a serial manner with rounds. In boosting, a base sub-model is firstly build using selected weak learner. Later, the inference results are analyzed, and wrongly predicted instances are paid more attention in the next round using weighting strategies to reduce residuals. This continues until termination conditions are met. The XGBoost objective function consists of loss function part indicating model deviation and regularization part indicating model variance. Suppose the loss function is l and the regularization is \(\Omega\), the objective function for each round is:Those boosted models are then ensembled and used consecutively and additively. XGBoost objective functions can be approximated using Taylor series to unary quadratic polynomial functions shown as the formula below:where the variable \(X = {x_0,\ldots , x_n}\) represents features in the training dataset. Usually, most features contribute little or even negatively and so they are given small weights or even discarded. Another reason to do so is to avoid over-fitting, i.e., avoid over-complex models, which only shows high accuracy during training while reduces generality. A good practice is trying to use as much data as possible during training to get models with high accuracy (low deviation) and high precision (low variance). The above ideas are combined together and lead XGBoost to a common good baseline.

Deep learning methods are developing rapidly though tabular-targeted deep architectures are limited. TabNet is a neural network algorithm for tabular data published by Google [4]. It implements instance-wise feature selection through a sequential attention mechanism similar to the additive model. Self-supervised learning is also realized through encoder-decoder framework. TabNet has some advantages over deep neural network (DNN) [23]. Similar to image processing, TabNet can encode the data of the table and obtain a representation of the original table data, which is a representation learning method. In this way, the model can learn the corresponding features from the original data without human intervention for feature extraction and corresponding processing. At the same time, the model can reduce its dependence on feature engineering [16] (i.e., feature cleaning, processing, selection, etc.). TabNet algorithm also has some advantages of tree algorithm [2] in table data prediction. For example, the decision manifold is good, the model can be learned sparsely, the model has high explanatory degree, and the training speed is fast. This makes it not only has the advantages of DNN, but also can be comparable with the current mainstream tree model in the prediction of table data. TabNet combines the idea of Sequential Attention with the behavioral logic of decision trees, and simulates the algorithmic logic of decision trees through the framework of multi-step neural networks.

A step of the algorithm corresponds to the establishment of a tree in the decision tree algorithm. The steps of the algorithm mainly include two key operations: 1.Attentive Transformer. The operation of this process is mainly to select some of the most important features for the next operation, that is, to simulate the feature selection step of the decision tree. The function of this layer is to calculate the Mask layer of the current step according to the output results of the previous step (the mask layer realizes feature screening through matrix operation).

The Sparsemax layer is a sparse version of Softmax. Fully connected (FC) layer is a fully connected layer where weight and bias are specially set, and it can make conditional judgment on the input feature vector. BN layer can reduce the size difference of different features in numerical value, and reduce the influence on the training process, and reduce the problem of gradient explosion and gradient disappearance. It is worth noting that the Mask vector of different samples can be different, that is to say, TabNet algorithm can make different samples select different features (instance-wise) to reduce the prediction error caused by too many samples. The traditional decision tree model uses the same features for each tree. 2. Feature Transformer. The main operation of this process is to process the input features into a more useful representation, that is, to simulate the decision tree and set the Feature threshold for Feature calculation.

The functions realized by BN (batch normalization) layer and GLU (gated linear units) layer at FC layer are similar to branching tree by setting threshold value in tree algorithm, and the output is the weight of influence feature in the final prediction result. Feature transformer is mainly composed of two parts. Related parameters in the layer of the first half are shared, that is, the parameter is jointly trained in all steps. The main reason for doing so is to extract the common part of feature calculation and determine which features have a high weight in the vast variety of books. In other words, each step is trained separately. In this way, its specific weight for features can be obtained for different samples, that is, the characteristic part of feature calculation can be obtained. Both parts are connected by residuals and multiplied by \(\sqrt{0.5}\) to ensure the stability of the network.

Using both attentive and feature transformers, TabNet is able to simulate the decision-making process of a tree-based model. This model has the characteristics of both DNN and tree algorithm, and has been proved to have good prediction effect in image processing and table data processing.

When considering the tradeoff between true positive versus false positive, or sensitivity versus specificity, an optimal cutoff value is needed. The optimal cutoff is calculated using Youden’s J score statistic (Youden’s index) [48], which is defined as:that is,It is a reflection of to which extend the algorithm is good at distinguishing normal versus abnormal instances at balance considering both sensitivity and specificity simultaneously. The bigger Youden’s index, the better algorithm is. From visual perspective, the selected cutoff value is actually in correspondence to the point on the curve that is most close to the top-left corner of the rectangular plotting area.

Some metrics are used to evaluate algorithms and their results. For prediction and classification, commonly used effective metrics include accuracy, receiver operator characteristic (ROC) curve, the area under ROC curve (AUC). Accuracy is a direct and simple method to provide a quantitative comparison, roughly. Imbalanced data occur now and then [21], and accuracy is not a suitable metric in this situation. Unfortunately, network traffic dataset, in most cases, is imbalanced dataset with much more benign instances than malicious ones. Thus, it is necessary to have a well-recognized metric to evaluate results during network traffic analysis.

For classification, a good way to present results is to use confusion matrix [41] which includes key values that can be used for other purposes. A confusion matrix is an N*N matrix, the horizontal and vertical indexes are related categories, the diagonal lines are correctly classified data, and outside the diagonal lines are incorrectly classified data. For a dichotomous problem, we can divide the classification into the following four types: (1) Positive samples are correctly predicted to be positive samples (true positive, TP). (2) Positive samples are wrongly predicted to be negative samples (false negative, FN). (3) Negative samples were mispredicted as positive samples (false positive, FP). (4) Negative samples are correctly predicted to be negative samples (true negative, TN). These four cases correspond to the values in the confusion matrix. Receiver operating characteristic curve (ROC) [13] and AUC (area under the curve) value [28] are indicators of classification effect evaluation based on confusion matrix. It is often used to evaluate the merits of a binary classifier. At present, ROC curve and AUC value are popular and common evaluation indexes in classification problems [14]. The horizontal and vertical coordinates of the ROC curve are mainly determined by the following two formulas:Abscissa (horizontal axis) indicates false positive rate (FPR), i.e., the proportion of samples that are predicted to be positive but actually negative in all negative samples. Ordinate (vertical axis) indicates true positive rate (TPR), i.e., the proportion of predicted positive samples and actually positive samples in all positive samples.

The specific drawing process of ROC curve is as follows: First, the classifier will give the probability of all samples being classified as positive samples. We took this probability as the score of the sample, sorted score from high to low, and classified positive and negative samples by score as threshold successively. (That is, each point constituting the ROC curve corresponds to a threshold.) Those higher than this value are positive samples, and those lower than this value are negative samples. For example, when threshold is the largest, TP = FP = 0, corresponding to the origin; when threshold is minimum, TN = FN = 1, corresponding to the point (1,1) in the upper right corner. The coordinates of a series of points (that is, the values of FPR and TPR) can be obtained by the above calculation formula. Finally, all nodes are connected to complete the ROC curve drawing.

The specific drawing process of ROC curve is as follows: First, the classifier will give the probability of all samples being classified as positive samples. So let us take this probability as an example and in the ideal case, the value of TPR should be close to 1, and the value of FPR should be close to 0. That is, the roc curve should be as close as possible to point (0,1) in the figure and deviate from the 45 degree diagonal.

The main advantage of ROC curve is that it can effectively deal with the classification problem under the condition of uneven distribution of positive and negative samples. When the number of positive and negative samples in a data set is too unequal, a phenomenon of class imbalance occurs. (In daily life, uneven sample distribution often occurs.) If we simply take the proportion of misclassified samples to all samples as the accuracy of the experiment, the effect is extremely poor. That is, the classifier can still obtain high accuracy when all samples are predicted to be positive or negative samples. However, the ROC curve can remain unchanged when the distribution of positive and negative samples changes.

AUC value is the area under the ROC curve, which can intuitively evaluate the quality of the classifier, ranging from 0 to 1. The larger the value, the better. Generally, the AUC value can be understood as a probability value, that is, the probability that positive samples rank before negative samples. The larger the AUC value is, the more likely the score of positive samples calculated by the current classification algorithm is to be higher than that of negative samples.

A labeled collection of datasets, Aposemat IoT-23 [10], which is available in public domain with malicious and benign IoT network traffic is used for testing our hypothesis. This collection is created by Avast AIC laboratory with the funding of Avast Software and was firstly published in 2020. It contains 23 datasets (corresponding to 23 scenarios) which consists of 20 datasets from a malware-conquered IoT network (botnet) and 3 benign traffic datasets. The datasets are numbered thus noted as Di in this work. The traffic was firstly captured in pcap format which is frequently used during network capturing [31, 40] and then processed by a analyzer to get flows, i.e., communication conversations.

Zeek is a passive and publicly available open-source network traffic data analyzer and supports performance measurement and troubleshooting [34]. It is often used for monitoring of network security status. It can be deployed to provide near real-time analysis to support investigations of suspicious or malicious activity. Through analyzing captured packets, Zeek finds communication conversations between devices and extract features.

Next, we make detailed statistics of the relevant features of these data, and introduce the feature engineering carried out in this experiment.

The extracted conversation features are conversation starting time, originator and responder IP address, originator and responder port number, transportation layer protocol (TCP or UDP), service (such as HTTP, DNS, DHCP, etc., decided mainly using protocol and port number), conversation duration, application layer payload total bytes sent by originator and responder, IP layer payload total bytes sent by originator and responder, connection termination status, total packets number sent by originator and responder. The malware datasets originate from pcap files sizes ranging from 2MB to 6GB (23 kilo to 271 million packets) and capturing duration from 1 h to 112 h. The number of flows in analyzed results ranges from 238 to about 74 million. The scenarios and malware also vary. It is worth mentioning that the above numbers regarding sizes, flows, packets, etc., are not positively correlated; thus, this IoT-23 dataset provides high diversity which is a good candidate to be investigated.

The conversation instances are labelled in Stratosphere laboratory with “label” and “type”. Label indicates actual actions including normal traffic benign and abnormal traffic such as HeartBeat, FileDownload, PortScan, Attack, Torii (virus’ names) among others. Type is a binary target with benign traffic as “benign,” and all non-benign conversations are marked as “malicious”. Type is the target that we use in this work to train and test.

We firstly analyze each dataset to get some initial overview including data cleanliness and relations among them. Data cleaning is important as many algorithms cannot work with missing values and may be misled by outliers [36]. To make dataset analysis applicable to other domains, robust and general statistical methods are used. For data cleaning part, datasets that contain missing values cannot be sent directly to TabNet so they are processed by k-nearest neighbor to impute missing values. All experiments are conducted using tenfold cross-validation (CV) to ensure the validity and robustness of results.

The proposed method can be represented in pseudocode as Algorithm 1.

The experiments are conducted inside Jupyter Notebook on a machine with Windows 10, Intel Core i9-10900K CPU 3.7GHz, 64 gigabytes RAM, NVidia GeForce RTX 3080 with 10GB RAM Graphic Processing Unit. The python version 3.6 is used together with PyCharm version 2021.1.2 community edition build 211.7442.25, runtime version 1341.57 amd64, and the key packages are NumPy version 1.19.5, pandas version 1.1.5 and libxgboost version 1.4.2.