Identification of encrypted and malicious network traffic based on one-dimensional convolutional neural network

There are four main network traffic classification methods [15‐17]: methods based on port identification [18], methods based on deep packet detection [19], methods based on statistical processing [20], and methods based on user behavior [21]. Most network protocol ports are based on security policies; thus, the identification accuracy of such port identification-based methods is relatively low, and the deep packet detection-based method cannot process the current encrypted network traffic data. With the ubiquitous usage of machine learning [22‐24], researchers are investigating approaches based on statistical processing and behavioral norms.

Traditional network traffic classification methods include clustering, support vector machines, C4.5 Decision Tree (C4.5), and etc. Most of these traditional methods have low accuracy and low classification efficiency. Anshu Priya et al. [25] proposed the use of K-Means clustering algorithm to analyze real-time network data traffic situations in universities. However, the clustering algorithm has poor classification efficiency for data categories with high similarity. Wang et al. [26] used C4.5 to classify P2P traffic, which is used to describe the behaviour characteristics of applications. The disadvantages of the C4.5 algorithm is that the training time is long and it is only suitable for processing small data sets. Coull et al. [27] proposed to classify p2p traffic by analyzing packet features and proposed traffic analysis of encrypted messaging services: Apple iMessage and other message classification. Mauro et al. [28] proposed to uncover encrypted WebRTC traffic by machine learning tools, using the random forest approach. Traditional feature-based statistical classifiers are becoming less suitable for today's massive data processing.

Deep learning has been gradually hybrided with more research fields to generate more efficient and appliable network models thanks to its powerful function extraction capability and efficient model parameter calculation. Segun I. Popoola et al. [4] proposed a deep neural network to classify network traffic in the scenario of Internet of Things, aiming at Zero-Day Botnet Attack Detection. However, the time cost of training model is large, so it cannot be applied to large-scale data. Shi Dong et al. [29] proposed an optimization method for abnormal network traffic detection based on a semi-supervised double-depth Q-network (SSDDQN). Based on the above, Shi Dong [30] proposed an improved support vector machine (SVM) algorithm, the cost-sensitive support vector machine (CMSVM), to solve the imbalance problem in network traffic identification. Wang et al. [31, 32] for feature extraction from raw traffic data after preprocessing in two dimensions of CNN-1D and CNN-2D. The authors demonstrated the superiority of these two methods by observing and elaborating the accuracy scores achieved in the experimental evaluation metrics, etc. Lotfollahi et al. [33] proposed Deep Packet: a new method for cryptographic traffic classification using deep learning. However, the shortcoming of deep packet detection technology is obvious, it is vulnerable against the same kind of network attacks, and the deployment of deep packet detection is difficult, lest additional burden on the processor. Zou et al. [34] proposed a method for cryptographic traffic classification method based on convolutional Long Short-Term Memory (LSTM) neural networks. However, after a long period of training and increasing of the number of layers, the problem of gradient explosion is easily encountered. Bu et al. [35] proposed a deep parallel network (NIN) neural network model. Since its introduction, Deep learning has played an increasingly important role in machine learning. Convolutional neural networks (CNN), recurrent neural networks (RNN), and long and short-term memory network (LSTM) models gained their recognition for their excellent performance in the field of computer vision.

There are many common traffic classification methods, each with its own advantages and disadvantages. For example, port number-based classification is the easiest to implement, but has low identification accuracy and limited applicability. The classification method based on deep packets has a high accuracy but cannot detect encryption services. Therefore, future research will focus on network traffic classification and identification using machine learning methods. As a part of machine learning, researchers are trying to apply deep learning to the field of network traffic recognition technology. In this paper, a lightweight neural network model is proposed to identify classified network traffic data types.

The one-dimensional convolutional neural network (HexCNN-1D) workflow is based on the network traffic recognition method. The input data of the model are the hexadecimal data obtained after preprocessing. After training the model, the network traffic identification work is completed according to the different traffic categories.

To prevent overfitting, we added an attention mechanism and a batch normalization layer to the design of the HexCNN-1D model. Normalization returns an uneven distribution to a normalized distribution. This allows the processing data to be distributed into sensitive regions of the activation function, speeding up model training and preventing gradient disappearance.

The flow of the HexCNN-1D algorithm based on a convolutional neural network is shown in Fig. 1.

Considering the large amount and load of network traffic data to be processed, the traditional one-dimensional convolutional neural network model design cannot meet the lightweight requirement of identifying the types and categories of encrypted and malicious traffic with higher accuracy. Therefore, we add a normalized processing module and an attention mechanism module within our model.

When designing the convolutional neural network model, the Batch Normalized (BN) module is considered as an addition to the normal convolutional neural network model [38]. The BN module can solve problems such as slow convergence rates and gradient saturation caused by internal covariate shift [39].\({{x}_{i}}^{(b)}\) represents the value of the \(i-th\) input node of this layer when the \(b-th\) sample of the current batch is input, \({x}_{i}\) for \([{x}_{i}^{1}, {x}_{i}^{2}, {x}_{i}^{3},\dots ,{x}_{i}^{m}]\) a row vector, length of batch size m, \(\mu\) and \(\sigma\) for the mean and standard deviation, \(\epsilon\) division by zero to prevent the introduction of a minimum quantity (negligible), \(\beta\) and \(\gamma\) for the shift and scale parameters.

Due to the uneven distribution of data, the model will pay more attention to sufficient data, which will affect the final classification effect. As mentioned in this paper [40], CBAM is a lightweight general module, that can be applied to any CNN model and plays a non-negligible role in the application of GAB and CAB [41]. GAB and CAB can be used to learn the recognition features, so as to better resolve the problem of low accuracy caused by uneven data distribution.

The channel attention feature \({M}_{c\_a}\) is calculated in Formula 2, where \(H\) denotes the height, \(W\) represents the width, \(C\) represents the number of channels, and \(ReLU\) represents the use of ReLU activation function, \(GAP\) represents the global average pooling, \({M}_{G-IN}\) denotes the use of 1 × 1 convolution layer to reduce the number of channels.

The number of channels required for each category is calculated by \({M}^{\mathrm{^{\prime}}},{M}^{\mathrm{^{\prime}}}\in {R}^{H\times W\times ck}\), where \(c\) is the number of channels needed to identify each category, and \(k\) is the number of classes. Half of the features are retained by \({M}^{"}({M}^{"}={M}^{\mathrm{^{\prime}}})\), and the Dropout function is removed to make a prediction with all the features.

Formula 3 calculates the output of GAB, namely the spatial attention feature map \({M}_{G-OUT}\), \({M}_{G-OUT}={M}_{G-IN}\). \({M}_{G-OUT}\) is used to store the subtle and different information of each network traffic category in the detailed network traffic data, which is used as the input to the subsequent CAB.

As can be observed in Formula 4, \({S}_{i}\) represents the degree of significant response to the feature mapping of each category, \(GMP\) represents the global maximum pooling, \({m}_{ij}^{"}\) represents the JTH feature of class \(i\) in \({M}^{"}\) and the score \(S\) of each category of network traffic is calculated by averaging the sum of \({M}^{\mathrm{^{\prime}}\mathrm{^{\prime}}}\) maximum pooling.

In Formula 5, \({M}_{i\_avg}^{\mathrm{^{\prime}}}\) represents the feature output mapping feature map of the class \(i\), and \({m}_{ij}^{\mathrm{^{\prime}}}\) represents the reaction of the JTH feature of the class \(i\) in \({M}^{\mathrm{^{\prime}}}\). The sum of the characteristic fractions of each class is calculated and averaged.

In Formula 6, \({A}_{CAB}\) is to multiply and average the calculated scores of each class and the semantic features of the class. It helps to differentiate areas of DR Grading.

Finally, as shown in Formula 7, \({M}_{C-OUT}\) is obtained by multiplying CAB and category attention \({A}_{CAB}\), enabling the model to obtain more accurate classification of different network traffic categories.

In this section, the public network datasets ISCX and USTC are used for experiments. The testing ratio of the training set was set to 7:3, and the sample set used in each experiment was described in detail.

In this research, four classification indices were used in the experiment: Accuracy, Precision, Recall, and F1-score. TP denotes the positive sample correctly predicted by the model, FN denotes the positive sample incorrectly predicted by the model, FN denotes the negative sample incorrectly predicted by the model, and TN denotes the negative sample correctly predicted by the model.

We use the ablation experiment and the confusion matrix [42] to validate the detection of different data traffic categories and the experimental results. Ablation experiments are commonly used in neural networks to learn about the network by deleting part of the network and studying its performance. The confusion matrix's function is to group the expected and actual results of all categories into the same table based on category. In this table, we can clearly observe the number of accurate and inaccurate recognitions for each category.

The ISCX dataset contains traffic characteristics and raw traffic (in PCAP format). In our experiment, the experimental environment was divided into two categories (VPN and non-VPN), nine and eighteen.

The UTSC dataset uses the class 7 + 3 (seven non-malicious traffic and three malicious traffic) categories in the UTSC dataset to determine the model's ability to detect malicious traffic. We have 1,000 of each, for a total of 10,000 samples. The experiment went through 50–60 iterations.

For hardware and software configuration, we have used python3, PC version of Windows 11, Processor 12th Gen Intel(R) Core (TM) i5-12500H 2.50 GHz, running memory 16.0 GB.

We iteratively optimized the hyperparameters of the model and conducted a lot of model tuning mainly for batch processing [43], optimizer, loss function, normalization operation, etc., as shown in Table 3 below, the optimal parameter settings of the model are provided. The Adam optimizer is capable of updating the model parameters by calculating gradient optimization. Softmax loss function, etc.

In order to evaluate the effectiveness of the model by adding normalized processing and attention mechanisms, we performed ablation experiments on HexCNN-1D. As shown in Table 6, the model is mainly processed by a one-dimensional convolutional neural network, followed by modules for normalization processing and attention mechanism.

First, a single one-dimensional convolutional neural network was tested to calculate the Accuracy, Precision, Recall and F1-score of the model. Then, the accuracy of F1-score and other indicators of the model were increased by about 3% after the addition of normalized processing. Finally, CAB and GAB were added to the base model, and the overall index increased by about 2%, indicating that the attention module improved the efficiency of the model in identifying network traffic categories.

We used the confusion matrix shown in Fig. 4 to verify the experimental data and the classification accuracy of the experimental results.

The experimental results show that the HexCNN-1D classification model adopted in this paper has higher accuracy in four experimental scenarios, and has achieved excellent recognition results in the scenarios of encrypted traffic and malicious traffic identification.