MFVT: an anomaly traffic detection method merging feature fusion network and vision transformer architecture

With the continuous development of artificial intelligence big data and cloud computing technology, intrusion detection technology is constantly updated using new technologies [20‐23] In 1980, Anderson [24] proposed the concept of intrusion detection technology, which aims to timely identify abnormal behaviors in the network and reduce losses caused by abnormal behaviors. Over the past 40 years, many methods have been used in intrusion detection, all of which aim to sense attacks with good predictive accuracy and improve real-time prediction. These methods all attempting to extract a pattern from network traffics to distinguish attack traffics from regular traffics.

Specifically, Table 1 briefly summarizes the methods used in intrusion detection. Currently, the traditional machine learning methods applied to the field of intrusion detection are mainly supervised learning, such as support vector machine (SVM) [25‐27], K-nearest neighbor (KNN) [28], and random forest (RF) [29, 30]. These methods mentioned above have a high false alarm rate and a low detection rate for attack traffics. It is a common problem in traditional machine learning methods to design a feature set that can accurately reflect traffic characteristics, and the quality of feature set directly affects the classification performance of the method. In recent years, although many researchers have been working on the problem of how to design feature sets [31, 32], how to design a set of suitable traffic feature sets is still an unresolved research topic.

Moreover, deep learning [33] has good self-adaptability, self-organization, and generalization capabilities. Therefore, it can be a good solution to the problem that traditional machine learning needs to manually design a group of feature sets. The use of deep learning can enable detection systems with higher detection efficiency and therefore has been widely studied by scholars in recent years. Yan [34] constructed an intrusion detection system based on convolutional neural network (CNN) and applied generative adversarial network to synthesize attack traces, and experimental results verified the effectiveness of the system. Zhang [13] proposed a deep hierarchical network-based intrusion detection model that combines CNN and long short-term memory (CNN_LSTM) network, and the CNN_LSTM model achieved good performance on the IDS2017 dataset. Lin [35] constructed a dynamic network anomaly detection system, which uses long short-term memory (LSTM) network combined with attention mechanism to detect anomalies. Zhang [36] proposed a two-layer parallel learning cross-fusion deep learning model (PCCN), which uses feature fusion technology to improve the extraction of features from small sample data, and experiments on ablation experiments showed good performance. Zhong [37] proposed HELAD, a network anomaly traffic detection algorithm integrating multiple deep learning techniques. Although HELAD has better adaptability and detection accuracy, its bit error rate is slightly higher.

In 2018, transformer architecture [38] first appeared in the field of natural language processing (NLP), and it has occupied an important position in the field of NLP. Transformer architecture has been continuously improved by subsequent scholars [39]. Vaswani [40] first constructed transformer architecture based on attention mechanism. Devlin et al. [41] proposed BERT, a new language representation model, which pretrains a transformer from unmarked text through joint adjustments of left and right contexts. BERT got the latest results from 11 natural language processing tasks at the time.

Influenced by the excellent performance of transformer architecture in NLP task, scholars began to extend transformer architecture to the field of computer vision and achieved good results. Chen et al. [42] constructed a sequence transformer to perform regression prediction of pixels and obtained competitive results in the image classification task. In 2020, Dosovitskiy et al. [43] proposed a vision transformer architecture, which uses a pure transformer to directly extract the features of image block sequences and obtain the most advanced performance on multiple image recognition reference datasets. Besides the most basic image classification tasks [44], transformer models are gradually applied to various computer vision tasks, and the number of vision models based on transformer architecture has gradually become more and more.

In this paper, the latest intrusion detection model based on feature fusion is improved and integrated into vision transformer architecture, and then a deep learning model (MFVT) that combines feature fusion network with vision transformer architecture is proposed for network anomaly traffic detection. The MFVT takes full advantage of the respective strengths of feature fusion and vision transformer architecture and further improves the detection accuracy of abnormal network traffic by combining with the CPR algorithm proposed by us.

The raw data processing algorithm (CPR) proposed in this paper mainly accomplishes the task of extracting raw traffic data from pcap files and processing them into the two-dimensional matrix that required by the network model [45, 46]. Figure 2 shows the entire data processing process.

Three steps are required to process the raw flow data into a two-dimensional matrix. The specific steps are as follows.

The first step is to extract the raw data of network traffic from the pcap file and then convert the extracted byte type data into binary type data.

In the second step, the converted packets are divided into flows according to the five-tuple, and the number of packets and bytes contained in each packet are limited when dividing the flow. If the number of data packets is insufficient, fill in the preceding item, and if the number of bytes contained in the data packet is insufficient, fill in 0. For the completion of this step, refer to the paper [47]. Through the above operations, a dataset with fixed dimensions can be obtained. The pseudocode is shown in algorithm 1.

In the third step, the network traffic data obtained after the first two steps contain high data dimensions and may have redundant features that are useless for network training, which need to be further extracted. In this paper, the data obtained from the first two steps are directly fed into the PCA algorithm to obtain the data of the required dimensions, and then the data are processed into a two-dimensional matrix. The pseudocode is shown in algorithm 2.

The main idea of PCA is to map the N-dimensional features to the K-dimension, which is a new orthogonal feature, also known as the principal component, and is a reconstructed K-dimensional feature based on the original N-dimensional features, as shown in Formula 1,2,3,4,5.Formula 1 indicates that the original data X is arranged into a matrix with n rows and D columns, and then the matrix is zero-averaged. \(x_{i j}\) represents the data in row i and column j of matrix X. In Formula 2, c represents the covariance matrix of matrix X. Formula 3 expresses getting the eigenvalue and eigenvector of the covariance matrix c, eig() is the function of getting the eigenvalue and eigenvector, w indicates the obtained eigenvector, and b indicates the corresponding eigenvalue. In Formula 4, the eigenvectors are arranged into a matrix in rows from top to bottom according to the corresponding eigenvalues. The first k rows are taken to form the matrix p, where sort() is the sorting function and slect() is the selection function. Formula 5 represents the dataset Y obtained after dimension reduction.

Figure 3 is the overall structure of the MFVT model, which composed of two parts.

First part is the feature fusion network, which is composed of two layers of parallel convolution networks. The first layer is stacked with two convolution layers, the first convolution has a step of 1, the second convolution has a step of 2, and the size of the kernel is 3. The second layer consists of a convolutional layer and a pooling layer, where the convolutional layer has a kernel size of 3 and a step size of 1, and the pooling layer has a step size of 2. The padding size used in the two-layer convolution process is all 1. To make full use of the features extracted by convolution layer and pooling layer, the extracted features are fused to improve the extraction effect of features for small sample data.The whole calculation process of the feature fusion network is shown in Formula 6–16. Formula 6 represents the padding operation, and Formula 7 represents the size change of the output matrix of convolution processing after the padding operation. Under the premise that padding_n is equal to 1, the stride=1 keeps the output size unchanged, and the stride=2 halves the output size.Where \(X_O\) represents the matrix data obtained after the original flow data is processed by the CPR algorithm, because the convolution manipulation will change the size of the input matrix, in order to keep the matrix size unchanged it is necessary to perform the padding operation by Formula 7.X represents the matrix after the padding operation,\(X_{i j}\) represents the specific data value in the matrix. W is the width of the matrix, and H is the height.

Formulas 6, 8,9, 10 represent the entire calculation process of the first layer in the feature fusion network. Formulas 8 and 10 represent the convolution operation, V represents the convolution kernel matrix, \(v_{i j}\) represents the specific value in the convolution kernel matrix, and k represents the kernel sizes. \(X_{1}^{1}\) represents the eigenmatrix obtained after the first convolution operation. Since the stride in Formula 7 is 1, the output size remains unchanged. \(X_{1}^{2}\) represents the matrix obtained after the padding operation of \(X_{1}^{1}\), and \(X_{1}^{3}\) represents the eigenmatrix obtained after the second convolution, and the output size is halved because the stride in Formula 7 is 2.Formulas 6, 11, 12 represent the entire computational process of the second layer in the feature fusion network, where\(X_{2}^{1}\) denotes the feature matrix extracted after the convolution operation, the stride=1 does not change the output size, and \(X_{2}^{2}\) denotes the feature matrix obtained after the maximum pooling operation, which halves the size of the output feature matrix.Formula 13 shows the scale changes of the features extracted from the first and second layers of the feature fusion network. Formula 14 represents the specific process of fusing the first layer with the second layer features. The fusion refers to the summation of the number of channels, but the data must be kept consistent except for the number of channels. C represents the number of channels, C(1) represents the number of channels is 1, C(32) represents the number of channels is 32 and so on, \(X_f\) represents the features extracted by the feature fusion network.The second part is composed of the vision transformer architecture. To combine vision transformer architecture with feature fusion network, the structure of vision transformer is modified in this paper. The main methods used include feature embedding, learnable embedding, and transformer encoder.

For feature embedding, standard transformer accepts sequence of token embeddings as input. To process the feature \(X_f\) learned by the feature fusion network, we reconstructed \(X_f\) into a flattened 2D block sequence \(X_p\). Formula 15 is a specific variation of the formula, the same as NLP, will be added to the sequence of images in the token classification, the sequence of images is cut into multiple patches by a picture to get the number of patches where p indicates.Learnable embedding, a learnable embedding \(z_{0}^{0}=x_{class}\) is preset for the feature block embedding sequence, \(x_{class}\) denotes the category vector whose state/feature \(Z_{L}^{0}\) at the transformer encoder output is used as the feature representation y, as shown in Formula 21. Learnable embedding is randomly initialized at the beginning of training and obtained by training.

Transformer encoder consists of several blocks, each containing a multi-head attention block and a multi-layer perceptron (MLP) block, with normalization applied before each block and residual concatenation applied after each block. Figure 4 shows the structure of head attention. Formula 16, 17 show how to get the multi-head attention values by head attention. Where \(W_{i}^{Q}\),\(W_{i}^{K}\),\(W_{i}^{V}\) and \(W^O\) are all weight matrices.Finally, the embedding vectors that combine category vectors and feature block embedding can be input into transformer encoder. The encoder built up by blocks can extract data features for classification just like CNN. The whole calculation process is shown in Formula 18, 19, 20, 21.The feature embedding block \(X_{P}^{1} E\) and the category vector \(X_{class}\) form the embedding input vector \(Z_0\). Formula 19 adopts skip connection, where MAS represents multi-head attention operation, LN represents normalization operation, L represents repeatable times, and \(Z_{l}^{'}\) represents the lth output. Formula 20 adopts skip connection, MLP represents the multi-layer perceptron block, L represents repeatable times, and \(Z_l\) represents the lth output. y represents the feature representation.

In this paper, ablation experiments were conducted on the MFVT model and CPR data processing algorithm under the environment shown in Table 2.

In this paper, A series of ablation experiments were designed using both IDS 2012 and IDS 2017 datasets.

The IDS 2012 dataset contains a week of network activity including both normal and malicious activity, with three days consisting of all normal traffics and the remaining four days consisting of a large amount of normal traffics with a specific type of attack traffics. IDS 2012 dataset contains attack traffic including internal penetration, HTTP denial of service, distributed denial of service using IRC botnet, and brute force cracking of SSH [18].

The IDS 2017 data collection period lasts for five days from 9am on Monday, July 3, 2017, to 5pm on Friday, July 7, 2017, of which Mondays only include normal traffic. The attacks implemented included brute force FTP, brute force SSH, DoS, Heartbleed, web attack, infiltration, Botnet, and DDoS [17].

Figure 5a is a bar chart of the amount of various attack traffics contained in the IDS 2017 dataset, and Fig. 5b is a pie chart of the amount of various attack traffics contained in the IDS 2012 dataset. It is observed from the figures that both IDS 2017 and IDS 2012 datasets have serious data imbalance problems. The data volume of DDOS, Hulk, and PortScan attacks in the IDS 2017 dataset is significantly larger than that of other types of attacks. The data volume of Infiltrator attacks in the IDS 2012 dataset directly accounts for 55% of the dataset

Authoritative evaluation metrics must be used to judge the merits of a network anomaly traffic detection method. The effectiveness of the machine learning-based network anomaly traffic detection algorithm can be evaluated by the metrics shown in Formula 26, 25, 23, 22, 24 [48]. TP represents the positive sample predicted to be positive by the model, which can be called the accuracy rate judged to be true. TN represents the negative sample predicted to be negative by the model, which can be referred to as the percentage of correct judgments that are false. FP represents the negative sample predicted by the model to be positive, which can be referred to as the false alarm rate. FN represents the positive sample predicted to be negative by the model, which can be referred to as the underreporting rate [37].

In this paper, two datasets of IDS 2012 and IDS 2017 were used for ablation experiments. In addition, this paper also carried out an exploratory study on the impact of model optimization methods on MFVT model detection performance on IDS 2012 dataset.

In the MFVT model, the size of the kernels used in the convolutional neural network is \(3*3\), the segmentation size set in the vision transformer architecture is \(11*11\), the number of the head in the multi-head attention is 12, and the number of blocks in the encoder is 12. In the process of model training, the data input batch used in this paper is 256, the epoch of the training iteration is set to 100, and the stochastic gradient descent (SGD) optimizer is used to accelerate the network convergence. The momentum is fixed at 0.9, the learning rate is fixed at 3e-2, weight_decay Set to 0, the loss function uses CrossEntropyLoss. All ablation experiments and results will be described in detail below.