Characterization of threats in IoT from an MQTT protocol-oriented dataset

The number of home devices that have sensors connected to the Internet or a local network has increased exponentially. These devices are designed to program events, collect information, or offer remote control capacities, retrieving a large amount and variety of data. The interconnection of these devices is what we call the Internet of Things (IoT).

According to Cisco [16], the number of connected devices is expected to reach 500 billion by 2030, making security in IoT systems of paramount importance. Security in IoT systems has special characteristics due to the wide variety of different networks and protocols used in them [18]. The devices often have limited processing capacity to be efficient, and security in IoT devices is often ignored or treated as a late occurrence. Furthermore, the short time to market and reduced costs that drive the design and development of the device do not help to alleviate this problem [64]. Because of this, different types of known vulnerabilities can be found in commercial baby monitoring devices (see [56], for instance).

Attacks on IoT systems can affect their functionality, with alterations in system messages, alterations in the operation of actuators, or the theft of private information gathered by the sensors. The security of IoT devices can also affect the general security of the Internet. For example, in 2012, Carna botnet [49] revealed that there were more than 1.2 million open devices that allowed logins with empty or default credentials. In January 2014, an Internet-connected fridge was discovered as a part of a botnet sending over 750,000 spam e-mails [57]. One of the most significant recent attacks, the Mirai attack in September 2016 [36], used the special features and vulnerabilities of IoT devices, their low level of security, and the large number of them that are always connected to the Internet, to infect them and create a botnet which attacked the service provider, Dyn, with a distributed denial-of-service attack (DoS). This attack took down hundreds of websites, including Twitter, Netflix, Reddit, and GitHub for several hours.

The analysis of network traffic has been useful in detecting intrusions in traditional networks thanks to the use of intrusion detection systems (IDS). These systems are improved by adding new detection rules or automatic classifiers obtained using Machine Learning (ML) techniques, and are tested with datasets that contain network traffic with normal traffic and traffic under attack. On the other hand, the strategies to improve an IDS mentioned above require prior knowledge about how the attacks we want to detect perturb the network in terms of the variables allowed by the protocol that governs that network. That is to say, we need to characterize the attacks inside the features space allowed by the governing protocol. Such characterization is usually faced making use of feature selection algorithms. However, such algorithms need to be fed with a dataset that contains network traffic of both types, normal and with traces corresponding to situations under attack. From everything said so far, it follows that these types of datasets are of great importance [41] and the task of capturing the information generated in an IoT network becomes a critical step in analyzing intrusions in the IoT [67]. Nevertheless, there are almost no public datasets with network information on security-oriented IoT systems that capture attacks on the particular vulnerabilities of these systems. Most datasets gathered from IoT attacks contain information from sensors that are oriented to optimize the system or learn about trends (see, for instance, [2, 12, 37]).

The goal of this article is twofold: To obtain a dataset of the network traffic of an IoT system undergoing an attack, the protocol entitled Message Queue Telemetry Transport (MQTT), widely used in IoT [3], has been studied in depth. When its working has been verified and its vulnerabilities studied, three types of attack have been carried out in an environment that simulates a real situation to capture network traffic undergoing these attacks and to investigate whether is it possible or not to characterize these attacks in terms of the trace left by them on the network.

To address the characterization problem subject to the conditions imposed, the most suitable procedure seems to be the minimum-redundancy maximum-relevance (mRMR) algorithm [22]. However, this algorithm has two drawbacks. First, it requires the user to fix the number of features that the algorithm will output. Second, it does not take into account any predictive model which let us to measure the goodness of the selected features for a prediction task. Therefore, the characterization has been addressed through a hybrid (filter/wrapper) feature selection algorithm based on the idea behind the mRMR feature selection algorithm. The score function used for the filter process is the normalized mutual information function (NMIF), while the wrapper process is carried out using a model selection algorithm in which the models are trained and tested over the reduced subsets of features provided by the filter process. The algorithm provides, for each type of attack, a small subset of features with low correlation between themselves and high correlation with the class variable. As a by-product, we get classification models for the different attacks. The generated predictors show a high accuracy in the testing stage, so the selected features give a good characterization of the attacks considered in this work.

The advantage of using an information-theoretic measure as the NMIF function is that the features provided by the algorithm explain the underlying nature of the IoT attacks considered in this work by themselves. Therefore, the algorithm may be useful to get expert knowledge by applying it to less-known types of attacks. Also, it may be useful to define detection rules to improve an IDS.

The dataset, together with the feature selection algorithm, then performs a characterization to obtain the most relevant variables to detect the different attacks, which confirms the consistency of the dataset.

As regards IoT security, there are two different lines of research. On the one hand, we have those problems related to the security of the whole network and, on the other hand, those related to the security of an isolated device in the network. The problem that concerns us in this work is a mixture of both. More specifically, we ask whether is it possible or not to distinguish the kind of an attack that is taking place on a single device in an IoT network by enabling machine learning models trained with the network traffic data.

In general, datasets that contain network information on systems under attack are very useful for feed intrusion detection systems (IDS) [42]. The biggest trouble is finding IoT network dataset composed by regular and anomalous traffic.

A well-known dataset is KDDD99 [34] which gathers network traffic over the TCP protocol in a system in which different attacks, such as DoS, User to Root (U2R), Remote to Local (R2L) and Probing Attack, are made and tagged. With this dataset, studies have been conducted for the development of IDS for IoT environments that focus on denial of service attacks performed by botnets of infected IoT devices [45, 50]. Although KDD99 is still a valid dataset, it is not sufficient for more modern networks with new protocols for IoT environments [55], being useful datasets like the one proposed for IoT environments.

Another dataset to highlight is the AWID dataset [35], which gathers TCP frames of data from a WLAN network over which several attacks have been made on the 802.11 security mechanism (i.e. WEP, WPA, WPA2) such as ARP Injection or Dictionary Attack. They use machine learning techniques to detect attacks on Wi-Fi networks with work that focuses on IoT, such as Impersonation attacks [7, 51].

When using the dataset to detect anomalies, it is possible that not all of the features considered give relevant information, and that some of them contain false correlations that make detection difficult, which leads to a decrease in accuracy and an increase in computational complexity. Feature selection algorithms and dimensional reduction are used for the characterization of datasets with statistical methods, information theory and machine learning techniques to optimize IDS and reduce the IDS model’s complexity.

Regarding the KDD99 dataset, in [14] the authors perform a characterization of the dataset to detect some attacks to IoT systems showing, as a result, that the characterization process improves the accuracy of IDS systems. However, they are for more generic attacks on IoT systems. Many other studies have been carried out on the reduction of its features to obtain the most relevant ones for the detection of attacks, see [27] for instance. For the AWID dataset, in [62] the authors make a selection of optimal features with a classifier that uses Support Vector Machines (SVM) and redundant features are removed using PSO-based algorithms. In [4], the authors implement Deep-Feature Extraction and Selection (D-FES), which combines stacked feature extraction and weighted feature selection on the AWID benchmark data set. In [39], the authors deal with the unbalance issue in the AWID dataset. They perform a characterization of anomalous traffic that avoids this unbalance using algorithms such as Word2Vec, KMeans and SMOTE.

In [33], the authors provide a survey on the features selection and machine learning algorithms used to face the intrusion detection problem in traditional networks in the last years. Regarding deterministic algorithms for features selection, the authors show that most of the research in the area is supported on algorithms based on Mutual Information, Entropy, Correlation Coefficient, Chi-Square, Relief-F, and Gain Ratio. Furthermore, they show that Mutual Information and Gain Ratio based algorithms are by no means the most used methods for selecting features in Intrusion Detection problems on general networks.

Regarding the special case of IoT, there are significant works on security that have been carried out based on the analysis of network traffic using machine learning techniques.

In [23], for example, the authors analyze how machine learning can help in detecting the activity of an IMSI catcher in a mobile network. It puts the devices that are in the neighbourhood under a man-in-the-middle (MitM) attack by trying to be the preferred base station in terms of signal strength. In [26], the authors propose an anomaly detection scheme with feature selection using the boruta algorithm based on Random Forest (RF) classification technique with good results using for the detection of intrusions with the DARPA [1] dataset with network traffic. On the other hand, [46] propose an intrusion detection and mitigation framework (IoT-IDM) for the protection of a network of intelligent devices implemented in domestic environments. IoT-IDM supervises the network activities of the devices and investigates if there is any irregular activity. When an intrusion is detected, IoT-IDMt is able to block the intruder. In [60], the authors use a convolutional neural network model to create a multiclass classification model. The proposed model is then implemented using convolutional neural networks in one, two, and three dimensions. In the pre-processing phase, the authors use a model-based feature selection technique called RFE (Recursive Feature Elimination) to select 64 relevant features. In [61], a framework for detecting anomalies in IoT networks is described. Using conditional Generative Adversial Networks (GANs), the authors generate real-world distributions for a given feature set to face the problem of data imbalance. The performance of the GAN models in classification tasks are evaluated using a Feed Forward Neural Network and tested on two network-based anomaly detection datasets and five IoT network-based anomaly detection datasets. The authors do not filter the set of features provided by the protocol. In [43], the authors propose a Deep Learning (DL) based Network IDS trained using a public dataset containing MQTT attacks. Again, the authors do not filter the set of features provided by the protocol. In [63], the authors create a dataset with network traffic in a IoT system governed by the MQTT protocol and under certain types of attacks. They validate the dataset by training and testing some common machine learning models. In the pre-processing phase, the authors filter the full set of features. However, the process is not done automatically but by hand attending at certain criteria. The filter process ends up with 33 selected features and they do not specialize the process to the types of attacks. In [31], the authors evaluate the effectiveness of several machine learning models to detect MQTT-based attacks. The authors consider three abstraction of features, namely, packet-based (29 features), unidirectional flow (18 features), and bidirectional flow (18 features). The authors dropped certain features to avoid specific features influence before giving the list of used features. Apart from this, there is no feature selection process. The authors train and test machine learning models using the three type of features separately showing the importance of bidirectional features in the classification task. In [15], the authors described the implementation of a lightweight anomaly-based IDS for IoT networks, focusing on attacks on MQTT. They build a dataset that contains attacks to the network, and then train and test several common machine learning models on it. Regarding the pre-processing phase, the authors make use of a model-based feature selection algorithm, the SelectKBest method, to select 24 relevant features.

In Table 1, we give a summary of the datasets considered in the works mentioned above, used to investigate cybersecurity in IoT environments.

Despite the importance of entropy based methods (such as mutual information and information gain) for selecting features for anomalies detection in traditional networks [33], the analogous problem when dealing with IoT environments has not been faced with such methods yet. It is especially surprising taking into account the advantages of such methods, which have been proved theoretically and in practice. Furthermore, even the appearance of datasets with IoT abnormal traffic very recently, no of them cover the type of attacks considered in this work. This motivates the construction of the dataset.

Although the existing gap regarding the use of Mutual Information based algorithms for selecting and characterizing attacks taking place in IoT networks is enough to motivate this research, there exists a theoretical motivation, as said above, which is worth pointing out. In [11], the authors prove that to maximize the conditional likelihood of the training labels (under certain filter assumption) in a given machine learning problem (no matter the context of the problem), is equivalent to find the minimum set of features that minimizes the conditional Mutual Information \(I(X_{\overline{\theta }};Y|X_{\theta })\) (see [11, Sect. 3.2]). Here, \(\theta \) is a binary vector of the same length as the available set of features where the ith component is equal 1 if the corresponding feature is taken into account, and 0 otherwise. Furthermore, they prove that IAMB algorithm [59] is in fact a greedy iterative maximization of the conditional likelihood [11, Corollary 6], and that the Joint Mutual Information (JMI) criterion [59] provides the best tradeoff in terms of accuracy, stability, and flexibility with small data samples. Note that minimizing \(I(X_{\overline{\theta }};Y|X_{\theta })\) means, in certain sense, to find the most meaningful features avoiding features that are redundant, which is the basis of mRMR algorithm [21].

In this article, we give a feature selection algorithm based on the ideas behind mRMR algorithm, close to that of [32], in which the goal is to find a balance between the maximization of the accuracy of a (ensemble of) machine learning models and the maximization of the conditional likelihood of the training labels.

Due to the lack of IoT network environment datasets, we have designed and implemented a procedure for getting this kind of well-structured data. This section is devoted to describing this procedure. A real environment is created for data collection. The environment scale allows to collect all the traffic generated by the IoT environment Local Area Network (LAN). The environment also collects generic network traffic when for example is browsing the Internet, obtaining all the real traffic generated in the network, in not only the specific traffic of the MQTT protocol.

This section details the different techniques used for accomplishing the objectives of this research, together with the datasets extracted that will be characterized.

To apply the feature selection algorithm described above we will have to adjust the parameters on which both the filter and the wrapper processes depend.

In this section, we describe the results and reflections based on the aforementioned experiments.

The algorithm finishes after 25 iterations. The best models (with the tuned hyper-parameters) together with their accuracies and the features selected by the filter routine of the algorithm are listed in Tables 4, 5 and 6 for the datasets DoS, man-in-the-middle and intrusion respectively. The models that appear in bold are the most accurate for each data set and, therefore, the output of the algorithm applied to each data set is precisely the set of features appearing in the selected features box corresponding to these models. The summary of the outputs of the feature selection algorithm is given in Table 7.

By analyzing the features that the algorithm has selected (see Table 8 for a brief description of them), we can say how the IoT attacks are reflected in the network traffic and we can explain their underlying nature. This implies that these attacks have been characterized successfully and therefore, the data set is consistent and the feature selection algorithm carries out the selection task well.

After analyzing the features chosen by the feature selection algorithm, we show the existence of a strong relationship to the nature of the different attacks to the MQTT protocol taken into account in this work. This is valuable because it allows experts to know what variables should be taken into account to define rules for an IDS to be able to detect specific malicious traffic.

Regarding the possibility to enrich or improve an IDS with machine learning techniques, we can look at the classifiers obtained as a by-product of the feature selection algorithm run over the constructed dataset. For a DoS attack, it is enough to consider 5 features to be able to train a DT classifier with an accuracy of 0.99377. In case of a MITM attack, we just need also 5 features to train a RF classifier with an accuracy of 0.96094. Finally, we can train a GB classifier considering only 8 features with an accuracy of 0.95384. This is summarized in Table 9.

Concerning the comparison of the results of this work with those found in the literature, it is worth pointing out the difficulty to establish a statistical meaningful contrast due to the absence of other analyses on this new dataset as well as to the nonappearance of other works studying and characterizing exactly the same type of attacks. However, we find that one of the most studied type of attack is DoS [15, 61, 63], which is also considered in this work. In Table 10 we include the existing literature about the construction of IoT security datasets and/or with a machine learning analyses on them. For each reference, we specify the datasets on which the analyses are done, the attacks considered, the number of selected features (complexity of characterization) to perform the analyses and, if possible, the accuracy obtained by the predictors when trying to detect traffic under a DoS attack. If the authors construct predictors over different datasets and with different machine learning models, we include the best accuracy. We also include the corresponding data obtained in this work. As the table shows, our dataset together with the feature selection algorithm provides not only a good characterization of malicious traffic but also a much simpler one. Furthermore, in case of DoS attacks, the simplicity of the characterization does not compromise the classification accuracy obtained by machine learning methods.

It is important to note that it was not the goal of this work to construct classifiers for each type of attack, so this point deserves a deeper and sharper analysis by considering different machine learning techniques.

IoT devices are currently growing exponentially. New security challenges must be addressed due to the special nature of these types of devices, mainly their low cost, which use to lead to security problems. Data Mining and machine learning techniques can be a good way of understanding the behavior of IoT environment vector attacks. However, these techniques require datasets describing certain events and there is a lack of this type of information available to the cybersecurity research community.

This work has made two main contributions. First, the development of a procedure for collecting datasets in IoT network environments that works successfully. Second, the design of a hybrid feature selection algorithm that is able to characterize the previous dataset, that contains a large number of network traffic features. Thus, the most important features of Man-in-the-middle, DoS and Intrusion attacks over MQTT IoT protocol have been identified.

The wrapper routine of the hybrid algorithm uses six machine learning classification models. This algorithm returns the most important features together with the aforementioned models trained on the filtered dataset. The high accuracy values obtained by these estimators validate our approach.

Taking these contributions into account, we propose to develop machine learning classifiers trained on this dataset with the selected features to feed IDS’s in a way that can detect attacks on the MQTT protocol in an IoT system.