Ensemble machine learning approach for classification of IoT devices in smart home

The application of the IoT concept in different economic sectors is becoming a key factor for business improvement. According to [1], 92% of companies believe the IoT concept will be important for their business by the end of 2020. Consequently, the companies consider that security, privacy, costs, and regulatory issues pose the greatest challenges of implementing and applying the IoT concept. Research [2] conducted in 1,430 companies (small, medium, and large) points to a number of advantages seen by the vast majority (95%) of adopters of the IoT concept. In doing so, more than half (53%) confirm significant benefits of implementing the IoT concept in business, while 79% of those surveyed believe that by applying the IoT concept, they achieve positive results in different areas of work that they would not otherwise be able to achieve.

According to Gartner, the largest representation and application of the IoT concept, according to the number of IoT devices used until 2017, was in the area of smart building environments. After 2017, the smart home concept is the environment that brings together the largest number of IoT devices [3]. More precise insight into the representation of IoT devices by individual areas of application is provided by the research of the company IHS Markit [4]. It can be seen that the smart home concept has the largest number of installed IoT devices (822.6 million) compared to other areas of application. The annual growth rate (prediction by 2021) is 19.6%, which makes the smart home concept [5‐7], along with the industrial IoT concept (CAGR 23.4%), the fastest growing area of application of the IoT concept. The classification of IoT devices is essential for several reasons. Successfully identifying IoT devices in a particular scenario and environment can be vital in identifying illegitimate devices, unauthorized devices, unwanted devices, devices that do not behave as expected, and have the potential to cause a security incident within the system. Besides, useful device classification and identification of new and hitherto unseen devices can enable more efficient traffic management as well as network capacity required in the environments in which IoT devices exist [8, 9].

The rest of this paper is organized as follows: the second chapter deals with the current research, their shortcoming, and the positioning of our research according to previous findings. In the third chapter, data collection approach is explained, which includes laboratory environment establishment, raw network traffic collection, data preprocessing, and device class definition as key activities for further classification model development. The fourth chapter explains the classification model development as well as the ensemble supervised machine learning method used for that purpose. In the fifth chapter, the results from the developed model were analyzed and discussed. In the final chapter, the authors give their conclusion and further research direction.

According to the forecasts presented in [10], by the end of 2020, approximately 31 billion IoT devices will be globally used, and until 2025 there will be 75 billion IoT devices. At the same time, 41%, i.e. 12.86 billion IoT devices will be installed within a smart home (SH) [11]. IoT device limitations in general, and thus SHIoT (smart home IoT) devices, are described in the research [12]. Limitations include hardware limitations, requirements for high autonomy and low production cost, which reduces the possibility of implementing advanced protection methods and increases the risk of many threats shown in [13]. The traffic generated by SHIoT devices or MTC (Machine Type Communication) traffic differs from the traffic generated by conventional devices, HTC (Human Type Communication) traffic, which was shown by research [14]. Specific features of MTC traffic have been used to solve several problems in the communication network. Research [15] looks at the impact of MTC traffic on QoS during integration with HTC traffic in the LTE (Long-Term Evolution) communications network. Identification and classification of IoT devices in smart cities [5, 6] and campuses and in smart environments using MTC traffic features have been presented by research [16] and [17]. Research [18] seeks to identify new requirements and challenges in the design and management of a mobile communication network imposed by the generation of MTC traffic.

SHIoT traffic can be observed through network activity features such as traffic volume (sum of the total traffic received and total traffic transferred), traffic flow duration (time between first and last packet in traffic flow), and device inactivity time (the period in which the device has no active traffic flow). The network behavioral modeling is an often-used approach to address communication network challenges such as detecting illegitimate events based on traffic generated by devices on the network. In general, current approaches seek to identify traffic characteristics at the network packet level and the traffic flow level [19]. The analyzed research shows more frequent consideration and use of traffic features at the level of traffic flow than at the level of network packages. Likewise, the mentioned studies use the presented features to identify individual devices or their classification based on the semantic characteristics of the observed devices. [20]. Authors in [21] developed a tool for automatic extraction of packet-level signatures of IoT devices form the network traffic. They have extracted packet-level features of 18 smart home devices, which was used as a basis for development of classification model with a recall of 97%. Although research represents high results of the developed classification model, it remains unclear how the model will behave on the previously unseen devices. It should be trained again for every new device that comes on the market. Such an approach is not suitable considering the nature of the IoT concept. In research [22], the authors present the LSIF (Locality-Sensitive IoT Fingerprinting) approach for identification of IoT devices. The presented approach does not require feature extraction from the traffic. Although this approach has its benefits, it is lacking in performance such as precision (93%) and recall (90%). Also, this approach is focused on the identification of individual devices, which raises already mentioned shortcomings. The research presented in [23] used artificial neuron network for the classification. They developed a model that can identify nine known devices with approximately 99% accuracy. In research [24] the primary goal is to develop a model for network anomaly detection caused by IoT devices. For that purpose, the authors first developed a classification model for profiling of normal behavior. They used J48 machine learning method for model development with precsion, recall and F-measure, 96.2%, 96.8% and 96.9%, respectively. The developed model is actual only for nine devices they used in research because profiling was done for an individual device. The lack of research can be noticed in the number of used devices and insufficient generalization of the problem, where for every new device a new model needs to be learned, trained, and validated. In research [25], the authors use the decision trees and deep learning based methods for identification, classification, and anomaly detection of IoT devices. This research tries to use a more general approach to the classification of network traffic by using tree classes of traffic (actuation, sensing, video streaming). Such an approach is suitable for recognizing normal (expected) behavior of network traffic generated by IoT devices, and it is useful in resolving problems such as anomaly detection. Negative sides of this research are the number of used devices (7), the amount of network traffic (5 days), and the results of the developed classification model (93.5%).

According to the above, the possibility of developing an efficient classification model of IoT devices based on the characteristics of the generated traffic flows is set as a hypothesis of the current research. The research aims to develop a classification model based on an ensemble supervised machine learning method that will be able to assign IoT devices to predefined classes based on the values of their traffic flows. Current research in this domain is trying to identify the individual device. Such an approach is not suitable in the fast-evolving, heterogeneous, and dynamic environment such as IoT, where the number of new devices is rising exponentially. Because of the mentioned, the approach in this research brings novelty and gives the opportunity to recognize a class of new and unseen IoT devices based on its network traffic behavior. Our approach tends to generalize the identified problem and develop a solution that would be adjusted to the nature of the IoT environment. Accordingly, IoT devices need not be observed individually in solving a problem such as certain types of management of IoT devices, detecting network anomalies generated by IoT devices, or identifying unauthorized IoT devices in the network. For that purpose, a classification model is needed that would be able to assign previously unseen devices to generic behavior profile. This research, compared to previous ones gives contribution in a larger set of observed devices, longer period and larger amount of collected data, innovative approach in the classification of IoT devices, and better performance results of the developed classification model.

This research has been conducted in three phases with the activities shown in Fig. 1. In the first phase the research problem was identified and laboratory environment established. The dataset was formed from primary and secondary data sources. In the second research phase index C_u was extracted for each device and IoT device classes were defined. The collected data have been preprocessed which included feature engineering and data normalization (dealing with null and categorical values). In the final, third phase the dataset was balanced, and the classification model was developed. For the model development, the ensemble supervised machine learning method was used. The developed model performance was measured using standard validation measures for the classification models such as confusion matrix, accuracy, kappa coefficient, TPR (True Positive Ratio), FPR (False Positive Ratio), F-measure, ROC (Receiver Operating Characteristics) curve and other.

One of the crucial research activities was primary data collection for which the laboratory environment with SHIoT devices was established. SHIoT devices are supplied by authorized distributors and representatives of each device manufacturer. They are connected to the communication network as recommended by the manufacturer, and in no way are the devices modified at the software and hardware level. Therefore, it is assumed that the devices that are used to collect legitimate traffic in this research work are as designed and are in no way previously compromised in terms of security.

The network topology, as well as the characteristics of the smart home environment, can be seen in Fig. 1. The devices are connected, directly or indirectly, by Wi-Fi communication technology to the Fortinet AP 221C wireless access point, except for Phillips Hue, which communicates with the rest of the local network via Ethernet (IEEE 802.3) communication standard. Some devices, such as the Blink smart camera, Netatmo smart thermostat, and Philips Hue smart lighting fixtures, use an IoT hub with which they communicate wirelessly, but with ZigBee technology. The reason is the energy efficiency of the device since they use the battery as the power source of the end device, which gives them advantages in terms of mobility and independence of the device from electricity as a power source. The IoT hub is connected to Wi-Fi (or Ethernet in the case of Phillips Hue devices) technology with a wireless access point. Based on the above, a wireless access point has been determined as an adequate collection point for traffic generated by SHIoT devices. Due to the known modes of operation and characteristics of computers, and thus wireless Wi-Fi networks, traffic in the communication network cannot be collected directly. Several methods are available for traffic collection, often using physical port mirroring on the switch. This method is efficient in several studies, such as [15, 26‐28], which provides a basis for the application of the same method in conducting this research.

A software-hardware platform consisting of a Fortinet AP 221C wireless access point, a Cisco 2960 Catalyst 48 PoE switch (Power over Ethernet) and an HP Pavillion dm1 workstation (Microsoft Windows 10 10.0.17134 build 17,134, × 64 processor architecture, AMD E-350, 1600 MHz 2 cores, 4 GB RAM) has been set up to collect traffic by port mirroring with Wireshark software tool version 2.6.3 installed.

As shown in Fig. 2, port mirroring is configured for the physical communication ports (FA0 / 1 and FA0 / 3) of the switch to which the wireless access point and IoT hub for the Phillips Hue device are connected. These ports are configured as a source, which means that all traffic coming to or from these ports will be mirrored (mapped) to the destination communication port (FA0 / 2). A traffic collection workstation is connected to this port (Fig. 2).

In order to develop a multiclass classification model of SHIoT devices, the logitboost method was used. The method used belongs to the ensemble machine learning methods and is based on the statistical method of logistic regression. Ensembles combine several models, as shown in Fig. 7, with each model solving the original problem to obtain a better composite global model with better performance than using a single model [45].

Boosting belongs to a set of ensemble methods that can convert multiple "weak" classifiers (models that predict the target class depending on the values of the observed feature vectors) into "strong" classifiers. In general, a "weak" classifier is a model whose class prediction accuracy is slightly better than random guessing, while a strong classifier is characterized by near-ideal performance. Boosting methods have proven to be a suitable classification technique that provides excellent results in solving problems from different domains [46]. Given the classification problem that is being addressed and the proven effectiveness of the boosting group of machine learning methods, the logitboost method was used in this research.

Model development, testing, and validation were performed using the WEKA software tool with the support of MS Excel 2016 during the preparation of the dataset for model development. Because a total of 59 features were selected in the feature selection process using the information gain method, the number of features was gradually reduced during the model development when validation measures for each model were compared. This process aims to develop a model that will use the least possible number of independent features that will not significantly negatively affect its performance.

Each model was validated by k-fold cross-validation with k = 10. The principle of operation of the k-fold cross-validation for k = 5 is shown in Fig. 10. Cross-validation is a statistical method intended to assess the performance of machine learning models on new, unseen data. This method is used to assess the behavior of the model over data that was not used in the learning phase. In doing so, the model is applied k times iteratively over the dataset. In each iteration, the dataset is divided into k parts. One part of the set is used to validate the model, while the remaining k-1 parts of the set are combined into a subset for model learning.

Tables 8, 9, 10, 11 show the performance and results of validation measures for a total of five models (M1,…, M5) with a different number of independent features used (M1–59 features, M2–48 features, M3–33 features, M4–13 features and M5–8 features). Features were reduced to the lowest IG value (Table 7). The initial dataset was divided into a 70/30 ratio, where 70% of the examples in the set were used for model learning, while 30% were used for model testing. This division, along with 60/40 and 80/20, is common in the development of models based on machine learning methods [58].

The performance of a classification model based on machine learning needs to be expressed through several different measures, especially when the model is multiclass, given that each measure has advantages and limitations [59]. Accuracy is one of these measures that represents the share of accurately classified examples in the set of all examples according to expression (13) where TP (true positive examples), TN (true negative examples), FP (false positive examples) and FN (false negative examples).

where:

Table 8 shows that all models have approximately the same classification accuracy (≈ 99.8%). The drop in accuracy is only noticeable with the M5 model, which uses eight independent features. The accuracy of the classification is 99.71% or 336 misclassified examples. The table shows a slight decrease in the accuracy of the classification for the M4 model (99.7956%) compared to the M1 model (99.8075%) which uses all 59 features.

Kappa coefficient (κ) expresses the measure of the success of the observed model according to the ideal with the correction of random selection [60]. The values of the kappa coefficient range from [0,1] where κ = 0–0.2 is an extremely bad model, κ = 0.2–0.39 is a bad model, κ = 0.4–0.59 is a moderate model, κ = 0.6–0.79 a good model, κ = 0.8–0.9 a very good model and κ = 0.9–1 an excellent model. According to the scale shown and the results seen in Table 8, all models show excellent characteristics according to the kappa coefficient, whereas the M4 model shows a minimal deviation from the M1 model (0.0001) with a significant reduction in the independent features used.

The accuracy of the model in predicting SHIoT device classes according to the traffic flow characteristics is also given by the confusion matrix shown in Table 9. Confusion matrix represents the performance measure for machine learning classification models where output can be two or more classes, representing the basis for other performance measures. In the confusion matrix shown in Table 9, the relation between traffic flow class affiliation predicted by the developed model and actual class affiliation of observed traffic flow is visible. Accordingly, a high number of traffic flows whose class affiliation is accurately predicted in relation to the number of instances whose class affiliation is incorrectly predicted is observed.

Additional validation measures are expressed through sensitivity, i.e., the rate or frequency of TPR and the rate or frequency of FPR are shown in Table 10.

The true positive rate results represent accurately classified examples of a class in the set of all examples assigned to that class according to expression (14). The false positive example rate represents the ratio of misclassified class examples in the set of all examples assigned to that class to expression (15)

where:

TPR true positive rate;

where:

Table 10 shows that models M1 and M2 provide the best results according to the TPR measure for class C1, for class C2 all observed models provide the same results, while for classes C3 and C4 model M5 provides slightly worse results compared to the others. From the aspect of FPR measures, the M2 and M4 models provide better or equally good results compared to other models, with the M5 model providing the worst results.

Additional validation measures that show the quality of the classification model are the precision or positive prediction value (PPV) and the F-measure shown in Table 10, as well as ROC and PRC (Precision-Recall Curve) curves whose values are shown in Table 12.

The measure of precision is used to express the number of correctly classified examples in relation to the total number of examples belonging to that class according to expression (16).

where:

According to the values expressed in Table 11, it can be seen that for class C1 the best results are given by model M1 while the worst results are visible with model M5. For classes C2 and C4, equally good results are observed for all models with the exception of the M5 model for the C2 class, while for the C3 class, the M2 and M4 models provide the best results.

The F-measure or F1 score represents the harmonic mean of the precision measures and the TPR according to expression (17) [59]. According to [61], the harmonic mean is more intuitive than the classical arithmetic mean to calculate the mean of the ratio.

The calculated values of the F-measure shown in Table 11 indicate the M5 model as the worst observed from the aspect of classes C1 and C4 while the other models show approximately the same results.

Table 12 shows the values of ROC and PRC validation measures. The ROC curve, or AUROC (Area Under the ROC Curve), is one of the most important and most frequently used measures that show the quality of the classification model.

ROC is, although in this case, expressed in tabular form, a graphical representation of the relationship between the rate of true positive classifications (TPR) and specificity, i.e. the rate of true negative classifications (TNR = 1–FPR). An example of a graphical representation of the ROC curve is seen in Fig. 11 for the M4 model. The area under the curve, AUROC, is interpreted as the average TPR value for all TNR values in the interval [0,1]. The closer the AUROC is to the value of 1, the better the performance of the classification model. A lower value of 0.5 represents the performance of the model equal to random guessing [8]. The values shown in Table 12 indicate the excellent performance of all observed models, i.e. almost all AUROC values are very close to the value of 1.

An alternative measure to the ROC is the PRC (Precision-Recall Curve), which is often used in cases of unbalanced datasets, whereas in the ROC measure a significant change in the number of false positively classified examples may result in small change in the rate of false positively classified examples. Therefore, since PRC uses the ratio of PPV and TPR, i.e. focuses on positively classified examples (TP and FP), it can better demonstrate the impact of many negative examples on the model performance. Because the dataset was stratified in this research, the PRC measure gives almost the same results as the ROC measure for all the observed models.

According to the analysis of the results, the M4 model was selected as the optimal model of SHIoT device classification, considering that its performance according to all presented measures does not deviate significantly from other observed models (TPR 0–0.001, FPR 0–0.001, PPV 0–0.001, F1 0–0.001, ROC 0.0001–0.0003 and PRC 0–0.002) with a significant reduction in the independent features used. In the M4 model, a total of 13 independent features were used compared to the initial 59, which, according to the IG value, had some influence on the dependent feature. The independent features used are shown in Table 13.

As can be seen from the table, the most relevant features are information-related to the length of packets in the observed traffic flow (z11–total length of sent packets in traffic flow; z12–total length of received packets; z13–the maximum length of sent packets; z17–the maximum length of received packets, z19–mean value of received packet length, z46–maximum packet length, z47–mean packet length value, z49–packet length variation), then information on interarrival packet times in traffic flow (z25–maximum interarrival packet time in traffic flow; z30–the maximum time between two consecutive packets sent in traffic flow). Features that provide information on the segments in the traffic flow (z61–the average size of the received segment), as well as features that provide information on the amount of data transmitted in the sub-stream (z60–the amount of data sent in the sub-stream; z71–the amount of data received in the sub-stream), proved to be relevant (Table 14).

The research presented in this paper provides a new approach in observing the behavior of IoT devices based on the generated network traffic. The goal, which was achieved by this research, was to develop an effective model of IoT device classification in a smart home environment, which is based on the outgoing and incoming traffic ratio coefficient of variation as a measure of device behavior predictability. The basis of the research and achieving the defined goal is the scientific hypothesis that it is possible to define classes of IoT devices and develop an effective classification model based on supervised machine learning methods acknowledging traffic characteristics generated by IoT devices in a smart home environment. The scientific hypothesis was proved by defining four classes of devices based on the coefficient of variation ratio of received and sent traffic. The defined classes conditioned the development of the classification model of IoT devices.

The mentioned coefficient was named C_u index, which was chosen as a dependent feature used for the purpose of defining a total of four classes of SHIoT devices using the method of the coefficient of variation classification. Based on the defined classes of SHIoT devices, a multiclass classification model based on the boosting method of additive logistic regression as a machine learning method was developed, which according to all validation measures, shows high performance. The accuracy of SHIoT device recognition to one of the defined classes based on independent traffic flow features is 99.79%. Relevant independent traffic flow features used in the development of the model were selected using the information gain method.

The research proved that it is possible to assign, with high accuracy, new and unseen devices, and traffic flow that they generate into predefined classes with the application of boosting methods of machine learning. Besides, this approach responds to the needs of the newly created IoT environment in which the number of devices is growing exponentially, and it is not possible (or requires substantial resources) to know traffic profiles for each device, but it is sufficient to identify which class the device belongs to. This innovative approach has the potential to lay the foundations for many other activities and research in the IoT concept problem domain. The detection of anomalies in the communication network caused by IoT devices is one of the future research directions that will use findings and conclusions gathered in this research. Besides further research, developed model can have real-life applications as a software solution that can upgrade functionalities of the existing solutions for the device and network monitoring and management in an environment where many various IoT devices exist. This solution can help to monitor device groups that have similar communication patterns, manage their behavior in the network, plan future communication capacities for various device classes or similar activities. This kind of a solution is only usable as a support process for various other activities. That is why future research, as well as future real-life applications and service, will have a foundation in the results achieved in this research.