ADS-B classification using multivariate long short-term memory–fully convolutional networks and data reduction techniques

Pattern-of-life (POL) modeling is a research area with many techniques and best practices [14‐18]. Military and defense personnel have an interest in POL modeling that includes more than modeling human day-to-day activities. For example, unattributed data from aircraft sensors, such as those collected from Air Traffic Control’s (ATC) primary radar, can allow inferences to be made about the transmitting aircraft with some analysis. ATC’s primary radar collects kinematic information such as location and airspeed, but is unable to obtain an aircraft’s identification without the aircraft providing it via its transponder. With this basic kinematic aircraft data, models can predict information such as aircraft model or engine type without it being directly stated in the original dataset. The benefit of ADS-B data is that these features are present in the dataset and can be used as truth data to build models for datasets that do not have the truth data.

Since this type of processing can be resource intensive, it can be difficult or, in some cases, impossible to train a deep learning model when dealing with limited computing resources. The amount of computing resources required to train a model is heavily influenced by the size of the training data. For this reason, it is important to understand how to best utilize the available resources by minimizing the data used to train the model. There are two ways to minimize the data: limit the number of features or reduce the number of training samples. In this research, using aircraft kinematic data, we examine the impact of varying these factors when predicting engine type. Since reducing the training data will inevitably reduce the accuracy of the resulting model, for the purpose of this paper, we define an acceptable model as one that predicts within 10% of the previous baseline research results of 89.2% accuracy [23]. Therefore, models that can achieve at least a 79.2% accuracy will be considered ‘acceptable.’

The research contribution of this paper can be summed up in the following points:

The rest of this paper is organized as follows: An exhaustive literature review and background information on ADS-B is provided in section two. In the third section, the methodology and process used to develop and evaluate each model is discussed. The fourth section presents the results. The conclusion is provided in the fifth section.

Automatic dependent surveillance-broadcast (ADS-B) is an aircraft sensor used throughout many regions of the world. Several researchers have utilized ADS-B data to identify flight patterns [8, 11, 24, 25], improve aircraft operations [26, 27], increase ADS-B transmission security [28, 29] and classify targets [7]. The transmissions are openly broadcast via an ADS-B Out transponder at 1090 MHz in many countries worldwide and at both 1090 MHz and 978 MHz within the USA. ADS-B data is received and collected via various methods to include other aircraft, ground stations, and satellites. Figure 1 depicts an ADS-B communications network.

Ground stations consist of sites where antennas collect ADS-B transmissions from passing aircraft. The ground stations are typically managed by commercial or government organizations, but hobbyists also collect the broadcasts. In the case of hobbyists, the aircraft transmissions are collected on a nearby device and transferred to one or multiple organizations that host the data for public use (e.g., the ADS-B Exchange [22]).

The ADS-B Exchange and similar services save the incoming data to their servers in two- to five-second intervals as a JavaScript Object Notation (JSON) file. Per the USA’s DO-260B standard and Europe’s ED-102A standard, each broadcast contains up to 70 features about the transmitting aircraft including the International Civil Aviation Organization (ICAO) identifier, altitude, airspeed, vertical speed, directional heading, position time, latitude, longitude, and ground status [22, 30].

Table 2 provides details on the countries that have an ADS-B mandate and the date when the mandate began or is projected to start. The chart shows that most countries, including the USA, Australia, the European Union Aviation Safety Agency (EASA), and many East Asia and Pacific island countries, began their mandate on or before 2020 which forces aircraft that fly within those regions to install the appropriate ADS-B Out equipment.

Within the USA, the mandate is not required for low flying aircraft and aircraft operating out of rural airports. Table 3 explains the US requirements in more detail. Within the USA, ADS-B Out is required for aircraft flying above 10,000 feet MSL and aircraft operating in locations near airports classified by Class B or Class C airspace.

Artificial neural networks (ANNs) have been shown to be successful at classifying a variety of datatypes. Many ANN variations have been developed over the years to classify the infinite number of datatypes researchers encounter. Long short-term memory (LSTM) networks were developed to classify time series data and convolutional neural networks were developed to classify images. Combining these two networks, the multivariate long short-term memory–fully convolutional network (MLSTM–FCN) is shown to improve upon previous methods to classify time series data [31]. The algorithm, developed by Karim et al., combined fully convolutional networks (FCN), LSTM networks, and squeeze-and-excite blocks. FCNs were utilized to allow for the CNN benefit of class action maps without the requirement of extensive hyperparameter preprocessing that is normally required by a CNN. LSTMs were selected to detect the importance of sequences of observations in time series data. Squeeze-and-excite blocks were added to the algorithm to improve the classification power of multivariate datasets by ensuring feature maps have a similar impact to subsequent layers. In their study, Karim et al. tested four variations of the algorithm against 35 different datasets to include voice, human signal monitoring, and other time series data. Their algorithms outperformed the current state-of-the-art algorithm in 27 out of the 35 datasets [31]. Additionally, a variation of the algorithm, the multivariate attention long short-term memory–fully convolutional network (MALSTM–FCN), predicted aircraft engine type using ADS-B data [19]. This research will be further discussed in the next section.

While LSTMs and FCNs are well established, the use of squeeze-and-excite blocks is a much newer technique. To understand how they improve the MLSTM–FCN, it’s important to understand how they work and what they provide. Squeeze-and-excite blocks were introduced by Hu et al. in 2018 to improve the representational capacity of a neural network [34]. They are used with convolutional networks to help model the dependencies between the channels. As the name implies, it consists of a squeeze mechanism followed by an excitation mechanism. In the squeeze step, the classifier uses global average pooling to aggregate the spatial information of each channel. Using the example of an image, where an image has dimensions of \(H\times W\times C\) for height, width, and (normally 3) channels, the squeeze step would pass the image through a global average pooling operator where it would become a shape of \((1\times 1\times C)\). Then, the excite step uses a gating mechanism to capture channel-wise dependencies [32]. The excitation step uses a multi-layer neural network with one hidden layer. The input and output layer are the same shape \((1\times 1\times C)\), but the hidden layer reduces the space by a reduction factor, r, making the number of neurons in the hidden layer C/r. Karim et al. and Hu et al. use a reduction factor of 16 [32, 34]. This \((1\times 1\times C)\) value is multiplied element-wise with the original \((H\times W\times C)\) input. The graphical representation of the squeeze-and-excite block developed by Hu et al. can be seen in Fig. 2.

Using the squeeze-and-excite block in conjunction with an LSTM and FCN, the MLSTM–FCN was developed to create a model for time series classification. Figure 3 shows the architecture Karim et al. designed for their research. This algorithm is used to develop all of the models in this research.

Multiple researchers have focused on using the ADS-B dataset to predict engine type with ADS-B kinematic data [8, 19]. One of the limitations with analysis on this dataset is that while the models have very few problems determining if the aircraft is a jet, they tend to have trouble predicting the difference between turboprop and piston engines. The reason for this is twofold. First, the dataset is heavily imbalanced toward jet engines which causes the data to have fewer samples of turboprop and piston engines from which to learn. Second, the kinematic differences between piston and turboprop engines are minimal.

As a method to remedy this problem, Basrawi et al. developed a Dual-Stage Deep Engine Classifier (DSDEC) [19]. Using the MALSTM–FCN algorithm as a basis for the model, the DSDEC algorithm employs a unique 2-stage approach. When using the model to make predictions, the first stage predicts if the aircraft is a jet or not a jet. Then, the ‘not jet’ predictions are fed to the second stage. The second stage predicts if the aircraft has a piston or turboprop engine. The results from both the first and second stage are combined to provide an engine prediction for each observation. Figure 4 portrays the architecture that is used.

Basrawi et al. are the first to classify aircraft engine type using ADS-B data with a deep learning model. Using the DSDEC method, researchers were able to identify jets with 98.4% accuracy, turboprop aircraft with 79.2% accuracy, and piston engine aircraft with 89.9% accuracy. However, the DSDEC method would still confuse turboprop aircraft as piston engine aircraft 17.9% of the time [19]. Table 4 shows the results achieved by Basrawi et al. compared to a support vector machine (SVM) and random forest (RF) as baseline from previous research[8].

During the experiments, static time steps were used and each time step was separated by two seconds. 300 time steps would be equivalent to 600 seconds or 10 minutes of flight time. Similarly, 100 time steps would be 200 seconds or 3 minutes and 20 seconds of flight time. While Basrawi et al. indicated that more research would be needed to determine the influence of the time step size, their results point to the possibility that longer flight observations improve the model’s predictive power. In fact, while the DSDEC algorithm was used against 21 different models to determine which hyperparameters led to the highest accuracy, none of the 100 time step models performed as well as the 300 time step examples.

Determining the exact dataset size needed to build an artificial neural network (ANN) classifier is an open research problem that may never have a complete solution due to the infinite number of ANN combinations. However, experts suggest several guidelines to ensure enough data is available to train a model with sufficient accuracy value [35‐39]. Guidelines can be broken into three important characteristics of the dataset: the number of prediction classes, the number of features, and the number of weights in the network. The following guidelines sum up the results from past research endeavors:

The information provided in this section shows that ADS-B sensor data is a useful resource when trying to understand POL modeling with aircraft. Previous research has used this data to predict aircraft characteristics [7, 8, 19]. When predicting engine type, researchers found that the MLSTM–FCN was able to train a model that achieved an overall accuracy of 89.2% [19]. This paper uses the information gleaned from research done with the MLSTM–FCN [32] and DSDEC [19] to learn how to minimize the input data size while maintaining a similar accuracy and loss.

The dataset used in this research is smaller than the one used by Basrawi et al. [19]. In this research, the training data from 1 December 2020 has 4,110 tracks/tensors, and the evaluation data from 16 November 2020 has 2,487 tracks/tensors. In the experiment completed by Basrawi et al., the training data ranged from 1 to 8 December 2020 consisting of 7,749 tracks/tensors. The evaluation data was also from 16 November 2020, but consisted of 4,158 tracks. This amounts to approximately half of the tracks for training and evaluation in this experiment. It is assumed that the effects of reducing the data size would be proportional regardless if one week or one day was used for training.

Although researchers have shown that the integrity of ADS-B sensor data is vulnerable to a variety of cyber attacks [41, 42], the dataset is assumed accurate for the purposes of this paper. While vulnerable, cyber attacks against ADS-B are not a common occurrence. Air traffic control-related attacks occur only a few times each year [43]. We consider this to be an acceptable assumption since the number of occurrences of message injects and other ADS-B attacks is low in comparison to the millions of flights that occur each year.

The process for converting raw ADS-B data into a model that can predict engine type can be broken down into five steps that will be explained in the subsequent paragraphs Fig. 5.

Table 9 outlines the performance across all phases. Figures 6 and 7 represent the data from Table 9. The line colors represent the input feature size and the color’s shade represents the learning rate. The 0.001 learning rate is darker shade and the 0.01 learning rate is the lighter shade. Blue represents the smallest input feature size, green the medium input feature size, and orange the full input feature set. As can be seen, the limited feature set has the best accuracy and loss with the medium and full feature set having very irregular performance. While not shown, recall and precision follow similar trends.

Model 1 performs the best. It has an 89.4% overall accuracy which is on par with the 89.2% accuracy achieved by Basrawi et al. [23], but with half the sample size. The confusion matrix for model 1 can be seen in Figure 8. The results for individual engine type are also similar to those collected by Basrawi et al. For comparison, they found that jet engines were predicted correctly 98.4% of the time, turboprop 79.2%, and piston 89.9% [23]. In this study, jets are accurately predicted 97.2% of the time, turboprops are 79.1% and pistons are 92.0%.

The results show that the limited feature dataset outperforms the medium and full feature sets. The larger feature sets never converge. This phenomenon is due to noise and overfitting to the training data. Experiment two is able to further analyze the lack of convergence. However, using the three feature sets created for this experiment, it is shown that altitude, airspeed, vertical speed, and location provide sufficient information about aircraft to differentiate engine type in most cases. The additional features do not provide any information that help the model learn more about aircraft engines. Instead, it makes the data more convoluted. This can be shown with the training history as seen in Figures 9,  10, and 11. These figures represent the training accuracy after each epoch where the feature set is varied between the figures, but learning rate and dataset size remain constant. The three images are from the models that had a 0.001 learning rate and used the full dataset (i.e., models 1, 3, and 5). While the training accuracy continues to increase, the validation accuracy does not. This indicates that the model is no longer improving and is instead overfitting to the training data. While only a subset of the models are shown in this manner, other models follow a similar pattern with the limited feature set performing the best in all cases.

The results from the limited feature set show that more data improves the classification power of the model. However, the reduction in accuracy is gradual until the one-eighth size dataset. Based on our previous definition of an ‘acceptable’ model, the 1/4 size dataset meets the minimum accuracy with 83.9%. For this reason, if computational power was limited, someone could drop the number of inputs to 1/4 of the full dataset and still be guaranteed to have a reasonable prediction accuracy rate.

The goal for experiment two is to find the subset of features that best predict engine type from the full size dataset. All possible feature combinations from 8 available features result in the creation of 255 models. The top 20 results from the 255 models are presented in Table 10. Airspeed, pressure, and vertical airspeed create the best-performing dataset. This result is unexpected since it does not include altitude. Jets fly at a different altitude than piston or turboprop engines. For that reason, it would seem like altitude would be an important feature. A possible reason for this might be that the observations only include the first 10 minutes after takeoff. Jet engines would not have the time to reach cruising altitude until the end of the 10 minute time frame.

Additionally, the results from experiment show that a combination of all of the features except for ‘PosTime’ provide the third best feature indicator for accuracy. However, not only is ‘PosTime’ not included in the 3rd best combination, but it is also interesting to note that all of the worst-performing models contain ‘PosTime.’ Table 11 shows the 10 worst-performing models. It is not until the 43rd worst performer (the model with just the ‘Trak’ feature with a 48.9% accuracy rate) that the ‘PosTime’ feature is not present.

While the goal of this study is to determine how to minimize the input data to make best use of low computational resources, military operations tend to have access to a vast number of sensors. Combining other related sensors to the ADS-B data could improve results. Other potentially useful sensors include weather, radar, and image data. Researchers should look at the effects of combining these sources to determine if they improve classification accuracy.

The raw ADS-B data represents the ‘PosTime’ feature as the number of milliseconds since Epoch (1970). During the processing of the data, ‘PosTime’ is modified to more closely compare to previous work with engine type prediction. Instead of milliseconds since Epoch, it represents the amount of milliseconds since takeoff. When creating a model with an LSTM, this is not the best use of the time feature. Keeping track of time of day, day of the week, or even just the date would allow the model to better learn aircraft schedules. Future work should look at modifying this feature into something that would improve the classifier.

This research uses only the first 10 minutes of flight to train each model. The first 10 minutes of flight consists of the takeoff phase and sometimes part of the cruising phase. Incorporating other phases, such as cruising and landing, could help to further separate the differences between engine types. The biggest benefit of using other phases is that jets, turboprops, and pistons have different characteristics when they get to the cruising phase. Jets fly at much higher altitudes during the cruising phase than other engine types and turboprop engines can reach greater speeds easier at higher altitudes than piston engines [48].