Detection of anomalies in cycling behavior with convolutional neural network and deep learning

Literature is extensive about safety assessment using observational studies, but in comparison a limited number of studies are applied to bicyclists [2]. In the InDeV project [38], the Safe VRU app was developed for self-reporting of accidents and near-accidents and has been used by more than 400 participants [37]. The target of the UDRIVE project [49] was to identify factors in CSEs involving a bicyclist; CSEs were identified manually and correlated to the features of the infrastructure [52]. In the project BIKEALYZE, data was collected by a mobile eye tracking, a GPS-based motion data acquisition complemented with acceleration and steering direction data; CSEs (e.g. collision avoidance, way-giving violations, abrupt braking, abrupt turnout) have been identified by video-based analysis and elicitation interviews [13].

In several studies participants had an active role to indicate any CSE they experienced via a push-button on the vehicle [60] or in a smartphone app [37] or by online questionnaires [24]. A study in Sweden [16] collected movement data of 20 bicyclists with an Inertial Measurement Unit (IMU) and GPS installed on an instrumented bicycle and analyzed cycling kinematics. Longitudinal and lateral accelerations have been considered relevant for analyzing cycling behavior. Another study showed that GPS data must be collected at least with a 1 Hz frequency to provide suitable speed profiles and to detect hard breaking of cyclists [41], while vertical accelerations acquired at least at 50 Hz by accelerometer sensors are required to analyze cyclist comfort and safety due to pavement unevenness [8]. Overtaking behaviour of motorized vehicles by measuring the lateral distance between the bike and passing vehicle and a statistical model was developed to predict the probability of an unsafe critical maneuver [40] and cyclists’ safety perception [48]. In [10], authors developed an algorithm to detect a cyclist downfall by combining acceleration and rotation thresholds. Strauss & Miranda-Moreno [55] proposed a procedure to use cyclist GPS data taken by a smartphone to calculate decelerations and correlate thresholds with the number of injuries. Despite the promising results, they concluded that more granular data and validation work needs to be done to improve the reliability of the correlation. Perceived risk resulted consistently with frequency of CSEs in bicycle paths classified by a panel of experts analyzing video recording and speed and heading GPS data [9]. All these research mainly rely on the identification of safety–critical events via self-reporting, manual review of video footage pre-selected thresholds and statistical methods to analyze data.

An extensive review [5] reported that the existing solutions for trajectory outlier detection were “algorithm based” (e.g. distance-based; density-based; pattern mining–based) with emerging machine learning–based schemes that learn the outlier detection from the training trajectories to identify anomalies in the newly inserted trajectories. Moreover, the research focused more on vehicle mobility [50, 59] and not to micro-mobility, such as bicycle, which also suffer of a “digital divide” when compared to the increasing opportunities for data collection through connected and automated vehicles. More specifically, no studies are reported for cycling trajectories [5].

Recently, deep learning and Convolutional Neural Networks (CNN) have been applied in road safety studies [25, 30, 31, 39] and driving style analysis [6, 15]. The convolutional autoencoders (CAE) allowed the extraction of valuable information from large quantities of complex and heterogeneous data, showed fast convergence due to the convolutional layers, and provided better performance with multi-dimensional data compression and feature learning making the procedure well suitable for managing the mobility data characterized by high volume, variability and velocity (i.e. big Data) [36].

Dong et al. [15] made the first attempt of adopting a deep neural architecture, based on Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), to extract features on the driving style directly from GPS data. More recently, Bichicchi et al. [6] applied an unsupervised Denoising Stacked Autoencoder (SDAE) to provide output layers from kinematic measures tracked with an in-vehicle 10 Hz GPS device. The RGB colors of the outcomes were associated with different path geometries encountered during the driving.

When applied to cyclist mobility, deep learning and CNN have been used in the bike-sharing prediction modeling, because the use of shared bicycles is susceptible to time dependence and external factors [44], such as weather [4, 47], bike rebalancing and land use characteristics. In [14], authors applied the Self Organizing Map artificial neural network to identify atypical trajectories from video sequences at fixed locations. More recently in [1], authors used video records from fixed cameras and trajectory data extracted by means of computer vision algorithms and Advanced Artificial-Intelligent (AI) techniques to model cyclists’ behavior and their interactions with pedestrians in a shared space.

The limitations of the existing works for classification of abnormal cycling behavior are summarized as follows: observational studies applied on bicyclist safety mainly relay on traffic conflict techniques applied to video tracking from fixed positions. Few studies used trajectory data to identify SCE, but by self-reporting or handled classifications. In [33], authors uses text mining analytics and an Artificial Neural Network (ANN) to extract information from near-miss and collision event descriptions, acquired from BikeMaps.org, a global tool for mapping collision and near-miss events. Deep learning is becoming widely applied to transport and road safety studies, but application to cycling are mainly focused on mobility choice. Results from previous studies about CNN for anomaly detections or modeling of motorized driver behavior, cannot directly be transferred to cycling because of its specific kinematic features and limited availability of advanced equipment for data collection as in standard naturalistic studies.

To the best of our knowledge, this is the first work extending the use of deep learning CNN to extract features of the riding style of bicyclists from GPS data and to detect anomaly events in cycling behavior.

Dataset preparation includes both collection and treatment of GPS data to extract features optimized to train the CNN model.

In this section, we propose a methodology that exploits a convolutional autoencoder for anomaly cycling detection in time series. Therefore, we will first provide CNN background and then, discussion on convolutional autoencoders (Sect. 3.2.1). Then, we will present the proposed methodology for anomaly detection in the application scenario (Sect. 3.2.2).

In a binary classification, ‘’Positive’’ and ‘’Negative’’ assignments refer to the classifier’s prediction, and the terms ‘’True’’ and ‘’False’’ refer to whether that prediction corresponds to the real observation. Given these definitions, the confusion matrix (CM) describes the performance of the classification model as shown in Table 2.

CM is useful for calculating two metrics of classification performance called Recall and Precision. Precision (P) measures the rate of true positive (TP) over the total predicted positive (TP + FP). The Recall (R) computes model’s ability to detect TP over the total number of real positives (TP + FN). For our classification with unbalanced data due to the small number of real Positive, the F-measure (F_β) is an effective quantitative metric to select the model setting which minimizes the errors [23]. F_β score, in Eq. (11), is the weighted harmonic mean of precision and recall, that ranges between 0 and 1.

As commonly used to emphasize Precision against Recall, we applied a weight β = 2, because we are more interested in limiting FN (i.e., missing detection of CSEs) rather than FP.

Our event detection criteria are illustrated in (Fig. 7) and defined as follows. Considering the observed time extension of a real CSE in the range of 0.8–3.1 s with an average of 1.4 s and the high time variability of the kinematic parameters in the cyclist riding, events with MAE > threshold of less than one second were not classified as “CNN positive” and detection sequences within 5 s have been classified as one CNN positives.

The model performance measured by R, P and F₂ is affected by several factors related to the data and CNN setting and varies by changing the classification threshold. Therefore, to evaluate the results, two major comparisons are carried out: first comparison case is carried out by using various model settings and features, while other comparison case is done between BeST-DAD and alternative detection approaches like Principal Component Analysis (PCA) and breaking acceleration threshold.

To maximize the performance of the model and to learn new insights about cycling behavior modeling, model setting and features relate to the following comparisons: CNN setting by changing (A) Training and testing, (B) Thresholds and (C) Size of the time windows; input dataset by varying (D) Use of SGF filter and (E) Use of only speed related variables: speed and longitudinal acceleration or (F) Use of only heading related variables: heading and Heading rate.

Furthermore, two scenarios have been selected for training and testing of the model:

(A1) Training and Testing CNN for each user, by using 80% of data for training and 20% of data for testing. The average training time in scenario A1 is 30.142 s for each user.

(A2) Training with the data collected by considering the entire dataset related to only one user and testing with 100% of the data from all the other users. The average training time in scenario A2 is 26.642 s at all. Results and values of Recall, Precision and F₂ are reported in Table 3.

For the first scenario (A1), where each user adopts its own model, we reached the absolute best performance with an average F₂ = 0.77. In the second scenario (A2), the model is trained with the full dataset data of one-by-one user and tested with 100% of data from the other users with an average F₂ = 0.72. Anyway, training with ID-4 as reference user and testing with all other users gave the best F₂ score of 0.77. While comparing the two different training approaches, we applied the best CNN settings in Table 3, with SGF, 40 TS-window, and all input parameters.

Results in Table 3 for scenario A1 show that training tailored CNN models for each user returned the best results. Anyway, in practical application, scenario A1 means the need to train BeST-DAD model for each user sharing his/her cycling data. While in scenario A2, transferring the model trained on one user returned slightly worse performances, but an overall average F₂ comparable with the previous scenario, as well.

The previous result illustrates the reason of “ID-4” selection as 100% training dataset for the further comparisons, given the good performance and availability of a larger dataset for testing purposes (i.e., 100% of data for user ID-1, …, ID-9) by using the full set of 41 real-positives for testing.

Results for the other validation scenarios (B, C, D, E, F) are presented in Table 4. First, we tested CNN setting for different thresholds (T) and time window sizes (TW) (scenarios B, C). Results in Table 4 confirm the best performance when T = 0.18 and TW = 40. Best values of TW and T also have meaning in explaining the cycling behavior. MAE = 0.18 is equal to the 88^th percentile of the overall Loss values, while MAE = 0.21 is the 93^rd percentile, showing that anomaly cycling behaviors for evasive maneuvers are quite rare (12% of the cycling time), but not exceptional events in the riding path especially in shared lanes as will be highlighted in Sect. 5.

Without the application of the SGF filter the high-frequency time-variability of the cycling data returned many CNN-positive with high Recall, but also many FPs returning a very low Precision (P = 0.36). It is worth noting that merging speed and heading parameters had the most important impact in improving the model performance. Results showed that without including heading derived parameters (i.e., HR, TA, CA), the model performance decreases significantly by missing several detections with more FNs and FPs (low R and P in Table 4). That is expected for cyclists, rather than other road users, because they apply both braking and swerving as evasive maneuver. Analyzing results, when Precision and Recall are compared it is noteworthy that Recall is always higher than Precision. This is significant because, in our application scenario missing CSE (FN) is of higher concern than False Positive. Moreover, FP may not be wrong detection of cycling anomalies, but often have been identified as changes of cycling behavior related to other events not classified as CSE (e.g., hard braking at traffic lights, avoiding pavement bumps, potholes etc.).

Finally, to further evaluate the effectiveness of deep learning in detecting CSEs, the performance metrics have been calculated also by applying a robust PCA statistic and the traditional empirical approach based on breaking acceleration threshold [56]. Results in Table 4 confirm the higher performance of deep learning for event classification.

In order to show the practical results of BeST-DAD for risk assessment and ranking, the procedure was applied to the trajectory data and anomaly detections located with the GPS coordinates in the different roadway sections composing a mixed cycling path (i.e. cycle track, roundabout, cycle track termini and bicycle/bus shared lane) (Fig. 8.).

The exposure to the occurrence of a conflict rises with the time the bicyclist spends in the road section and therefore cycling time was considered as an exposure metric to rate the number of anomalies among different roadway components to make the results comparable.

Risk Rate = N. of anomalies/cycling time.

The mean cycling time is reported in Fig. 9 which also shows the total number of BeST-DAD anomaly detections in the different road sections travelled during the test and the comparison between observed and CNN risk rates in a normalized scale (0–1).

Figure 9 shows cycle track termini at the highest risk rank followed by the roundabout. The normalized Risk Rate are also calculated to allow for comparisons with the observed risk rate reported in previous study [9]. It should be noted that in Fig. 9, the very good consistency between the two-risk ratings for cycling track and shared lane. Also consistent are the higher rating for lane termini and roundabout, even if BeST-DAD has classified a higher risk at the Cycle track termini than the one observed by actual CSE. The cycle termini end with a sharp curve before the lane crossing (Fig. 9) where cyclists were required to steer and often stop showing as anomalies in riding behavior (e.g. hard braking and steering) that have been correctly detected by BeST-DAD although not specifically related to traffic conflicts and therefore classified as FPs.

Despite the good performance of the CNN trained on the reference cyclist, we can expect larger deviance increasing the number of users. Because the CNN model depends on both the user and the specific road environment, transfer learning and cooperative learning [43] can be applied in real time to the model trained and transfer its knowledge to the specific user and road environment. To model the cycling behavior, our study used extended GPS NMEA contents (i.e. Speed, Heading) at high 10 Hz acquisition frequency which are not common in present smartphone and data mobility-data providers (e.g. Strava) following mainly the targets of profiling users' destinations and flows or fitting activity which need low frequency data acquisition (e.g. positions at 0.2 Hz). Therefore, frequencies up to 1 Hz are not yet available in standard smartphones whose development is moving mainly in the direction of improving the localization accuracy while already appropriate are speed and heading accuracy. To achieve the suitable higher frequency, an alternative to be evaluated is the capability of using SGF also for sub-sampling the GNSS signals at higher frequencies than the actual sampling rate [41].