Extraction of Vehicle Turning Trajectories at Signalized Intersections Using Convolutional Neural Networks

Several studies have been conducted in the past few decades to grasp, as possible, the turning behavior of vehicles at signalized intersections. In general, significant characteristics concerning the intersection layout and the turning vehicle were highlighted [26‐29]. As an example, Alhajyaseen et al. [30] underlined that the path of the turning vehicle is strongly related to the intersection’s angle, the vehicle’s type, and speed. However, it is agreed that the turning maneuver of vehicles is a further complex phenomenon whose variability extends to be related to highly dynamic factors [31, 32]. For instance, the turning behavior was observed to depend on inter- and intra-subject factors concerning drivers such as the perception of traffic environment, information processing, and the ability to react correctly and to cooperate with others [33, 34]. Other factors such as the waiting time of the turning vehicles [35], the relative speed of the vehicles in conflict [36], and gaps [37] were observed to impact on the decision behavior of turning vehicles.

Since understanding the complex mechanism of turning vehicles’ paths is crucial to achieve an effective traffic control and safety assessment at intersections, a reliable simulation model is required so that the variations of the turning vehicles are reproduced with high resolution. Classically, the vehicle’s turning path was simulated via one-dimensional models [38‐40]. In such simulations, a set of lane-based models are defined, in which the longitudinal and lateral movements are separately represented by a car-following model and a lane-changing model, respectively. Despite their simplicity and applicability to be involved in decision-making approaches, the one-dimensional models are unable to accurately reproduce the variations of turning trajectories [41].

As an alternative to the traditional one-dimensional trajectory models, the two-dimensional model has emerged as a viable simulation technique for vehicle turning paths as it breaks the lane-based assumption. Accordingly, the longitudinal and lateral movements are simultaneously simulated, and therefore, the characteristics of the turning paths are reliably reflected [42]. However, these approaches produce the path of the turning vehicles only without the speed and acceleration profiles [30], which are usually estimated using other independent models. In this context, a microscopic simulation model that generates vehicles’ turning trajectories was developed by Tan et al. [43] using different models for path (Euler spiral-based approximation method) and speed profiles. More recently, Wei et al. [44] established a left-turning vehicle’s path model by means of extracting trajectories from recorded videos and analyzing their distributions, velocities, and flow-changing characteristics. On the basis of this effort, the same authors [44] proposed the idea of setting left-turning guidelines at signalized intersections, which was verified as an effective tool to reduce traffic conflicts and improve traffic efficiency. Also, Ma et al. [45] proposed a three-phased (i.e., plan-decision-action) model to estimate the vehicle’s path at mixed-flow intersections. However, combining different models for turning path and speed profile does not ensure the spatial and temporal consistency between the location and the speed of turning vehicles. In an attempt to overcome such limitation, Dias et al. [7, 46] applied the concept of minimum jerk to fit the trajectories of manually tracked free-flow turning vehicles at signalized intersections in Japan. The proposed approach simultaneously estimates the temporal and the spatial profiles of the vehicle turning maneuvers with acceptable accuracy. Yet, the same authors [7, 46] did not discuss the limitations of their proposed approach and the procedure to generate the maneuvers of turning vehicles in microsimulation.

As an alternative to manual trajectory extraction, researchers have proposed several methods for semiautomatic and automatic tracking of the turning vehicles. For instance, Shirazi and Morris [47] proposed a semiautomatic technique to extract vehicles’ trajectories from traffic footages. This method requires first to identify the locations of the vehicles in each video frame manually. After that, vehicle tracking is performed using a detection-track mapping matrix which utilizes nearest global matching. Yet, despite them producing accurate results, semiautomatic techniques are considered laborious since they initially require some steps to be performed manually [48‐50].

Automatic extraction techniques, on the other hand, are deemed more promising since they provide swift results with minimum manpower involved. In this context, Hsieh et al. [11] proposed an automatic vehicle tracking method which implements a background subtraction technique for vehicle detection along with a Kalman filter for tracking. Similarly, Shirazi and Morris [51] used a Gaussian mixture model to detect vehicles at signalized intersections together with a Kalman filter for trajectory extraction. Apeltauer et al. [12] developed another automated method for trajectory extraction in which vehicles are detected using a two-stage classifier trained based on multi-block local binary pattern (MB-LBP) features. This method also requires applying background subtraction in order to generate the foreground mask. Also, Khan et al. [13] developed a comprehensive framework for automatic trajectory extraction of the vehicles from traffic footages acquired by unmanned aerial vehicles (UAVs). This framework involved video preprocessing and stabilization, vehicle detection and tracking, and ultimately, management of the extracted trajectories. Similar to [11, 12], Khan et al. [13] carried out the vehicles’ detection using background subtraction algorithm.

CNNs are deep, biologically inspired feed-forward artificial neural networks (ANNs) which have been developed based on a core model for mammalian visual cortex. One of the most attractive features of CNNs is their ability to classify images regardless of their scale and orientation [52]. A typical CNN designed to deal with 28 × 28 pixel RGB images is depicted in Fig. 1. The structure consists of alternating convolutional and sub-sampling layers followed by a number of multilayer perceptrons (MLP) layers (i.e., fully connected layers). Each convolutional layer contains a number of filters (neurons) having a specific kernel size (\( k_{x} = k_{y} = 5 \) in this illustration). These filters are responsible for extracting particular features from the input image called the feature maps. The convolutional filters are basically matrices of size \( \left( {k_{x} ,k_{y} } \right) \) containing certain values referred to as the weights. At each neuron, 2D convolution is performed between the input image and the filter’s weights. The output of this operation is processed by an activation function and then passed to the next subsampling layer, which decimates the feature maps extracted by the previous convolutional layer by a predefined sub-sampling factor. As shown in Fig. 1, after being processed by a sufficient number of convolutional and subsampling layers, the input image is reduced to a 1D array. This array is then processed by the MLP layers resulting in an output vector that represents the classification results.

The process of computing the weights of the convolutional filters and MLP layers is defined as CNN training. Before carrying out the training process, it is necessary to define the CNN’s structure in advance. This includes the number of convolutional and MLP layers as well as the kernel size and subsampling factor at each level and the number of neurons in each layer of the CNN. Such hyperparameters are usually picked by trial and error since there is no systematic approach for determining the optimal CNN structure within an acceptable computational time [16]. Afterward, the weights in both convolutional and MLP layers are initialized randomly. A large set of images is then used to train the CNN according to a back-propagation (BP) algorithm. The objective of this training process is to adjust the CNN weights in an iterative manner until the summation of squared error between the target values and the CNN output is minimized. The BP operation is not explained in this paper for brevity. The interested reader is referred to [53] for more details about training CNNs.

Instead of training a new CNN starting from randomly generated weights, researchers often apply a technique called transfer learning in which a pretrained CNN is fine-tuned to learn a new task. Networks such as AlexNet [54] and GoogLeNet [55] are commonly used as a starting point in deep learning applications. Previous studies have shown that this approach is faster and more efficient than training CNNs from scratch [56].

It must be noted here that CNNs are only designed to classify the input image into a number of predefined classes without being able to detect and localize specific objects within the image. Therefore, CNNs alone cannot satisfy the main requirement of this study, which is to detect and track vehicles automatically. To bridge the gap between image classification and object segmentation, Girshick et al. [57] have proposed a method called regions with CNNs (R-CNN). As shown in Fig. 2, this method consists of three components: (1) a region proposal algorithm that generates a large number of candidate detections, (2) a large CNN that extracts features from each proposal, and (3) linear support vector machines (SVMs) to process the extracted features and classify each candidate regions.

The south approach of Lekhwair signalized intersection located in Doha City, State of Qatar, was videotaped for a duration of two and a half hours. The video was recorded at a frame rate of 30 fps and a resolution of 3840 × 2160 pixel. The same video was used in the current study for both R-CNN training and trajectory extraction operations.

The images required for carrying out the training of the R-CNN were acquired by randomly selecting 26 frames of the video. To reduce the computational time required for the training, the images were cropped to the region around the middle of the intersection and the resolution was reduced to 1920 × 1080 pixel. For each image, vehicles were manually labeled by bounding boxes, and consequently, a dataset for the coordinates of 536 vehicles in total was obtained. Based on the images and the bounding boxes dataset, the R-CNN training process was carried out using “trainRCNNObjectDetector” function available in MATLAB Computer Vision System Toolbox. AlexNet was used as a starting point for this deep learning task. Details about the architecture of this network can be found in [54].

The trajectories generated by the aforementioned approach describe the locations of moving vehicles in image coordinates (pixels) with respect to time. In order to map the trajectories to the real-world coordinates, it is necessary to define the homography matrix corresponding to this transformation. To do so, it is required to have four known points in both real-world and image coordinates. Then, the homography matrix \( {\mathbf{H}} \) can be computed as follows:where \( \left( {x_{w,1} ,y_{w,1} } \right) , \ldots , \left( {x_{w,4} ,y_{w,4} } \right) \) are the real-world coordinates of any four noncolinear points and \( \left( {x_{1} ,y_{1} } \right) , \ldots , \left( {x_{4} ,y_{4} } \right) \) are the image coordinates (in pixels) of the same four points. The matrix \( {\mathbf{H}} \) can be then calculated as the first eigenvector (reshaped into a 3 × 3 matrix) of \( {\mathbf{H}}^{{\mathbf{T}}} {\mathbf{H}} \). Next, the homography matrix can be used to map any point from image coordinates \( \left( {x_{i} ,y_{i} } \right) \) to world coordinates \( \left( {x_{w} ,y_{w} } \right) \) as follows:

In order to verify its accuracy, the proposed tool was used to extract the trajectories of 18 free-flowing left-turning vehicles. The same vehicles were also tracked manually in order to identify the ground truth of the trajectories. The manual trajectory extraction was performed simply by tracking each vehicle at a 0.5-s rate, while drawing a bounding box around the vehicle in each frame. The location of a manually tracked vehicle position at a certain time was taken as the centroid of the bounding box. The manual trajectories were then transformed into real-world coordinates as explained in Sect. 4.1. A comparison between the automatically and manually extracted trajectories is shown in Fig. 5. The error between the trajectories extracted by the proposed tool and their manually extracted counterparts was calculated at each point of the trajectories. The error \( E_{p} \) was defined here as the distance between an automatically extracted point and the corresponding manually extracted one according to the following equation:where \( x_{\text{a}} \) and \( y_{\text{a}} \) are the coordinates of the automatically extracted point and \( x_{\text{m}} \) and \( y_{\text{m}} \) the associated manually extracted point. The error distribution of the points corresponding to the 18 trajectories is shown in Fig. 6. The average error across all points of the trajectories is 16.5 cm with a standard deviation 11.8 cm. These results support the ability of the proposed tool to automatically track vehicles with an acceptable level of accuracy.

Originally, the principle of minimum jerk was proposed in the mid-1980s by Flash and Hogan [58] to describe the planar movements of the human arm, after which the use of this method has gained more popularity and general acceptance. Successful applications of the minimum-jerk method have been reported in various contexts including human goal-oriented locomotion [59], robot-limb movements [60], autonomous vehicle maneuvers [61, 62], driver-following behavior [63], and more recently a preliminary application for modeling the trajectory of turning vehicles [7, 46].

In principle, the minimum-jerk model suggests that the drivers tend to optimize the smoothness of turning maneuvers by minimizing the time integration of the jerk. Thus, according to [58], the cost function to be minimized can be given as:where \( t_{f} \) is the time elapsed by the turning vehicle to cross the intersection.

As indicated by Flash and Hogan [58], the solution of the minimization problem given in Eq. (4) can be obtained in the time domain as a set of fifth-order polynomials for \( x \) and \( y \) as follows:where \( a_{i} \,{\text{and}}\,b_{i} \) (\( i = 0,1, \ldots ,5 \)) are unknown coefficients to be obtained using twelve boundary conditions corresponding to the x- and y-components of location, velocity, and acceleration at the initial and final points of the vehicle’s trajectory. By applying the boundary conditions corresponding to the x-direction on Eq. (5), the following system of equations can be obtained:where \( x_{0} \), \( v_{x0} \), and \( a_{x0} \) are the position, velocity, and acceleration, respectively, in the x-direction at the starting point of the turning maneuver, and \( x_{f} \), \( v_{xf} \), and \( a_{xf} \) are those corresponding to the end point. Likewise, applying the boundary conditions corresponding to the y-direction yields a system of equations similar to that of Eq. (7) but in terms of the coefficients \( b_{i} \left( {i = 0,1, \ldots ,5} \right) \) and the parameters \( y_{0} \), \( v_{y0} \), \( a_{y0} \), \( y_{f} \), \( v_{yf} \), and \( a_{yf} \).

In order to compute the coefficients \( a_{i} \) and \( b_{i} \left( {i = 0,1, \ldots ,5} \right) \) corresponding to each of the extracted trajectories, it is necessary to identify the x- and y-components of position, velocity, and acceleration at the points at which the vehicle starts and ends its turning maneuver, along with the time taken during the maneuver \( t_{f} \). Once these values are known, the two systems of equations (described in Sect. 5.1) can be easily solved to obtain the coefficients \( a_{i} \) and \( b_{i} \) corresponding to the trajectory’s turning curve.

Therefore, it is necessary to accurately identify the two points along the trajectory at which the turning maneuver starts and ends. To do so, we utilize the spline-fitting method presented in [30]. According to this method, the trajectory of a left-turning vehicle at a signalized intersection can be represented by a spline consisting of five segments. The spline starts with a straight line followed by an Euler spiral having a curvature profile that varies almost linearly with a gradient of \( 1/A_{1}^{2} \). This spiral is followed by a circular segment with a curvature of \( 1/R_{ \hbox{min} } \). The end of the spline consists of another Euler spiral having a nearly linear curvature profile with a gradient of \( - 1/A_{2}^{2} \) followed by a straight line. As shown in Fig. 8, there are four main locations that define the beginning and the end of each Euler spiral and circular segments. These locations are basically the points of discontinuity along the curvature profile of the vehicle’s path. The points of interest here are points 1 and 4 in Fig. 8, which represent the starting and ending points of the turning maneuver.

A MATLAB code was written to fit the aforementioned spline to each of the extracted trajectories in order to identify the four points of curvature discontinuity along the vehicles’ turning paths. The code applies the nonlinear programming solver “fmincon” available in MATLAB Optimization Toolbox to compute the optimal location of the four key points (described in Fig. 8) so that the error between the tracked path and the fitted spline is minimized. Four constraints were imposed to enforce continuity of the fitted spline at the four points. Also, another four constraints were applied to make sure that there are no sudden jumps in the curvature profile at the key points. The fitting of the two Euler spirals was conducted according to the approach proposed in [64]. Figure 9 displays four examples of splines fitted to their corresponding automatically extracted paths.

After identifying the two points of interest along each of the extracted trajectories (as explained in Sect. 5.2), the parameters \( x_{0} \), \( v_{x,0} \), \( a_{x,0} \), \( x_{f} \), \( v_{x,f} \), \( a_{x,f} \), \( y_{0} \), \( v_{y,0} \), \( a_{y,0} \), \( y_{f} \), \( v_{y,f} \), \( a_{y,f} \), along with \( t_{f} \), were computed for each trajectory. Figure 10 displays the probability distributions for these 13 parameters. As shown in the figure, a normal distribution was fitted for each parameter. One-sample Kolmogorov–Smirnov test (95% confidence level) indicated that each of the 13 parameters comes from a normal distribution with the mean and standard deviation shown in Table 1.

Monte Carlo simulation with 500 trials was conducted using the developed models. In each trial, random values of \( x_{0} \), \( v_{x,0} \), \( a_{x,0} \), \( x_{f} \), \( v_{x,f} \), \( a_{x,f} \), \( y_{0} \), \( v_{y,0} \), \( a_{y,0} \), \( y_{f} \), \( v_{y,f} \), \( a_{y,f} \), and \( t_{f} \) were generated according to the normal distributions described in Fig. 10 and Table 1. The resulting parameters were then used to compute the coefficients which determine the shape of the trajectory (\( a_{i} \) and \( b_{i} \)) by solving the two systems of equations described in Sect. 5.1. The coefficients were then used as per Eqs. (5) and (6) to obtain the simulated paths shown in Fig. 11a.

The distributions of the simulated paths, and those of the observed/actual trajectories, were analyzed and compared along three selected cross sections (drawn in Fig. 11a). Figure 11b–d depicts a comparison between the observed and the simulated distributions. Kolmogorov–Smirnov test (performed at 95% confidence level) revealed that the simulated distributions at the three cross sections are not significantly different from the actual/observed counterparts. Furthermore, the comparison shown in Fig. 12 indicates a reasonable agreement between the observed and simulated speed and acceleration profiles, which supports the reliability of the proposed model in generating accurate and realistic vehicle turning maneuvers.

Finally, Fig. 13 provides a concise summary of the overall procedure followed to develop and validate the proposed minimum-jerk-based trajectory model, starting from trajectory extraction and ending with the use of Monte Carlo simulations to generate trajectories of the turning vehicles.