Early warning system for drivers’ phone usage with deep learning network

With increasing of private car, the traffic accident rate is also rising. The inattention caused by mobile phones in driving is one of the main reason for traffic accident [1, 2]. By detecting the drivers’ behavior of playing phone, it is of great significance to prevent traffic accidents from the perspective of the driver. Therefore, how to efficiently detect the drivers’ behavior in driving has attracted increasing attention in recent years.

At present, the detection methods of drivers’ mobile phone playing are mainly divided into on-site law enforcement and automatic detection by law enforcement officers [1‐4]. However, the two monitoring methods are not efficient enough and can not be widely used in all private cars. Thus, in this work we develop the auto-monitor system in car to detect the drivers’ behavior in driving.

In recent year, some related researches have been proposed to detect the behavior of driver based on machine learning method. In previous studies [3, 4], an activity parsing algorithm was employed to identify whether the driver was using a mobile phone. It used the And-Or Graph (AoG) to represent the hierarchically compositions of the phoning activity to efficiently improve the detection performance.A driver phone calls detection based on voice feature recognition was proposed [4]. It could recognize the drivers’ voice from the collected audio data and determine whether the driver was participating in the current phone call or not. In Yasar’s workb5, a neural network application was used to detect mobile phone usage with a outside camera. Berri and Osorio [6] developed a 3D vision system, using a frontal Kinect v2 sensor, to monitor the driver and monitor the use of mobile phones by drivers. Some studies [7] show that the drivers’ face region could be localized by using the deformable part model (DPM), and a local aggregation based image classification technique could be utilized to classify a region of interest (ROI) around the drivers face for detecting the cell phone usage. Moreover,the Supervised Descent Method (SDM) [8] based facial landmark tracking algorithm was demonstrated to be able to track the position of face landmarks for detection,thus to determine if a driver is holding a cell phone in dirving. Despite the previous studies had made some progress, the influence due to the interference from the environment, e.g., different light and irrelevant background, was ignored, and the detection area in most previous studies was limited to face area only. All these limits the application of previous methods [1‐8]. Recently, it has shown that the interference from the environment has a great impact on safe driving detection [8].

In this work, we construct a lightweight deep network model to detect the behavior of playing mobile phone in the complex environment. The proposed monitor system is divided into two parts: the vehicle mobile terminal and the PC terminal. The vehicle mobile terminal quickly detects the drivers’ pictures collected by the in-car camera. The PC terminal can automatic communicate with the government department in the case of the drivers’ phone usage. The major contributions of our whork are: This paper is organized as follows: In Sect. 1, the research backgroud is presented. In Sects. 2 and 3, the construction of the early warning system for drivers’ playing mobile phone, as well as the details and functional of each module of monitor system are presented. Sect. 4 gives the experimental results and correspondig discussion. The conclusion are provided in Sect. 5.

Equipment response efficiency is a key factor when designing them. Specifically, if we want to remind drivers to correct driving behavior in time, we need to improve the recognition efficiency. In this regard, we need to correctly analyze ssdlite_mobilenet_v3(small) algorithm, and our recognition target is relatively simple. We can reduce unnecessary calculation by simplifying the model.

Table 1 shows the proposed method can achieve 99% accuracy in 100 images. And IoU represents the ratio of intersection and union of prediction frame and real frame. While IoU is set to 0.5, the accuracy achieve 99.7%. However while IoU is set to 0.75, the accuracy is 94.5%; The inference time is 46 ms.

We also compared the proposed method with the Yolov3 [16] and Faster-RCNN [12] network. As shown in Table 2, the accuracy of the three methods is high, but the running times of three method are different. The proposed network achieve 46 ms, while Yolov3 [16] network cost 4799.8 s, and the Faster-RCNN is demonstrated to be not applicable to raspberry pi used in this system due to its high computational cost [12]. Moreover, the ssd_mobilenet_v3 and Yolov3 models were demosntrated to have the better accuracies used in this system than that of the Faster-RCNN, as shown in Table 2. Note the accuracy of corresponding Faster-RCNN shown in Table 2 was achieved in the AI Studio server [17].

In this paper, we design a drivers’ phone usage detection system. It is composed of the mobile terminal and PC part. It used mobilenNet combined with the single shot multi-box detector to achieve object detection. Compared with other deep network, the proposed model can achieve high classification performance with less computational cost. It is a lightweight network that can fulfill a request of practical application. The proposed system can also applied in other detection fields, e.g., fatigue driving or driving without seat belt.