Hybrid computer vision system for drivers' eye recognition and fatigue monitoring

Abstract

This paper presents a hybrid visual system for monitoring driver's states of fatigue, sleepiness and inattention based on driver's eye recognition. Safe operation in car conditions and processing in daily and night conditions are obtained thanks to the custom setup of two cameras operating in the visible and near infra-red spectra, respectively. In each of these spectra image processing is performed by a cascade of two classifiers. The first classifier in a cascade is responsible for detection of eye regions based on the proposed eye models specific to each spectrum. The second classifier in each cascade is responsible for eye verification. It is based on the higher order singular value decomposition of the tensors of geometrically deformed versions of real eye prototypes, specific to the visible and NIR spectra. Experiments were performed in real car conditions in which four volunteer drivers participated. The obtained results show high recognition accuracy and real-time processing in software implementation. Thanks to these the system can become a part of the advanced driver’s assisting system.

Introduction

Road accidents are the leading cause of death in industrialized countries. These are mostly caused by human errors occurring in different activities related to vehicle driving, as well as low violations such as speeding. Also drivers' fatigue, sleepiness or inattention can lead to serious accidents. As reported by the European Union only driver tiredness constitutes a significant factor of about 20% of crashes involving heavy commercial vehicles [29]. These problems were recognized and countermeasures were undertaken in many countries. For instance, in the European Union there are plans to significantly reduce a number of fatal accidents in the nearest years by changing low regulations and also employing new technologies [28], [30].

Thanks to recent hardware and software developments computers can be used to monitor drivers' conditions with the potential of alerting in dangerous situations [45], [5]. In this respect hybrid intelligence offers new methods and tools which can be successful in solving such difficult problems, as reported in literature [11], [1], [20], [60], [12], [46], [8]. Such driver's supporting methods become parts of the Driver Assisting Systems (DAS). It is a matter of incoming years to have DAS practically available in modern cars. One of the key functionalities of DAS is reliable detection of driver’s fatigue and sleepiness, mostly based on observation of his/her eyes[6], [26].

Face and eye recognition belong to the fundamental tasks of computer vision [27], [56], [57]. Eye recognition for monitoring driver's behavior, fatigue, sleepiness or inattention is one of the research topics which were greatly enhanced for the last years [21], [22], [40], [41], [42], [43], [51], [52]. Eye detection can be divided into active and passive methods [58]. In the former group, a custom hardware is required which supplies special lighting, such as near IR [61], [5]. On the other hand, passive methods assume only the natural illumination spectrum. This group can be further partitioned into the methods which use color or monochrome images [7]. Reported eye recognition methods rely on detection of some characteristic eye features. These, in turn, can be obtained with template matching [33], projections [62], Hough transform [45], [62], gradients [36] or wavelets [44], to name a few. Frequently used indicator of driver's state is the percentage of eye closure (PERCLOSE) [23], [24], [54]. However, the existing solutions still lack sufficient accuracy or speed to reliably operate in real car conditions.

In this paper a visual system for driver's eye recognition is proposed which allows real-time operation in moving vehicles and under various conditions. For this purpose the original hardware setup was constructed which consists of two cameras operating in the visual and near infra-red (NIR) spectra. NIR illumination is obtained with a NIR light emitting diode (LED). This paper follows our previous work focused upon a method of processing images acquired in only daily conditions with color camera [18]. The system presented in this paper contains a new vision path processing NIR signals which allows operation in night conditions.

In the presented system there are two independent blocks of visual signal processing. Each of them is composed of two different classifiers connected in series: The front-end is responsible for the detection of eye candidates, followed by the trained classifier whose role is to refine the candidates and to respond whether the eyes are well visible. The two front detectors for the visible and NIR spectra are different, however. In the former, operation starts with the skin segmentation from color images [18]. This stage is followed by the skin region detection obtained with the adaptively window growing method (AWG) [14]. Then, eye candidate regions are selected thanks to the proposed eye model. Finally, eyes are verified by the classifier operating with the higher order singular value decomposition (HOSVD) of the tensor of geometrically deformed images of real eye prototypes [15]. On the other hand, processing of the NIR images starts with the detection of pupil candidates. This is done in the associated integral image which allows very fast computation of sums of pixel values in any rectangle within an image. Then iris and finally eye regions are detected based on the novel eye model, designed especially for the NIR signals. Eye regions are then refined to only those which fulfill predefined geometrical relations. Finally and similarly to the visible spectrum path, eye region candidates are recognized by the HOSVD classifier, this time trained with the prototypes from the NIR images, however. In this paper we focus mostly on the new NIR processing path. As already mentioned, part of the system operating in the visible spectrum is described in our previous paper [18].

Driver's safety is especially stressed in the presented system. This relates to the hardware construction and especially to the NIR lighting module. It is well known that long exposition to the NIR illumination of high power, although not directly visible, can be annoying or even dangerous for the exposed person [2]. Therefore in our system we use only one LED of low power synchronized with the shutter of the camera.

Experiments were conducted in different car conditions with the help of four volunteer drivers. Presented experimental results obtained in real car conditions show high accuracy and real-time operation of the system in software implementation.

This paper is organized as follows. Architecture of the system and the custom hardware setup are presented in Section 2. The proposed method of eye detection in NIR spectrum is dealt with in Section 3. Similarly, detection in the color images is discussed in Section 4. In Section 5 the method of eye recognition in the tensor space of deformed pattern prototypes is presented and discussed. Additionally, the Image Euclidean Distance method is presented which increases accuracy of eye recognition. Experimental results are presented in Section 6. This paper ends with conclusion in Section 7 as well as with the literature references.