Fusion of eye movement and mouse dynamics for reliable behavioral biometrics

Information fusion is a very popular tool for improving biometric identification system performance. According to [3], fusion may combine multiple representations of the same biometric trait, multiple matchers using the same representation and, finally, multiple biometric modalities. Multimodal fusion may be done on various levels: (1) a feature extraction level, in which multiple traits are used together to form one feature vector; (2) a matching score level, in which results (typically similarity scores) obtained from different biometric systems are fused; and (3) a decision level, in which only output decisions (accept/reject) from different biometric systems are used in a majority vote scheme.

There are a lot of examples of multimodal biometric fusions. The most popular are fusions of physiological modalities like face and iris [4, 5] or fingerprint and iris [6, 7]. There are also works that present a fusion of the same modality measured by different sensors [8]. Finally, fusions of different algorithms processing the same data on matching score or decision levels have improved biometric identification results significantly [9, 10].

Analyzing the research regarding mouse event-based biometric identification, we find various approaches and many features of mouse movement that have been studied. Data obtained as a dynamic mouse signal consist of recordings including low-level mouse events such as raw movement and pressing or releasing mouse buttons. These are typically the timestamps and coordinates of an action and can be grouped in higher-level events such as move and click, highlight a text, or a drag and drop task. Based on these aggregated actions, a number of mouse-related features have been developed and applied for user identification.

Experiments available in the literature may be differentiated by various aspects. The first of them is the type of experiment, which includes edit text tasks [11], browser tasks [11, 12] and game scenarios [11, 13]. Ahmed and Traore [14] collected data during users’ daily activities. Similarly, online forum tasks for gathering mouse movement signal were utilized in the studies presented in [15]. A different type of experiment was proposed in the research presented in [16], in which a user had to use a mouse to follow a sequence of dots presented on a screen.

Studies may also be analyzed in terms of the environments used. In one group of experiments, participants worked on computers without any specially prepared environment [11, 12, 14]. Another approach was to use a controlled environment to prevent unintended events influencing the quality of samples [16‐18]. Zheng and el. [15] conducted tests in a self-prepared environment involving routine, continuous mouse activities as well as using an online forum.

Research can also be classified by the time in which an authentication takes place. There are studies that collected such data only at the beginning of the session [16] or continuously during the whole session [11, 13, 14, 18]. Since data gathered during experiments have to be processed to be useful in further analysis, each registered mouse movement signal is divided into small elements representing various mouse actions. Among such elements, several features can be distinguished, forming two types of vectors: spatial and temporal. The first describes changes in mouse position and includes mouse position coordinates; mouse trajectory; angle of the path in various directions; and curvature and its derivative. The second type of vectors depicts quantities related to mouse movement like horizontal, vertical, tangential and angular velocities, tangential acceleration and jerk.

The mouse movement dynamic has also been used in research applying various fusion methods. For example, in [19] a fusion of keystroke dynamics, mouse movement and stylometry was studied. Keyboard and mouse dynamics were also used in [20], yet this time were fused with interface (GUI) interactions. Two types of fusion were utilized: feature level fusion and decision level fusion.

We have also found studies in which: (1) two multimodal systems that combine pen/speech and mouse/keyboard modalities were evaluated [21]; and (2) fingerprint technology and mouse dynamics were used [22]. A different type of mouse dynamic-related fusion was utilized in [23]. This fusion considered only mouse movement, yet divided it into independently classified feature clusters. Subsequently, a score level fusion scheme was used to make the final decision.

Eye movement biometrics have been studied for over 10 years [24, 25] on the assumption that the way in which people move their eyes is individual and may be used to distinguish them from each other. Two aspects of eye movement may be analyzed: the physiological, concerning the way that a so-called oculomotor plant works, and the behavioral, which focuses on the brain activity that forces eye movement. Therefore, plenty of possible experiments may be utilized.

The most popular experiments focus just on forcing eye movements, as the physiological aspect seems easier to analyze and more repeatable. The simplest example of such an experiment is a so-called jumping point stimulus. During such a scenario, users must follow with their eyes a point displayed on a screen periodically changing position [24, 26, 27]. Studies with this kind of stimulus mostly measure physiological aspects, as subjects are instructed where to look and cannot make this decision autonomously.

The other popular type of experiment is recording eye movement while users are looking at a static image [25, 28, 29]. The content of the image may differ, but the most popular content so far is images with human faces. This results from the conviction that the way in which faces are observed is different for everyone [28, 30, 31]. A changing scene (movie) is the other possible stimulus [32, 33].

Another kind of experiment is recording eye movement while users fulfill some specific visual tasks. This seems to be a promising scenario; however, there are only a few research papers published so far including text reading [34], following with eyes more complex patterns [35] and normal activity like reading and sending emails [36].

When eye movement recordings are gathered, the next problem is how to extract attributes that may be usable for human identification. Various approaches have been proposed, one of the most popular of which involves the extraction of fixations (moments when an eye is relatively still to enable the brain to acquire a part of an image) and saccades (rapid movement from one fixation to another) and performing different statistical analyses on them. Simple statistics may be applied [37‐39] or more sophisticated, like comparisons of distributions used [40]. In ref. [26], an interesting attempt to use eye movement data to build a mathematical model of the oculomotor plant has also been presented. Other approaches analyze the eye movement signal using well-known transformations like Fourier, wavelet or cepstrum [24, 41, 42]. There are also some methods that take spatial positions of gaze data into account to build and then analyze heat maps or scan paths [28, 30].

The results obtained in all the aforementioned experiments are far from ideal. Additionally, it is difficult to compare results of various experiments because scenarios, hardware (i.e., eye tracker) and participants vary between them all. Unfortunately, authors are reluctant to publish their data, which would enable future comparisons. A notable exception is the EMBD database (http://​cs.​txstate.​edu/​~ok11/​embd_​v2.​html) published by Texas State University and databases used in publicly accessible Eye Movement Verification and Identification Competitions: EMVIC 2012 [27] and EMVIC 2014 [31].

Although it seems natural that the eye movement modality may be combined with other modalities, to the best of our knowledge there have been only two attempts to provide eye movement biometrics in fusion with another modality. In ref. [43], eye movements were combined with keystroke dynamics, but the results showed that errors for eye movements were very high and the improvement when fusing both keystroke and eye movements was not significant. In ref. [44], eye movement biometrics were fused with iris recognition using low-quality images recorded with a cheap web camera.

The analysis of the existing methods used for biometric identification in both previously described areas encouraged the authors to undertake studies aimed at compounding signals of eye and mouse movement in a user authentication process. There are several reasons that such studies are worth undertaking. Both signals stem from human behavioral features, which are difficult to forge. Their collection is easy and convenient for users, who naturally use their eyes and a mouse to perform computer-related tasks. Furthermore, the devices that acquire these signals are simple and cheap, especially when built-in web cameras are used, and can be easily incorporated in any environment by installing the appropriate software. The important feature of the considered solution is also the fact that both signals can be registered simultaneously, which makes data collection quicker. Additionally, if necessary, the method may also be used for covert authentication.

A novel type of experiment that was based on entering a PIN was designed for this purpose.

Data obtained from both eye and mouse movements were processed to construct dissimilarity matrices [2] that would provide a set of samples for training and testing phases of a classification process. A similar approach was used in [17] for mouse dynamics; however, it has never been applied for eye movement data. Taking the above into consideration, the research contribution may be listed as follows:

All data were gathered with one experimental setup consisting of a workstation system equipped with an optical mouse and the Eye Tribe (www.​theeyetribe.​com) system for recording eye movement signal at sampling rate of 30 Hz and an accuracy error of less than 1\(^{\circ }\). It is worth mentioning that this eye tracker is affordable ($100) and convenient to use, unlike most of the eye trackers used in the previous research of eye movement biometrics. The eye tracker was placed below a screen of size 30 \(\times\) 50 cm. The users sat centrally at a distance of 60 cm. Three such systems were used simultaneously during the data collection phase. The low frequency usage was motivated by the idea of checking whether valuable data may be obtained even for frequencies available to commonly used web cameras. Additionally, mouse movements were recorded with the same frequency.

All tests were conducted in the same room. At the beginning of each session, participants signed a consent form and were informed about the purpose of the experiment. Each session for each participant started with a calibration process ensuring adjustment of an eye tracker to the eye movement of the particular user. Users were asked to follow a point on the screen with their eyes. After nine locations, the eye tracker system was able to build a calibration function and measure a calibration error. Only users obtaining a calibration error value below 1\(^{\circ }\) were allowed to continue the experiment.

In the next step, circles with 10 digits (0–9) were evenly distributed over the screen, displayed (Fig. 1). The participant’s task was to click these circles with the mouse to enter a PIN number. The PIN was defined as a four-digit sequence, for which every two consecutive digits were always different. Both mouse positions and eye gaze positions were recorded during this activity. It was assumed that people look where they click with the mouse; therefore, eye and mouse positions should follow more or less the same path. One such recording of a PIN being entered is called a trial in subsequent sections. A trial is a completed task of entering one PIN, during which eye and mouse movements were registered. To make simulation of a genuine–impostor behavior possible, all participants entered the same PIN sequence: 1–2–8–6.

There were several sessions with at least a 1-week interval between sessions. During each session, the task was to enter the same PIN three times in a row.

A total of 32 participants took part in the experiments, and 387 trials were collected. As each user entered the PIN three times during one experiment, the trials were grouped into sessions. Each user’s session consisted of three subsequent trials. The gathered trials were used to prepare three collections differing in the number of sessions registered for one user:

The aim of the preparation phase was to separate different signals from eye and mouse movements recorded during the experiments. A signal is defined as a characteristic feature that can be extracted from each trial. This analysis concerned only parts of recordings collected between the first and fourth mouse click.

As a result, 24 separate signals were calculated: 11 signals for mouse, 11 signals for gaze and two additional signals representing mouse and eye position differences (Table 1). Depending on the length of the recording, each signal consisted of 105–428 values (from 5 to 21 s).

The second step in the authentication process was to define a set of samples that could be used as input for a classifier. The input for this phase was the fusion of 24 mouse and eye signals prepared for each trial earlier.

Three different feature extraction algorithms were used:The detailed description of each is presented in the following sections.

At the end of the feature extraction phase, several sets of samples were collected:All these sets were built separately for all collections of trials (C2, C3 and C4) described in Sect. 3.2. Each set, divided into N training and M testing samples, was then evaluated using the cross-validation method (Table 2). It is very important to emphasize that the division into training and testing sets was not random. Consecutively collected trials tend to be more similar to each other than trials collected after longer intervals; therefore, due to the short-term learning effect [47], including them in both training and testing sets may produce improperly obtained better accuracy results. Hence, the general rule was not to use trials of the same user gathered in the same session for both training and testing purpose. Detailed analysis of this phenomenon can be found in Sect. 5.2.

Building a rule according to which a fold was related to one session was a motivating factor. Therefore, collection C4 was divided into fourfold representing four sessions. As a result, all samples of one user from the same session were always in the same fold and were used together as either training or testing samples. A similar procedure was applied for C3 and C2 collections, dividing them into three and twofold, respectively. For such a folding strategy, a testing set always contained three trials of each user recorded during the same session (one by one).

A classification model was built based on N training samples, with usage of an SVM classifier [48]. Using data of a similar structure utilized in our previous research [49] and a grid search algorithm, we obtained the best results for the RBF kernel with \(\hbox {gamma}=2^{-9}\) and \(C=2^{15}\). Therefore, these values were used in the current research. The sequential minimal optimization algorithm was used [50] with the multiclass problem solved using pairwise coupling [51]. The classification model was then used for classification of M testing samples. For each of them, the classifier returned a vector of probability values that a given sample belongs to a particular user. If the number of users is denoted by U, for every testing sample we obtain a U element vector representing distribution of probabilities for each of U possible classes. A set of such M vectors (for all testing samples) forms a matrix of size \(M \times U\).

Initially, during the testing phase, all trials in a testing set were classified separately giving independent distributions for each trial a: \(\hbox {Ptrial}_{\mathrm{a}}\). These distributions were subsequently summed up and normalized for trials related to the same session (let us recall that there were three trials for one session). Having probability vectors of three trials (a, b and c) of the same user gathered during the same session, the probability vector for the session was calculated as:where set represents the set of samples used. Such a probability vector was the outcome of the method using the statistic features. However, an additional step was designed for \(hist_{\mathrm{bin}}\) and \(\hbox {matrix}_{\mathrm{signal}}\) types as both corresponding methods for the feature extraction define more than one set. The histogram method provided different sets for a particular number of bins (10, 20, 30, 40 and 50)—altogether five sets—whereas in the distance matrix approach we obtained 24 sets, each for one signal. Hence, the result in these cases was determined as a sum calculated for all bins or signals sets. The vector of probability distribution, after the last step, included values as those presented in Eq. 5, where X represented the number of sets used (a number of bins or a number of signals, 5 or 24, respectively).The result of this step was three probability distributions:These three distributions were then used in the subsequent evaluation step to check their correctness. It should be emphasized that in the process of the probability distribution evaluation, a fusion of features characterizing eye movement and mouse dynamic was applied.

The last step of the classification process was to assess the quality of models developed in the previous phases. The result of the testing phase was probability distributions for every possible class U (user identity). As was explained in the previous section, distributions were calculated using three trials from one session so the number of distributions was \(S=M/3\), where M was the number of testing trials. The result was a matrix \(P: [S \times U]\), where each element \(p_{i,j}\) represented the probability that the ith testing sample belongs to user j.

At first, the correctness of the classification c(i) for every ith distribution on the basis of its correct class u(i) was calculated as:Then, the accuracy of the classification for the whole testing set was calculated:The next step was calculation of acceptance \(a_{i,j}\) for different thresholds th. The value of thresholds ranged from 0 to 1.Based on this acceptance, it was possible to calculate FAR and FRR for different thresholds. It can be easily predicted that all samples were accepted for a rejection threshold th = 0; thus, FRR = 0 and FAR = 1. When increasing the threshold, fewer samples were accepted, hence FRR increased and FAR decreased. For th = 1, no samples were accepted, consequently FRR = 1 and FAR = 0. FAR and FRR dependency on rejection threshold value is presented in Fig. 4.

Equal error rate (EER) was calculated for the rejection threshold value for which FAR and FRR were equal (as visible in Fig. 4).

The next research question was to check whether a fusion of gaze and mouse biometrics gives results better than a single modality. For this purpose, two additional experiments for the C4 dataset were performed: one using only mouse-related signals and one using only gaze-related signals. Both concerned only the matrix method, which yielded the best outcomes in the previous tests. Table 4 presents a comparison of these results to the fusion of both modalities.

The row denoted by “Gaze” corresponds to the efficiency of the algorithm when only 11 signals derived from eye movement were taken into account. The same regards the “Mouse” row, which shows results for 11 signals derived from mouse-related signals. The results presented in the “Fusion” row are calculated on the basis of all 24 signals (11 mouse + 11 gaze related + 2 based on mouse–gaze differences). All these outcomes revealed that mouse dynamics gave better accuracy and lower errors than eye movements. Most importantly, the fusion of mouse and gaze gave results significantly better than both modalities alone.

The learning effect is a phenomenon characteristic of biometric modalities that measures changes of human behavior over time [47]. It is sometimes treated as a kind of well-known template aging problem, but its nature is slightly different. While template aging is related to biometric template changes over a long time (e.g., a face gets older), the learning effect addresses short time changes in human behavior. It is obvious that a tired or sad person reacts differently than a rested and relaxed one. Various beverages and food such as coffee or alcohol may also influence people’s behavior. For this reason, it is very important to register behavioral biometric templates with some considerable time interval to avoid short-term similarities and extract truly repeatable features. This phenomenon has already been studied for eye movement, and the results showed that eye movement samples collected at intervals of less than 10 min are much more similar to each other than samples collected at 1-week interval [52].

During the tests described in Sect. 4, we tried to avoid this problem by the appropriate preparation of training and testing folds of samples. We ensured that during the cross-validation, samples related to a user’s session were never split into two folds (see Sect. 4.3) and the time interval between two sessions of the same user was never shorter than 1 week. We called this folding strategy “session-based folding,” as data for the whole session was always in either a training or testing set.

However, we decided to raise the research question to check whether mixing samples derived from one session in training and testing sets did indeed result in better classification performance. Therefore, the additional cross-validation experiment was performed with a different fold preparation strategy. As there were always three trials in each session, this time every set was divided into three folds: The first trial of the session was in fold 1, the second attempt in fold 2 and the third one in fold 3. We called this folding strategy “mixed sessions folding,” as this time trials from the same session were always divided into separate folds.

Using such folds for cross-validation ensured that there was always a sample of the same user from the same session in both training and testing sets. The classification results are compared to the previous ones and presented in Table 5.

As could be expected, the accuracy for modified folds was higher and errors were lower because it was easier for the classifier to classify a trial with two other trials from the same session (i.e., very similar). The errors were lower for both modalities, but the difference for gaze-based biometrics was more significant. As given in Table 5, accuracy for the gaze was even better than for the mouse. Accuracy for the fusion reached 100 % because the correct class had the highest probability for every sample, but EER was not 0 % because it was not possible to find one threshold that worked perfectly for every sample distribution. If a threshold perfectly separated probabilities of genuine and impostor classes for one sample, the same threshold did not work perfectly for other samples.

When designing our research, we decided to involve the fusion technique on the decision level for the distance matrix method and on the feature level for the statistic one [3]. The next planned step is to extend all methods to involve fusion on various levels. For this purpose, various feature selection methods are also planned to be taken into consideration.

Additionally, we plan to conduct the same experiments for more participants. Data were collected for 32 participants used during the experiment. Such a pool of data seem to be enough to draw some meaningful conclusions; however, a much larger pool is necessary to confirm our findings. Moreover, our experiments showed that a higher number of training samples guarantees better classification performance. Therefore, it may be expected that more than three training samples (as was for our best collection) should improve the results. Five to six sessions are planned for each participant. With more data to analyze, it would be possible to calculate weights for each of the elements of the fusion. Weighted fusion would probably give even better results.