Handwriting Biometrics: Applications and Future Trends in e-Security and e-Health

Online handwriting biometric systems belong to behavioral biometrics as they are based on actions performed by a specific subject [1]. This is complementary to morphological biometrics, which are based on direct measurements of physical traits of the human body [1, 2]. From a human behavior and health condition perspective, online handwriting biometrics are more appealing and informative than other popular biometrics traits such as fingerprints or iris [3, 4]. Although health applications based on online handwriting have not been explored in-depth yet, there is a considerable set of possibilities that will probably be developed in the near future, such as diagnosis/monitoring of depression, neurological diseases, and drug abuse [5]. It can be noted that nowadays, most of the published research in biometric signal processing is based on image and speech. This might be linked to the fact that these signals are simpler and cheaper to acquire compared with others such as online handwriting where specific digitizing tablets are needed. Fortunately, online handwriting signals are more present in our society than a few years ago due to the increasing popularity of tactile devices and their corresponding cost reduction [6]. Thus, the price of the acquisition device is no longer a drawback nowadays. Also, most commercial off-the-shelf (COTS) smartphone devices incorporate handwriting capabilities. As a result, we forecast a considerable growth in applications in this field. Indeed, in a near future, these pen-based handheld devices (tablets, phablets, cellphones, etc.) will help to protect people’s sensitive data, for example, incorporating automatic signature verification and writer identification. Moreover, the same devices could be used to monitor user fine motor control to detect stress, aging, and health problems [5]. In this dual context, privacy issues will have to be seriously investigated. This will be especially important when the double use is possible. Ideally, samples acquired for security purposes (e.g., grant access to a facility or authorize a payment) should not be able to reveal health information of the user. Conversely, samples acquired for health monitoring should be anonymized and not convey user identity information [7]. Figure 1 shows an overview of the main application scenarios of biometric handwriting in both security and health fields. It is worth pointing out that this is not a biometric-specific problem. For instance, in [8], similar problems are discussed in relation to loyalty cards provided by supermarkets. The advantage for the customer is that he gets discounts and points that can be exchanged for prizes. The advantage for the company is that it can rationalize the products’ distribution along the supermarket and take better care of their prices’ politics, stock, etc. However, consumers are not aware about the risk of permitting that the supermarket stores this information. A secondary/double use could be other companies willing to pay for information about which kind of products is purchased by a given person (tobacco, alcohol, drugs, etc.), which kinds of videos are rented, how much money spends each month, etc. Doubtless, consumption habits are more personal and private than some biometric traits, and special care must be taken for this information, although the offers of the companies seem quite tentative.

The challenges and opportunities of handwriting biometrics for e-security and e-health outlined before are particularly well suited to be advanced in the framework of cognitive computing. Combined work considering at the same time both disciplines (biometrics and cognitive computing) can be seen in a few selected works in the past [9‐12], but still many synergies between them are to be exploited in future research and development.

According to the Cognitive Computing Consortium, cognitive computing systems must have five key attributes, as listed below [13, 14]:

A handwriting biometric system should be aware and take advantage of the rich amount of information that can be extracted from a very simple handwriting task, as described in the next sections. This is because handwriting is a cognitive task in which synchronized neuromotor orders are fired from the cortex to carry out the planned action [15, 16]. Signal processing and pattern recognition techniques applied to analyze this procedure can reveal very useful information about the writer, including its identity and health state.

The present paper summarizes the current research in handwriting biometrics with applications to e-security and e-health under the umbrella of cognitive computing principles: adaptiveness, interactivity, iterativity, and context. We focus our discussion on the accuracy, performance, utility, application factors, and challenges of handwriting analysis in such application scenarios, also foreseeing trends and potential research lines for the future. More comprehensive surveys of the state of the art in handwriting analysis for security and health can be found elsewhere [3, 517‐19].

The acquisition devices considered in online handwriting allow to capture various properties of the moving pen (or fingertip [20]) during the whole writing process in real time. These time-stamped sequences of points typically give the digital representation of the signal. For instance, digitizing tablets typically acquire the following information:

An example of this information is given in Fig. 2 for both healthy and unhealthy handwritings.

From this set of dynamic raw data, further information can be inferred or derived, which is usually more suitable for certain applications (e.g., handwriting velocity, duration, width, height) [22, 23]. Some digitizing devices such as Wacom Intuos tablet and Samsung Galaxy Note are able to track the pen tip movement even when it is not touching the surface (see Fig. 3). Thus, it is possible to record the X and Y coordinates of in-air movements when pressure is null. Unfortunately, this is only possible when the distance between the tip of the pen and the surface is less or equal to approximately 1 cm; otherwise, the tracking is lost. Nevertheless, the in-air time spent is still known because the acquisition device provides a timestamp for each sample [24]. In-air information cannot be obtained with “classic” off-line methods (scanning the ink-pen handwriting once the task is finished), and the timing information is also lost or very difficult to recover [24, 25].

An accurate complete recording of in-air information requires other 3D sensors. Possibly, the more appropriate for handwriting are the Motion Capture (MOCAP) sensors. These sensors are based on either video or inertial measurement units (IMUs), i.e., accelerometer, gyroscope, and magnetometer. The first ones, e.g., Vicon system (www.​vicon.​com), use the image from multiple cameras to calculate the 3D position of the target, which could be the tip of the pencil or a fingertip, where a marker is attached. The second ones are based on IMUs, such as neuronmocap (https://​neuronmocap.​com), which require a glove in which sensors are plugged in to track the 3D trajectory. In both cases, it is possible to sample the position of the marker or IMUs around 100 times per second (i.e., 100 Hz). The major problem of the first ones is the occlusion while the downside of the second is the offset drift of the accelerometers. Besides, the first one must be used in a fixed space while the second is portable. Anyway, both procedures are reliable systems for 3D tracking which, although not massively used in nowadays handwriting biometric applications, is a promising research line to be explored. We have everything in hand to design 2D and 3D systems from the acquisition hardware point of view. While 2D models and algorithms are a reality, the potential of 3D systems is yet to be explored.

According to [26], verification and recognition are two classical studies carried out with handwriting specimens. In the context of biometric security, applications based on handwriting tasks are mainly based on signatures. Several international competitions summarize the state of the art achieved by dozens of teams, such as BSEC [27], SVC [28], and SigWIComp [29]. Although less known, there are also some studies (e.g., [30‐32]) where biometric recognition is based on handwritten text, either text dependent or independent.

Several authors have demonstrated the individuality of handwriting. It has been assessed in the offline case, for example, in [21]. These researchers collected a database of 1500 writers selected to be representative of the US population and conducted experiments on identification and verification. Regarding writer identification, they reached an accuracy of about 83% at the word level (88% at the paragraph level and 98% at the document level). These results allowed the authors to conclude that the individuality hypothesis, with respect to the target population, was true with a 95% confidence level. Another study [30] complemented the previous work of [32]. They analyzed the individuality of four handwritten words (been, Cohen, Medical, and referred) taken from 1027 US individuals who wrote each word three times. The combination of the four words yielded an identification accuracy of about 83% and a verification accuracy of about 91%.

With regard to the online case, some authors have addressed the issue of the individuality of single words and short sentences. Hook et al. [31] showed that single words (the German words auch, oder, bitte, and weit) and the short sentence Guten Morgen exhibit both considerable reproducibility and uniqueness (i.e., equal items written by the same person match well while equal items written by different people do not match so well). They used a small database consisting of 15 writers that produced, in a single session, ten repetitions of each item captured by a prototype of a digitizing pen. In [33] the English words February, January, November, October, and September were used (25 repetitions of each word donated by 45 writers). The identification rate reached 95%. In [34] a writer identification rate of 92.38% and a minimum detection cost function of 0.046 (4.6%) were achieved with 370 users using just one word written in capital letters. Results were improved up to 96.46% and 0.033 (3.3%) when combining two words. In [35], signatures, initials, and keywords were compared.

More recently, in [20] writer verification rates of 96.2% were achieved for handwritten passwords consisting of 7 digits written with the fingertip over the touchscreen of COTS smartphones and 94.1% for 4-digit passwords. In a subsequent study, the same authors were able to obtain similar authentication accuracy for more challenging and realistic acquisition conditions (in the wild including cross-device comparisons over a diverse pool of COST smartphones consisting of 94 different models) by considering characters and symbols instead of only numerical digits to construct the passwords [36].

In a separate line of research, some authors have also explored handwriting information from drawings and other graphical information generated with stylus- or finger-based inputs for authentication [37], usually resulting in worse performance as compared to natural handwriting or signature data [38]. Anyway, the results obtained based on graphical inputs may be acceptable for some low-security applications or helpful as a second authentication factor (e.g., in smartphone login applications [39]).

It is also worth pointing out some forensic challenges approached mainly in offline handwriting. The two main tasks addressed by forensics are forgery detection and disguising. Forgeries refer to a writer imitating the handwriting of other writers while disguising means a writer trying to conceal his usual writing habits and to make the new writing as different as possible to his own writing [40]. An open question is, knowing that a handwriting specimen is a forgery, how to detect who has forged the handwriting. Some attempts have been made [41] to track this problem but with poor results. Regarding disguising, this problem has also attracted much attention in the forensic area. This problem arises when the author changes his own handwriting. The purpose could be for the later denial of the own handwriting, in which case it is also termed as “auto-simulation.” Note that both genuine and disguised texts are written by the author of the specimen but with different intentions. A writer produces a genuine text that allows author identification. Whereas a disguised handwriting is written to make it look like a forgery for a possible later denial. These cases have been mainly studied in handwriting signatures [42, 43]. The difficulty of forensic-based challenges can be faced by online systems. Since they have the advantage of having the timing information, it is expected better results in a near future.

Although most of the many existing studies related to handwriting and handwritten signatures have been based on on-surface movements (see, e.g., [1, 18]), some studies have pointed out the importance of in-air movements as well. In [44] the authors performed an entropy analysis of handwriting samples acquired from a group of 100 people (for more information, refer to BiosecurID database [45]) and observed that both types of movements contain approximately the same amount of information. Moreover, based on the values of mutual information, these movements appear to be notably non-redundant. This property has been advantageously used in several fields of science. For instance, the authors in [21, 46] proved that in-air movement increases the accuracy of Parkinsonic dysgraphia identification. Specifically, when classifying the Parkinsonic dysgraphia by support vector machines (SVM) in combination with the in-air features, they reached 84% accuracy which is 6% better in comparison with classification based on on-surface features only. When combining both feature sets, they observed 86% classification accuracy. Similar accuracy rates were obtained in a different study by [47], which analyzed various handwriting tasks to discriminate between Parkinson’s disease (PD) vs. elder healthy controls. Better accuracy rates around 96% were also obtained in [47] when comparing PD vs. young healthy controls.

The in-air movement also supports the diagnosis of Alzheimer’s disease (AD) [48]. These researchers observed that patients with AD spend 7 times longer in-air when compared with a control group. In the case of on-surface movement, it is only 3 times longer. Similarly, [49] found out that the in-air duration can be a good measure for performance analysis of children with a high-functioning autism spectrum disorder. The in-air movement has also been used for the identification and the quantitative analysis of developmental dysgraphia in child population [50‐52]. In [50], it was proved that kinematic features derived from this kind of movement (especially jerk, which is the rate at which the acceleration of a pen changes with time) provide good discrimination power between children with dysgraphia and the control group.

On the other hand, there is a successful research line in handwriting analysis for health applications based on kinematic models. There exist many theories that have tried to model the speed profile of human movement in general and handwriting in particular [53]. Among the models which provide analytical representations, the kinematic theory of rapid human movements [54‐56] and its delta- and sigma-lognormal models have been used to explain most of the basic phenomena reported in classical studies on human motor control [57] and to study several factors involved in the fine motricity [58].

As can be seen in Table 1, the sigma-lognormal model represents a breakthrough due to its feasibility and reliability to describe a wide range of human movements. One of the main advantages of this model is that it considers physical body features such as the neuromuscular system responsible for the production of human movements and thus reflects some personal characteristics difficult to impersonate. To work out the sigma-lognormal parameters, the robust Xzero algorithm implemented by the ScriptStudio application was proposed in 2007 [78] and iDeLog was proposed in 2020 [79]. Since its first version, it was quickly widespread, and several improvements have been published, for instance, [80]. From a fundamental perspective, the powerfulness of this methodology relies on the lognormality principle [81] which states that the lognormal velocity profiles observed in handwriting and signature reflect the behavior of individuals who are in perfect control of their movements [22]. As a corollary, motor control learning in young children can be interpreted as a migration towards lognormality while, as aging and health issues intensify, a progressive departure from lognormality is occurring. For the greater part of their lives, healthy human adults take advantage of their lognormality to control their movements efficiently [82].

For a recent review of handwriting analysis to support neurodegenerative diseases diagnosis, see [3, 17, 46, 83].

Behavioral biometrics, in addition to security and health applications, can provide additional information, known as metadata. Sometimes also referred to as soft biometrics [84], they can be based on system hardware specifics (technical metadata) or on personal attributes (non-technical metadata) [85, 86]. System-related metadata represent the physical characteristics of biometric sensors and are essential for ensuring the minimum acceptable quality of the raw biometric signals. Previous work in personal related metadata has shown that it is possible to estimate some metadata like script language, dialects, origin, gender, and age by statistically analyzing human handwriting.

As examples of gender recognition based on handwriting, we can mention the following works. In [10] using only four repetitions of a single uppercase word, the average rate of well-classified writers is 68%; with sixteen words, the rate rises to an average of 72.6%. Statistical analysis reveals that the rates mentioned above are highly significant. In order to explore the classification potential of the in-air strokes, these were also considered. In that case, results were not conclusive, with an average of 74% well-classified writers when information from in-air strokes was combined with information from on-surface ones. This rate is slightly better than the one achieved by calligraphic experts. However, we should keep in mind that this is a two-class problem and even by pure chance we would get 50% accuracy.

A system that classifies handwriting into demographic categories using measurements such as pen pressure, writing movement, stroke formation, and word proportion has been proposed in [17]. The authors reported classification accuracies of 77.5%, 86.6%, and 74.4% for gender, age, and handedness classification, respectively. In this study, all the writers produced the same letter. Authors in [87] also addressed the classification of gender and handedness in the online mode. The authors used a set of 29 features extracted from both online information and its offline representation and applied SVM and Gaussian mixture models (GMM) to perform the classification. The authors reported accuracy of 67.06% for gender classification and 84.66% for handedness classification. In [88], the researchers separately reported the performance of the offline mode, the online mode, and their combination. The accuracy for the offline mode was 55.39%.

Emotional states, such as anxiety, depression, and stress, can be assessed by the Depression Anxiety Stress Scales (DASS) questionnaire. A new database that relates emotional states to handwriting and drawing tasks acquired with a digitizing tablet has been presented in [89]. Experimental results show that the recognition of anxiety and stress was better than depression recognition. This database includes samples of 129 participants whose emotional states were assessed by the DASS questionnaire and is freely distributed for those interested in research in this line.

Security and health applications have been discussed in the previous sections as fields isolated from each other. They are indeed studied separately for most of the scientific community. However, in some cases, both should be fully considered jointly as they cannot be completely isolated one from the other (see Fig. 4). This is particularly true in the context of the Personal Digital Bodyguard vision [5] which aims at supplementing people’s sensitive data protection with signature and writer verification.

Most biometric security applications only try to determine the identity of a subject or to verify if he/she is the person who claims to be. However, in the context of the acquisition of biometric signals with different devices and data quality [54, 90], it may be beneficial to gather some additional information for the system to be context-aware [12], as envisioned by the cognitive computation initiative. In the following, we summarize three of such possible scenarios:

A robust biometric security system should be able to detect the stress situation based on characteristics that cannot be easily controlled by the user. Detection of user stress from signature or handwriting is a challenging research topic that can indeed improve security systems.

Another possibility is to be affected by depression. Drawings for analyzing depressive disorders in older people were used in [92].

These two examples indicate that even if in the context of biometric signature verification one can conclude that the signature is genuine, this may not be enough. One may in addition take into account aspects such as the health state of the subject. Considering both aspects (identity and health), it can be concluded that, from a legal point of view, a signature is not valid because the subject may not have been in a sound state of mind. In such situations, the biometric authentication of the individual does not solve the problem and some additional considerations should be taken into account. This is not just related to health. Another similar situation where a genuine biometric sample is used fraudulently is a replay attack. In a replay attack, the biometric signal is usually genuine, but it was acquired/recorded in the past and presented again and should be considered as a fake attempt [93].

One of the main concerns of biometrics applied to security is privacy [8]. Technological advances allow to store, gather, and compare a wide range of information on people. Using identifiers such as name, address, passport, or social security number, institutions can search databases for individuals’ information. This information can be related to salary, employment, sexual preferences, religion, consumption habits, medical history, etc. This information can be collected with the consent of the user, but in some cases, it could also be extracted from biometric samples without the knowledge of the user. Thus, the user could be completely unaware that some additional and private information can be extracted from his biometric samples [98]. Therefore, there is a potential risk.

Let us think, for instance, in sharing medical information. Obviously, in the case of an emergency, this sharing among hospitals would be beneficial. On the contrary, if this information is transferred to a personal insurance company or a prospective employer, the insurance or the job application can be denied. The situation is especially dramatic when biometric data collection is intended for security applications to grant access to a facility or classified information, but a third party tries to infer the health condition of the subject. For instance, in the case of eye biometrics [92], an expert can determine that a patient has diabetes, arteriosclerosis, hypertension, etc.

For any biometric identifier, there is a portion of the population for which it is possible to extract relevant information about their health, with similar implications to the ones described in the previous paragraph. This is not a specific problem of handwriting signals. Some other biometric signals exhibit the same potential problems, for example, speech disorders and hair or skin color problems. An important question is what is exactly disclosed when biometric scanning is used. In some cases, additional information not related to identification might be obtained. One possible scenario could be a company where an attendance system requires workers to sign each day. The main purpose of this task could be to check if the worker is at his workplace during the working days. However, once the handwriting is provided, the company could decide to analyze the signature to detect some pathologies or drug abuse and to dismiss those workers who do not show good health. And last but not least, once we provide our biometric samples, they can stay in a database for dozens of years and due to technological advances, they can be used in a simple way in the future to extract additional information that was not intended during acquisition. For this reason, we should think about technical solutions to preserve privacy and legal regulations to avoid such foreseeable issues.

Sometimes the situation is just the opposite. With the growth of interest in the fields of e-health and telemedicine, scientists started to develop automatic handwriting analysis systems that can be used for disease diagnosis [99], rating, or monitoring [82]. Basically, a database of healthy control samples from healthy individuals and pathological samples is acquired, disseminated and used for research purposes. Its registers are usually anonymized removing (destroying) any link to donor names and/or passports or national identity codes. However, this anonymization process could be useless if the acquired samples permit a biometric identification based on handwriting or drawings. This might not be feasible at present, but it might be possible in the near future with the improvement of the processing algorithms.

A related topic is de-identification for privacy protection in multimedia content. De-identification in multimedia content can be defined as the process of concealing the identities of individuals captured in a given set of data (images, video, audio, text), to protect their privacy. This will provide an effective means for supporting directives and laws related to personal data protection, e.g., the EU’s General Data Protection Regulation (GDPR).

Security and health applications of handwriting analysis cannot be considered as separate fields anymore. While significant scientific developments exist in the field of biometric recognition using the handwritten signature for security [4, 18], few efforts have been made in general handwriting [20] and drawing [38] for security. Few efforts have also been deployed in health applications [3, 17, 47, 83, 99] as well as in possibilities of combining security and health.

As a summary of the research discussed in the present paper, on the one hand, the state of the art in applications based on online handwritten tasks can be considered:

On the other hand, it is difficult to find published research in joint applications where security is studied in combination with one of the following health-related aspects:

Similarly, there is a lack of research in applications where health is the main goal and the following issue is addressed at the same time:

For the future, we see four main areas of research in online handwriting analysis for security and health, as shown in Table 2:

These four areas of future research can be further broken down as follows:

In order to advance these research lines, adequate and realistic handwritten data is needed for experimentation. That data should necessarily involve several acquisition sessions, approved by an ethical committee, in order to cope with the temporal variability of the users as well as different tasks:

Fortunately, some databases are already available for research in several scenarios:

In this paper, we have pointed out future trends and challenges in biometric research on signature and handwriting. Special emphasis is given to the fact that, contrary to other biometric traits, handwriting signals are of interest in both e-security and e-health. Some challenges are identified that should attract the interest of the research community towards a more secure society.

We consider that this paper could help future researchers working in handwriting analysis to identify the main research topics to be addressed in the next years, as well as to attract new researchers from other fields.