Anomaly detection in facial skin temperature using variational autoencoder

VAE is a generative model based on deep learning. Figure 1 shows overview of the VAE algorithm. X, \(\widetilde{X}\), NN represent input, output, and a neural network, respectively. The VAE network is divided into an encoder section and a decoder section. Given observation \(X = \{\overrightarrow{x_1}, \overrightarrow{x_2},\ldots , \overrightarrow{x_N}\}\), VAE identifies probability distributions (\(p(\overrightarrow{x_*}|X)\)) that produce unobserved value (\(\overrightarrow{x_*}\)). VAE is designed on the assumption that latent variables that serve as explanatory variables are normally distributed. The encoder performs dimensional compression of X, and it calculates the mean vector (\(\overrightarrow{\mu _{\phi }}\left( x\right) \)) and variance (\(\Sigma _{\phi }\left( x\right) \)), which are parameters of the normally distribution. The blue part in Fig. 1 indicates that sampling is performed from the standard normal distribution. The points are then sampled from the latent space distribution (\(\overrightarrow{z}\)). In the decoder, the model likelihood parameters (\(\overrightarrow{\eta _{\theta }}\left( z\right) \)) are calculated, and the reconstruction error can be computed. Finally, the reconstruction error is backpropagated through the network. Since the reconstruction error to be optimized includes a regularization term that brings the mean (\(\overrightarrow{\mu _{\phi }}\left( x\right) \)) to 0 and the variance (\(\Sigma _{\phi }\left( x\right) \)) close to the unit matrix, the distribution of the latent variable \(\overrightarrow{z}\) has a shape close to a standard normal distribution. There is a tendency to regularize the organization of the latent space by bringing the distribution returned by the encoder closer to the standard normal distribution. For this reason, VAE can avoid overfitting and achieve a high recall compared to a normal autoencoder.

We explain the concept of anomaly detection in thermal face images using VAE. Figure 2 shows the conceptual diagram. Normally, with machine learning, it is necessary to have a dataset with a complete number of samples for each class. However, it is difficult to collect data when a person is abnormal (e.g., data on people who are not feeling well). Therefore, in this study, an evaluation of the model constructed using only the data in the normal condition, which is relatively easy to collect, was performed. In this study, we defined thermal face images in normal state as Normal, in anomaly state as Anomaly. Only Normal is used for anomaly detection using VAE. First, by learning a large amount of Normal using VAE, the anomaly detection model was constructed. Second, testing data (Anomaly and Normal) were input to the anomaly detection model. If the pattern was similar to Normal, the testing data were decided Normal; otherwise, the testing data were decided Anomaly.

Anomaly detection using VAE requires a large amount of normal data. In this study, we verify whether the proposed algorithm is effective in anomaly detection in facial skin temperature. Anomaly is thermal face images in which the skin temperature is forcibly changed by holding the breath. The Normal were thermal face images obtained when the subject was in the normal state. Image sizes are 640 \({\times }\) 480 pixels. The number of normal data was 4976, of which 90% was used as learning data and 10% was used as testing data. The number of Anomaly serving as the testing data was 195 in accordance with the testing data of Normal. The difference in the facial skin temperature value in a single image is small. The thermal images may also be biased by room temperature and other disturbances. In this study, the thermal face image was normalized, such that the maximum temperature was 1 and the minimum temperature was 0 for learning the relative amount of skin temperature inside the face. Specifically, the 640 \({\times }\) 480 pixel thermal image output from the thermographic device was cropped to 180 \({\times } \)180 pixels, leaving the face area. This thermal image after cropping is defined as a thermal face image. The normalization was applied independently to each of thermal face images.

When performing VAE learning, a part of the thermal face image is randomly cut out at 8 \(\times \) 8 size pixels, and this patch is used as learning data, as shown in Fig. 3. The local thermal image of the skin area was expanded to 10,000 sheets. Figure 4 shows overall of the anomaly detection model. In this study, convolutional layers were placed before the FC layer of encoder to extract the features of the skin temperature pattern of the skin blood vessels. Along with that, deconvolutional layers were placed after the decoder. The construction of the encoder is depicted in Table 1. The structure of the VAE encoder consisted of two convolutional layers and one fully connected layer. In this table, Conv, BatchNorm, and FC indicate the convolutional, batch normalization, and fully connected layers, respectively. The mean vector and variance were output from FC. The structure of the decoder is paired with the structure of the encoder, and the structure is opposite to that of the encoder. The construction of the decoder is depicted in Table 2. ConvTrans indicates a transpose convolution. The gradient descent method was used for VAE parameter learning, and the optimization algorithm at that time was Adam. The number of dimensions of the latent variable \(\overrightarrow{z}\) was searched from 4, 5, 6, 7, 8, 10, and the best model was selected. The number of epochs was 20 and the batch size was 128.

When testing, for all test data, the spatial unregularized anomaly score was calculated with reference to [25] and used as an index for detecting facial thermal images with some kind of abnormality from multiple facial thermal images. Unregularized anomaly score represented the error between the input vector and the reconstructed vector. To use the unregularized anomaly score to determine Anomaly, it is necessary to set a threshold mostly. However, the threshold is not defined in this study. In this paper, the statistics of Normal and Anomaly of unregularized anomaly scores were calculated to see whether it is possible to separate Normal and Anomaly. The equation for the spatial unregularized anomaly score is shown below:The above equation is directly related to the reproduction error. Nx and \(x_{i}\) represent the number of pixels in the image and any pixel value, respectively. \(\mu _{xi}\) represents the maximum posterior probability estimation of the latent variable \(\overrightarrow{z}\).

The experimental systems consisted of an infrared thermography device (FLIR A600-Series, FLIR systems Co., Ltd). The size of the thermal image was 640 \(\times \) 480 pixels, and the temperature resolution was less than \(0.05\,^\circ \)C. The infrared emissivity of skin was \(\varepsilon \) = 0.98. The viewing angle of the infrared thermography is \(60^\circ \).

Healthy young subject (male; aged, 24) participated in the experiments. The subject provided informed consent about the experiments and objects of this study prior to agreeing to participate in the experiment. For the introduction to the real environment, subject was asked to cooperate in the experiment as usual as possible without modifying their daily activities such as food intake, sleep, smoking habits, etc. The experiment was conducted in an experimental room (24.7 \(\pm \,{6.0}\,^\circ \)C). The ultimate goal of this study is to construct a system that recognizes the normal state of facial skin temperature for estimating driver drowsiness and checking stress in daily life. Therefore, we conducted the experiment in the room, which the environment temperature was not controlled, to simulate the daily life environment. In the experiment, the skin temperature of the face was measured twice. The sampling frequency of thermal images was 10 Hz. An infrared thermograph was placed 1 m in front of the subject. In the first time, the subject did not control anything, and they sat in a chair for about 10 min, with their face as still as possible. This thermal face image measured for about 10 min was regarded as Normal. The second time, the subject held his breath held for 50 s, which raised the blood pressure to promote fluctuations in facial skin temperature. The measurement was performed in iterations of 30–50 s, and the thermal face image obtained at this time was regarded as Anomaly.

Figure 5 shows examples of normalized thermal face image samples.

Figures 6 and 7 show the unregularized anomaly scores of two arbitrary samples. The unregularized anomaly scores are mapped on a log scale. The blue color indicates the degree of abnormality. When the unregularized anomaly scores were observed for all test data, they were categorized into two types of map, Figs. 6 and 7. In Fig. 6, the input thermal images are samples that vary significantly across facial skin temperature. When observing unregularized anomaly score, there is no abnormal part in the entire face in the normal and abnormal parts in the nose, the left and right cheeks, and around the mouth in the anomaly space.

This is suggested that VAE learned a large number of samples in which no part of the entire face had a significant change in temperature. In Fig. 7, the nose was abnormal even in the normal space, but parts other than the nose were not recognized as abnormal. As a result, the VAE recognized a portion where the skin temperature was lowered as an abnormal portion and a portion where the skin temperature did not change as a normal portion. That is, the VAE algorithm was effective in capturing changes in temperature within the face. When constructing the anomaly detection model, we only use the thermal face images of Normal. In other words, the unregularized anomaly score calculated with the model that only learns Normal may be able to identify the state of the test image (Normal or Anomaly) even if it is not known whether the test image is Normal or Anomaly.

The histograms of unregularized anomaly score shown in Figs. 6 and 7 are shown in Figs. 8 and 9, respectively. The horizontal axis represents unregularized anomaly score and the vertical axis represents frequency of unregularized anomaly score. The vertical axis of these histograms is plotted on a log scale. For both samples, the general shape of the histogram distribution varied significantly. Abnormality patterns are categorized into two types, both of which have different distributions for anomaly and normal; therefore, the same can be said for all test data. Figure 10 shows the average and variance of the unregularized anomaly score for all test data. Since the anomaly space deviates from the normal space, the proposed algorithm is useful for detecting abnormal skin temperature fluctuations.