main-content

This book focuses on a development for assessing mental changes using eye pupil reactions, namely extracting emotional change from the response to evaluate the viewer's interest in visual information.

The pupil of the eye reacts to both brightness and emotional state, including interest, enjoyment, and mental workload. Because pupillary change is a biological signal, various artifacts influence measurements of eye images. Technical procedures are required to extract mental activities from pupillary changes, and they are summarized here step by step, although some procedures contain earlier techniques such as analog video processing.

This study examines the possibility of estimating the viewer's interest and enjoyment of viewing movies by measuring the dynamic pupillary changes, blinking, and subjective interest responses. In evaluation of pupil size, there was a significant difference in pupil size between the higher and the lower shot for the degree of subject interest response in each kind of movies.

The first part of the book shows a pupil reaction model for brightness changes to extract mental activities. Pupil reactions were observed for various visual stimuli in brightness changes. With regard to the characteristics of pupillary changes, a model with a three-layer neural network was developed and the performance was evaluated. Characteristics of pupil reactions during model development are summarized here.

The second part examines the possibility of estimating the viewer's interest and enjoyment of television programs by measuring dynamic pupillary changes, blinking, and subjective interest responses.

The final part describes a development of estimation model of pupil size for blink artifact. The model development was able to estimate pupillary changes and pupil size while the viewer was blinking and was applied to pupillary changes in viewing television programs.

### Controlling the Effects of Brightness on the Measurement of Pupil Size as a Means of Evaluating Mental Activity

Abstract
It is well-known that pupil size responds to both brightness and mental activity. However, the correlation between these two phenomena is not clear. This study is about the changes in pupil size in response to mental activity while the effects of brightness are controlled. As a first step, pupillary changes were measured at various levels of brightness, with verbal instructions that were designed to stimulate the mental activity being given and not given. No interaction between pupillary change due to the effects of brightness and pupillary change due to mental activity was found. As a second step, white, gray, and black patterns were presented to the subjects, and pupil sizes were measured at varying levels of brightness. From these measurements, it was possible to develop an an experimental formula that expresses the relationship between pupil size and brightness. Next, the results of the first experiment were compensated using the extracted function, and analysis of variance (ANOVA) showed that brightness did not have an effect on pupil size. Therefore, a method for removing the effects of brightness upon pupillary changes was developed. Finally, the extracted function was applied to the evaluation of pupil size as a function of mental activity for the patterns presented at several levels of brightness. Corrected pupil sizes correlated with pupil sizes when patterns of pictures were presented at the same levels of brightness.
Minoru Nakayama, Ikki Yasuike, Yasutaka Shimizu

### Pupil Reaction Model Using a Neural Network for Brightness Change

Abstract
Pupil reaction models for brightness change are developed in order to introduce emotional pupillary changes into the evaluation of a video. The models are designed with experimental pupil reactions to temporal changes in brightness using a linear model and a layered neural network model. Their performance in reproducing the training data, and the possibility of applying these models to short video clips are evaluated.
Shigeyoshi Asano, Ikki Yasuike, Minoru Nakayama, Yasutaka Shimizu

### A Neural-Network-Based Eye Pupil Reaction Model for Use with Television Programs

Abstract
If the effects of variations in brightness can be controlled, it is possible to use changes in pupil size for the evaluation of educational TV programs. This paper describes an improved, neural-network-based method for studying pupil reactions and the feasibility of evaluating TV programs by investigating changes in pupil size. This neural-network pupil reaction model uses M-sequences and Markov-sequences to register changes in pupil size based upon responses to changes in brightness, and this paper shows it to be more effective in removing the effects of brightness on pupil size than other previously reported methods. The authors also discuss methods for removing the influence of brightness from pupil response, as well as methods for evaluating changes in pupil size. It is also shown that this new method permits the objective evaluation of TV programs through the measurement of changes in pupil size while viewing TV programs.
Shigeyoshi Asano, Minoru Nakayama, Yasutaka Shimizu

### An Estimation Model for Pupil Size of Blink Artifacts While Viewing TV Programs

Abstract
It is well known that pupil size measurement is influenced by noises and blinks. This paper describes the development of an estimation model for the pupil size of blink artifacts. The development of the model was based on a three-layered perceptron, using a backpropagation method. First, the model was trained using pupil responses to changes in brightness, after which the model could estimate pupil temporal changes. Second, the model was trained using pupil responses with artificial blinks. The model could also estimate pupillary changes and pupil size during blinks. The model was applied to pupillary changes during TV program viewing. The accuracy of removing the influence of brightness was evaluated using one-way ANOVA. The pupil size sampling rate was improved from 3 to 30 Hz using the processing model.
Minoru Nakayama, Yasutaka Shimizu

### Estimation of Eye-Pupil Size During Blink by Support Vector Regression

Abstract
Pupillography can be an index of mental activity and sleepiness, however, blinks prevent its measurability as an artifact. A method of estimation of pupil size from pupillary changes during blinks was developed using a support vector regression technique. Pupil responses for changes in periodic brightness were prepared, and appropriate pupil sizes for blinks were given as a set of training data. The performance of the trained estimation models was compared and an optimized model was obtained. An examination of this revealed that its estimation performance was better than that of the estimation method using MLP. This development helps in the understanding of the behavior of pupillary change and blink action.
Minoru Nakayama

### Frequency Analysis of Task Evoked Pupillary Response and Eye Movement

Abstract
This paper describes the influence of eye blinks on frequency analysis and power spectrum difference for task-evoked pupillography and eye movement during an experiment which consisted of target following tasks and oral calculation tasks with three levels of task difficulty: control, 1$$\times$$1, and 1$$\times$$2 digit oral calculation. The compensation model for temporal pupil size based on MLP (multilayer perceptron) was trained to detect a blink and to estimate pupil size by using blink less pupillary change and artificial blink patterns. The PSD (power spectrum density) measurements from the estimated pupillography during oral calculation tasks show significant differences, and the PSD increased with task difficulty in the area of 0.1–0.5 and 1.6–3.5 Hz, as did the average pupil size. The eye movement during blinks was corrected manually, to remove irregular eye movements such as saccades. The CSD (cross-spectrum density) was achieved from horizontal and vertical eye movement coordinates. Significant differences in CSDs among experimental conditions were examined in the area of 0.6–1.5 Hz. These differences suggest that the task difficulty affects the relationship between horizontal and vertical eye movement coordinates in the frequency domain.
Minoru Nakayama, Yasutaka Shimizu