Continuous estimation of emotional change using multimodal responses from remotely measured biological information

Since emotions have an inordinate amount of influence on daily life behaviors such as perception, memory, and decision making, it has recently become necessary to consider user emotions in the field of human–computer interface (HCI). This is especially important for future HCI applications that are expected to work in concert with humans. In this paper, we introduce an approach for estimating the emotions of video viewers in chronological order from remote measurements captured using a red–green–blue (RGB) camera. Facial expression and physiological responses such as heart rate and pupil were measured by analyzing facial videos. In addition, for the purpose of improving the estimation accuracy of emotional states and ease of measurement, we analyze the relationship between the features obtained from brain waves and the physiological signals acquired in a non-contact measurement. In this study, the Pearson’s correlation coefficient (COR) between the continuous Arousal estimated signal using the features from the electroencephalograph, which is the conventional method, and the subjective evaluation value was 0.1391, while the COR between the Arousal estimated signal and the subjective evaluation value using the features of facial expression, pupil diameter and heartbeat obtained from the camera without contact was 0.1609. In valence, the COR between the estimated signal using EEG features and the subjective evaluation value was − 0.0130, while the COR between the estimated signal from non-contact biometric features and the subjective evaluation value was 0.2879. In both cases, the proposed method improved the accuracy of continuous emotion estimation.