Assessment of human operator functional state using a novel differential evolution optimization based adaptive fuzzy model

https://doi.org/10.1016/j.bspc.2011.09.004Get rights and content

Abstract

With the development of human–machine systems, there has been a growing concern about the consequences of operator performance breakdown under excessive level of workload, especially in safety-critical situations. Assessment and detection of the operator functional state (OFS) enable us to predict the high operational risks of operator. This paper adopts the psychophysiological signals and task performance measures to evaluate OFS under different levels of mental workload. Four indices extracted from electrocardiogram and electroencephalogram, including heart rate (HR), ratio of the standard deviation to the average of HR segment, task load indices (TLI1 and TLI2), are chosen as the inputs of the proposed model. A technique of differential evolution with ant colony search (DEACS) is developed to optimize the parameters of Adaptive-Network-based Fuzzy Inference System (ANFIS). The optimized ANFIS model is employed to estimate the OFS under a series of process control tasks on a simulated software platform of AUTOmation-enhanced Cabin Air Management System. The results showed that the proposed adaptive fuzzy model based on ANFIS and DEACS algorithm is applicable for the operator functional state assessment.

Introduction

During the modern development of automation technique, highly automated systems require few manual activities for individuals to perform. Human becomes the monitor and decision maker in the automatic system. Consequently, it brings more information processing loads and broader ranges of responsibilities to individuals. Therefore, the increased responsibility and decision making authority to each individual probably cause that the human operators become the weakest links in the operational loop [1], [2], [3]. The operator's performance degradation (because of vigilance decrease, fatigue, inattentiveness, sleepiness, pain, etc.) was the reason of some serious disasters [4]. To reduce this kind of disasters, researchers have begun to turn attention to the study of the performance of operators, particularly in safety-critical applications such as public transportation (railway [5], aviation [6], [7], driving [8]) and manufacturing industries (chemical [9] and nuclear plants [10]).

Adaptive automation has been proposed to solve the above problem. In adaptive automation, the control tasks can be reallocated dynamically between operators and automatic systems according to the operators’ performance level and whole system operation requirements. For example, when high operator mental workload is detected, some of the manual control tasks may be taken away and switched to the automatic system. On the other hand, high level of automation continuing for long time would result in lack of concentration for the operator. Proper manual control tasks should be reallocated to the operator.

There are three primary approaches to generate criteria for task reallocation [1]: (a) critical events logic where automation is engaged consistently in response to an environmental stimulus; (b) model-based approaches where automation is ‘scheduled’ based on a priori models of optimal operator performance; and (c) continuous assessment of operator function and mental state. For the third approach, continuous operator functional state (OFS) assessment is usually achieved through the following ways: (1) subjective assessments of the operator's mental workload; (2) performance of the operator on the primary task, and sometimes secondary tasks; (3) psychophysiological measurements of the operator. The authors’ current work is to investigate accurate and reliable assessment of OFS by psychophysiological measurements.

The psychophysiological signals, including electrocardiogram (ECG), electroencephalogram (EEG), eye blink, respiration, blood pressure, electrodermal activity, etc., have been demonstrated to be sensitive to the changes in OFS. Among those signals, the indices from ECG and EEG have been commonly used to assess the mental workload.

Heart rate (HR) and heart rate variability (HRV) are efficient indices of mental workload [11], [12], [13], [14]. In the previous works of ECG [15], [16], the authors found that HR and HRV1 (ratio of the standard deviation to the average of HR segment) did reflect the changes of operator mental workload.

EEG is another important signal for assessment of OFS. Researchers demonstrated that the oscillations in theta and alpha bands may reflect workload, attention, and performance levels of operator [17], [18], [19]. The increment in theta band power on the midline frontal sites as well as the decrement in alpha band power is relative to the mental workload and attention. Previous works have been done on analyzing the correlation of OFS with variables of EEG [20]. The task load index (TLI) [21], i.e. ratio of theta to alpha power from different sites, is selected for OFS assessment. In this study, HR, HRV1, and TLI (TLI1 and TLI2) are chosen as the inputs to the proposed model for OFS assessment.

Several studies have been done on the psychophysical signals and artificial neural networks (ANNs) to classify the levels of OFS [21], [22], [23]. Wilson and Russell [24] reported high accuracy of OFS classification using ANNs, i.e. 85%, 82%, and 86% for baseline, low and high task difficulty conditions, respectively. However, due to the characteristics of ANNs (the implicit knowledge representation), the relationship between input features and OFS was not explained.

Adaptive-Network-based Fuzzy Inference System (ANFIS) with the explicit knowledge representation was employed to estimate OFS [25], [26]. ANFIS can simulate and analyze the mapping relation between the input and the output data by a learning algorithm to optimize its parameters. It is widely used to build complex and nonlinear relationship between a set of input and output data [27], [28]. In [25], basic ANFIS was used for OFS assessment. The premise parameters and consequent parameters of ANFIS were identified by gradient descent method and least squares estimation, respectively [29]. The results showed that the generalization capability of ANFIS was not satisfied. Because of over-fitting, the basic ANFIS model output is not reliable in real application.

In this paper, a differential evolution algorithm with ant colony search (DEACS) is proposed to optimize the parameters of ANFIS. Differential evolution (DE) [30], [31] is a population-based stochastic optimization algorithm, which has the capability of global optimization and quick convergence. In DE algorithm, there are two significant control parameters: mutation factor and crossover factor. The two factors determine whether DE algorithm can find a good optimal solution. In the original DE algorithm, some recommended settings of the two factors were given through many optimization experiments for test functions. These are empirical settings from statistical results, and would be constant at different stages of evolution. To improve the performance of DE, some researchers developed new strategies to adjust the two factors by linear variation [32] and random selection [33], [34], etc. However, neither linear variation nor random selection is related to the evolution state. Therefore, the idea of ant colony search is employed to find out the proper combination of mutation and crossover factors adaptive to each individual according to current optimization performance. ACS [35] simulates the behavior of ant colony system. When ants prefer in probability to one path (i.e. a combination of mutation and crossover factors) since there are a greater quantity of pheromone on this path, the quantity of pheromone would be further increased. The positive feedback effect of ACS is applicable to selecting the combination of the factors as the most appropriate choice for DE. Instead of being constant or varying by some specific rules, the two factors can be determined adaptively. Applying ACS avoids the inefficiency which results from the improper selection of the two factors. The developed DEACS–ANFIS model is established to assess OFS. The obtained results are compared with ANFIS and DE based ANFIS (DE-ANFIS).

The rest of the paper is arranged as follows. In Section 2, the data acquisition experiment is described in details. Section 3 presents the ANFIS and DEACS algorithm, respectively, and establishes an adaptive fuzzy model based on DEACS–ANFIS for OFS assessment. In Section 4, the experimental results are illustrated, and the discussions are presented. Section 5 includes the conclusions and future works.

Section snippets

Data acquisition equipment

The experiments were carried out on AUTOmation-enhanced Cabin Air Management System (AUTOCAMS), which was developed by Hockey et al. [36] and modified by Lorenz and Parasuraman [37]. AUTOCAMS simulates a life support system of a spacecraft. It requires the subject (operator) to manage a semi-automatic system which is to regulate atmospheric conditions of the cabin so as to maintain comfortable atmospheric condition. There are five sub-systems (Fig. 1) corresponding to five critical parameters

ANFIS based on differential evolution with ant colony search

In this section, an adaptive fuzzy model is proposed. ANFIS is utilized to build the structure of the proposed model. An improved differential evolution with ant colony search is adopted to optimize the parameters of ANFIS.

Training and checking data

The DEACS based ANFIS model (DEACS–ANFIS) is used for OFS assessment. The inputs of the model include HR, HRV1, TLI1 and TLI2, which were found to be most sensitive to the changes of OFS [15], [16], [39]. TIR is selected to be the output.

Different from the works [25], [26] using 7.5 min, the sampling interval of inputs and output data is set to 2 min. The modified sampling interval allows more data information for analysis and facilitates the real-time assessment of OFS. However, it increased the

Conclusions

An adaptive fuzzy model, which is called DEACS–ANFIS, was proposed for operator functional state (OFS) assessment. Four psychophysiological signals, including heart rate (HR), heart rate variability (HRV1), task load indices (TLI1 and TLI2), were selected to be the inputs to the proposed model, and time in range (TIR) was the output. The ANFIS architecture provided a transparent assessment of OFS, and helped to easily understand the relationship between psychophysiological signals and OFS. The

Acknowledgements

The authors would like to thank the editor and anonymous referees for their valuable comments. They would also like to thank Dr. Bei Wang for her English correction and advice. This work was supported in part by the National Natural Science Foundation of China under Grant No. 61074113, 60775033 and 61075070, Shanghai Leading Academic Discipline Project B504, and Fundamental Research Funds for the Central Universities WH0914028. The 2nd author (J. Zhang) would like to gratefully acknowledge

References (41)

  • G.F. Wilson et al.

    Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding

    Hum. Factors

    (2007)
  • P. Averty et al.

    Mental workload in air traffic control: an index constructed from field tests

    Aviat. Space Environ. Med.

    (2004)
  • M.A. Recarte et al.

    Mental workload while driving: effects on visual search, discrimination, and decision making

    J. Exp. Psychol. Appl.

    (2003)
  • C.S. Connell

    Toward an understanding of DCS control operator workload

    ISA Trans.

    (1995)
  • M. Kumar et al.

    Fuzzy evaluation of heart rate signals for mental stress assessment

    IEEE Trans. Fuzzy Syst.

    (2007)
  • N. Hjortskov et al.

    The effect of mental stress on heart rate variability and blood pressure during computer work

    Eur. J. Appl. Physiol.

    (2004)
  • A.H. Garde et al.

    Effect of mental and physical demands on heart rate variability during computer work

    Eur. J. Appl. Physiol.

    (2002)
  • A.J. Tattersall et al.

    Level of operator control and changes in heart rate variability during simulated flight maintenance

    Hum. Factors

    (1995)
  • R.F. Wang et al.

    Process control operator's heart rate variability analysis based on wavelet packet transform

    J. Nanjing Univ. Sci. Technol.

    (2009)
  • J.H. Zhang et al.

    Use of heart rate variability analysis for quantitatively assessing operator's mental workload

  • Cited by (45)

    • Spatially-coded SSVEP BCI without pre-training based on FBCCA

      2023, Biomedical Signal Processing and Control
    • Decoding SSVEP patterns from EEG via multivariate variational mode decomposition-informed canonical correlation analysis

      2022, Biomedical Signal Processing and Control
      Citation Excerpt :

      The BCI system does not rely on the conventional muscles or neural pathways surrounding the brain but instead decodes specific neural patterns, typically measured by electroencephalogram (EEG), and translates them to computer commands for controlling external devices [1,2]. BCI has made great progress in medical treatment [3,4], military [5], entertainment [6], driver fatigue detection [7,8], and other fields. It not only brings good news to patients suffering from stroke, epilepsy, and mobility impairment [9–11], but it also provides convenience to people's lives [12,13].

    • SSVEP signal classification and recognition based on individual signal mixing template multivariate synchronization index algorithm

      2021, Biomedical Signal Processing and Control
      Citation Excerpt :

      BCI technology provides a new communication and control method for patients with physical inconvenience to realize wheelchair manipulation [3], prosthetic limb control [4], stimulation of nerve recovery [5] and other functions by using brain ideas. In addition, this technology has also been applied in virtual reality games [6], military communication control [7], operator fatigue state detection [8,9] and other medical fields. BCI systems can induce different EEG signals according to different experimental paradigms.

    View all citing articles on Scopus
    View full text