A cognitive brain–computer interface monitoring sustained attentional variations during a continuous task

Since its first definition by William James a century ago (James 1890), attention has been generally defined by its cognitive role in the information processing occurring in the brain, rather than by its neuronal substrate or physiological function. This way of defining attention is what makes it so complex and difficult to study (Richard 1980).

The first cognitive models of attention describe it in terms of filters (Treisman 1964; Deutsch and Deutsch 1963; Norman 1968). In these models, percepts are filtered according to their relevance to the organism to be driven towards working memory for conscious processing and possibly long-term storage. Afterwards, the notion of attentional load emerged: attention was modeled as a limited capacity system (Kahneman 1973), where limited mental resources are shared between the different information-processing tasks the brain has to run. Attention becomes responsible not only for selection of relevant activities, but also for inhibition of distracting stimuli. This model includes every cognitive task the brain has to run in his theory of attention and not simply the process of selecting between competing simultaneous stimuli, on which the earlier bottleneck theories had focused. The attentional capacity of a person can vary depending on his arousal (Berlyne 1960), which, in turn, can be influenced by the attended tasks—this is for instance one explanation of driver’s drowsiness (Chen et al. 2018; Zeng et al. 2018). Later on, attention models became modular and more executive (Johnston and Heinz 1978; Posner and Snyder 1975): attention is considered as a flexible mental process, where voluntary control of attention requires more resources than automatic attentional discrimination. Attention was finally modelled from a neurocognitive perspective (Posner and Petersen 1989; Petersen and Posner 2012), as a unified system for the control of mental processing involving a network of anatomical areas independent from the sensory input processing areas of the brain, with three main subsystems of attention: alerting, orienting and detecting.

Attention has several submodalities [e.g. overt or covert attention, focused attention, selective attention, divided attention, alternating attention, vigilance, see Gaume et al. (2015) for a detailed discussion]. We are interested specifically in sustained attention, i.e. the self-directed maintenance of cognitive focus in non-arousing conditions while possibly ignoring distractors and avoiding mind wandering (Roberton and Garavan 2004; Clayton et al. 2015; Gaume et al. 2016). This state of maintaining attention is related to what is called concentration outside the field of neuroscience. Some authors use the term sustained vigilance or just vigilance to designate this state (e.g. Warm et al. 2008), but it is more widely accepted that vigilance refers to a condition of sustained arousal with the purpose of ensuring that important or surprising percepts will not be missed (Gaume 2015) (whereas we are interested in this study in the maintenance of overt attention towards a non surprising or even boring stimulus without mind wandering).

A recent framework for visual attention (Knudsen 2007) defined attention as a competitive selection process. This framework is well-compatible with real-time modelling in cognitive BCI. It introduces an explicit relationship between attentional networks and the direct control of the sensory organs that generate the incoming stimuli. It also integrates the concept of salience filters to describe the set of brain networks that automatically estimate the pertinence of an incoming sensory input (Gaume et al. 2016; Gaume 2015). The present investigation was grounded in Knudsen’s attentional model, with a neuro-cognitive perspective coming from EEG modeling, where attention is considered as the result of three coordinated processes (Stuss et al. 1995; Clayton et al. 2015):The cognitive control involved in sustained attention tasks is correlated with frontal midline theta-band activity (Cavanagh and Frank 2014; Clayton et al. 2015; Wokke et al. 2017), congruent with [A] monitoring processes. Increased gamma-range oscillations are believed to promote task-relevant activity (Reinhart et al. 2011; Akimoto et al. 2013; Ahveninen et al. 2013; Potes et al. 2014; Clayton et al. 2015) congruent with [B] excitation processes. Alpha power has been linked with inhibition of task-irrelevant sensory modalities in task-irrelevant cortical areas (Makeig and Inlow 1993; Bollimunta et al. 2008; Snyder and Foxe 2010; Toscani et al. 2010; Anderson and Ding 2011; Mazaheri et al. 2014; Clayton et al. 2015; Wang et al. 2016), congruent with [C] inhibition processes.

Brain–computer interfaces (BCIs) are communication systems that enable a direct and real-time exchange of information between the brain and the external world [see Nicolelis (2011) for an introduction to this subject]. The first BCI development attempt, which also served as a proof of concept, was carried out in 1973 by Jacques Vidal and his team in California. Their experiment was called the “BCI project” and was meant to evaluate the feasibility and practicality of utilizing the brain signals in a man–computer dialogue (Vidal 1973). The goal of BCI systems is to create communication pathways that differ from the normal input/output channels used by the brain, namely the sensory organs to capture information about the world and the peripheral nervous system coupled with the muscles to interact with the environment (Wolpaw et al. 2000). The purpose of such alternative pathways is frequently viewed as a means of assisting in the rehabilitation of disabled or paralysed persons, to whom BCIs can be of great help by either replacing a defective sensory input or providing substitute ways to interact with the world. However, many other applications can emerge from the development of real-time brain signals decoding and stimulation techniques. They include applications of neurofeedback, and applications outside of the medical world, such as in BCI games (Marshall et al. 2013), performance analysis (Tharawadeepimuk and Wongsawat 2017), sonification (Sayan et al. 2019), silent communication devices¹ or ways to improve cognitive activity.²

Thorsten Zander introduced a slightly different classification in which exogenous BCIs are referred to as reactive while endogenous interfaces are separated into active and passive, respectively if the subject consciously triggers control signals or if the interface passively monitors the user’s brain state (Zander et al. 2010; Zander and Kothe 2011). To further refine this classification, we call sensory BCI a device with a control signal that is a correlate of sensory processing, motor BCI a device that uses activity from the motor cortex, and cognitive BCI a system that monitors cognitive functions (Gaume et al. 2015). The long-term goal of our research is the development of a cognitive BCI able to monitor the variations of sustained attention in real time. This “attentionometer” could for example warn its user immediately and objectively that its attention shifted towards a distractor. It could also be used to determine its user’s ability to pay attention at certain moments of the day, for example to know he is capable of driving. Such a device could also be used to train sustained attention by providing a continuous feedback to the user. More generally, being able to monitor our attention as directly as, for instance, the position of our arm, would probably allow us to learn how to consciously regulate our attention, and to find ways to concentrate easily and comfortably during long periods of time. This is the principle of neurofeedback (Lachaux 2011; Arns et al. 2017; Gaume et al. 2016).

The challenge that is adressed in the present work is to find a neural correlate of the variations of sustained attention that can be monitored in real time and that, ideally, does not require invasive hardware nor the performance of a specific task. Classical approaches to monitor sustained visual attention are based on the Mackworth Clock (Mackworth 1948), and involve long sessions of visual monitoring in search for unusual events. The objective measure used in that case as an indicator of attention is the response time of the subjects to the appearance of targets. However, changes in sustained attention can be measured either as fluctuations, or deteriorations of performances along time (Clayton et al. 2015). Most subjects indicate that their attention fluctuates between targets and that each unusual event recaptures their focus. The discrete response time measurements used in such experiments are therefore more related to deteriorations of attention, while the experiment fails to capture the fluctuations occuring in between the targets. In addition, all subjects who show a significant increase in their response time over the course of the experiment report drowsiness and dozing. Consequently, it is difficult to disambiguate such an experiment from the monitoring of alertness and vigilance (i.e. sustained arousal). Therefore, we deem this experimental approach not optimal for the monitoring of sustained attention. Indeed, it only requires a discrete involvement of attention and only allows a discrete measurement of performance. Consequently, the design of our cognitive BCI involves a continuous task: the subject has to focus continuously, the evaluation of attentional load relies on continuous stimulations, and the performance of the subject is measured continuously.

We therefore introduce an experimental paradigm involving a continuous task for the monitoring of variations of visual sustained attention. Furthermore, we intend to design a task that minimizes the involvement of cognitive functions other than sustained attention. We therefore decided to focus on the attention required to continuously update the visual information we get from our sensory inputs instead of a task based on continuous processing of the information stored in working memory, which would involve working memory load in addition to sustained attention.

Subjects of the experiment reported that the “medium” rounds and the “hard” rounds required more or less the same concentration. We therefore expected that classification would work better between “easy” and “medium” or between “easy” and “hard” rounds, than between “medium” and “hard” rounds. This is consistent with the results shown on the tables. It is also consistent with the fact that the features selected to discriminate “medium” and “hard” rounds are different from those selected for the other classifiers.

In order to estimate BCI performances, LOSO classification accuracies were used. Classical BCI models are designed and tested within subjects. Instead, LOSO scores are estimates of the generalization performance of the classifier onto new subjects, and it is therefore an estimate of transfer learning (Cheng et al. 2017). The choice of the LOSO validation approach was made to avoid biases in the estimates. Indeed, we might achieve higher accuracies on this database using within-subject analysis instead of cross-subject classifications. However, a within-subject model would increase the risk of modeling subject-related motor and/or cognitive confounders, which would be a serious flaw for a cognitive BCI (Gerjets et al. 2014). Furthermore, a LOSO approach allows us to design a cognitive BCI system which does not need calibration. In other words, it can be applied right away on new subjects without retraining.

Pairwise classification (“easy” vs. “medium” or “hard” rounds) reached 75% even with short 5 s epochs. Accuracies above 85% were obtained using 30 s epochs. Note that these accuracies were obtained without any calibration of the model to the left-out subject. Due to the bias-variance dilemma (Geman et al. 1992), if sources of variability exist in the data, it creates a lower bound (the so-called irreducible error) for classification accuracy. Considering that BCI illiteracy affects 15–30% of the population (Jeunet et al. 2016), the irreducible error is 15%, and therefore accuracies higher than 85% would be suspicious and might indicate a flawed model. These accuracies are therefore close to the optimum, which illustrates the robustness of the BCI model and show that prediction of the current cognitive load required by a task can be monitored using EEG. Such performances are largely sufficient for several biomedical applications. Indeed, neurofeedback and monitoring applications do not need a high accuracy, but a high specificity (above 85%), so that it would trigger only useful alarms. We can achieve such a detection by increasing specificity at the cost of sensitivity, consequently our system can already be used for biomedical applications in its present state.

In the “easy” versus “medium” discrimination, the LOSO classification accuracy estimations reached a plateau when the number of features exceeded ten, thereby proving that the LDA classifiers designed with these features are not very prone to overfit our data. However, in other discriminations, and especially in the “easy” versus “hard” discrimination, accuracy increased with the number of features and did not seem to reach a plateau (see Table 5).

Frontal theta power was a good predictor of the task load, providing a classification accuracy of 66% to separate the “easy” task from higher demand “medium” and “hard” tasks. This result is in accordance with previously reported observations: an increase of frontomedial theta is the best known correlate of sustained attention (Klimesch 1999; Clayton et al. 2015). Joint EEG and fMRI analysis have shown that prefrontal theta oscillations play a key role in salience and executive functions (Mas-Herrero and Marco-Pallars 2016) serving as a common substrate for the engagement of pre-supplementary motor area and the dorsomedial prefrontal cortex. Knudsen’s model does consider salience filters and executive functions as core components of attention (Knudsen 2007). Augmentation of frontal theta has also been widely reported in association with working memory load (Gevins et al. 1998; Jensen and Tesche 2002; Gevins and Smith 2003; Sauseng et al. 2005; Cavanagh and Frank 2014; Hsieh and Ranganath 2014; Fairclough and Ewing 2017).

High beta activity in the left central region provided good classification results. This activity may be linked with motor control, since the task involved the control of a joystick with the right hand. Nevertheless, the high beta frequency band, containing spectral activity between 18 and 25 Hz, does not overlap with the sensorimotor rhythm, and may therefore be linked with cognitive load and attention more than with direct control of motor activity. This is consistent with the work of Wróbel (2000), which supports that the 15–25 Hz frequency band is a general carrier for attention in the brain. Hypotheses about the sources of this frontal activity could be made, however precise reconstruction would require a high number of electrodes (64\(+\)), which is why this method has not been used with our equipment.

Attention is based on brain networks involving coupling of several frequencies (Clayton et al. 2015). In order to target these complex interactions, the feature selection approach which was used in our investigations extracts biomarkers carrying complementary information. Instead of relying on one single marker, the resulting cognitive BCI model combines multiple features (see Table 4). Overall, the attentional load is predicted by a combination of prefrontal theta power, broad spatial range gamma power, fronto-central alpha power and fronto-central beta power. This combination of channel and frequency ranges is coherent with the existing knowledge about the underlying networks of attention: frontomedial theta rythms for attentional monitoring processes, gamma-range oscillations for excitation processes, and alpha power for attentional inhibition processes. As we already explained above, the beta range activity could be correlated with either cognitive load or sensorimotor confounders of the task.

Another potential confounder is the variation of arousal during the task. The session was organized with successive trials of increasing difficulty, which may induce a correlation of mental fatigue with the task condition (the very first trial was always easy, the very last always hard). A randomized order would be a better control for this mental fatigue confounder. However, a randomized order would also introduce an anxiety confounder : not knowing the level of difficulty in advance was more stressful when the difficult condition occurred, whereas anxiety levels were more comparable between conditions when the subjects could anticipate it. Both mental fatigue and anxiety can have an impact on arousal, we chose to prevent anxiety.

To conclude, we presented in this paper an experiment aimed at discriminating low versus high sustained attention states in a continuous task. This continuous task approach introduces a new type of cognitive BCI design, where we attempt to control the cognitive load using an ecological video-game where a cognitive constraint is continuously maintained to prevent load fluctuations. We showed that features such as average frequency band power, which can be estimated continuously and do not require discrete stimulations, provide good classification rates even with short epochs (5 s). A leave-one-subject-out cross-validation approach was used to avoid overfitting. Nevertheless the present investigation is yet insufficient to prevent modeling confounders of attentional load. For instance, event though we removed eye blinks using SOBI, we cannot rule out that some features participating in the classification process may be related to eye movement or electromyographic artefacts, which might act as a potential confounder of sustained attention. Consequently, other experiments will be required to differentiate potential confounders (Gerjets et al. 2014), i.e. the features specific to the motor control task of this investigation, emotional, arousal or stress correlates [see for instance Mühl et al. (2014)], artifacts, and the other cognitive loads that are not specific to sustained visual attention.