EEG–EMG coupling as a hybrid method for steering detection in car driving settings

Recent advances in sensing and control techniques have made it possible to design cars with a large set of features that can take control of specific aspects of the driving task (e.g., cruise control, autonomous driving) or provide information to the driver to support specific maneuvers (e.g., lane departure). Because driving is a complex behavior involving interrelated motor and cognitive elements such as attention, visuospatial interpretation, visuomotor integration, and decision making (Calhoun et al. 2002; Calhoun and Pearlson 2012), to achieve ever-higher levels of driver support, it is important to investigate and characterize the related brain processes underlying the driver’s actions.

In the years, several attempts were made to characterize the neural correlates of driving in scenarios with different levels of ecology and neural variables, spanning from hemodynamic (Walter et al. 2001; Spiers and Maguire 2007; Mader et al. 2009; Calhoun and Pearlson 2012; Schweizer et al. 2013), to magneto- (Fort et al. 2010; Sakihara et al. 2014) and electrophysiological activity (Schier 2000; Haufe et al. 2011, 2014; Gheorghe et al. 2013; Khaliliardali et al. 2015; Kim et al. 2015; Zhang et al. 2015; Brooks and Kerick 2015; Brooks et al. 2016; Garcia et al. 2017; Vecchiato et al. 2018, 2020). In particular, these latter studies show the possibility of using electroencephalography (EEG) to decode the driver’s cognitive processes in simulated and real car scenarios with the ultimate goal of predicting the upcoming action. Although the success in the classification of salient driving events such as braking (Haufe et al. 2011, 2014; Kim et al. 2014, 2015; Hernández et al. 2018; Wang et al. 2018; Teng et al. 2018; Lin et al. 2018; Nguyen and Chung 2019) and steering (Gheorghe et al. 2013) actions, the level of accuracy is still moderate. Most importantly, the detected neurophysiological features are elicited just around a few milliseconds before the upcoming driving event, making it difficult to implement electronic assisting devices. In fact, despite the noteworthy advancement of recent years, there are still several technological and psychophysiological issues that limit the utilization of the EEG for the real-time identification and monitoring of brain-related activity in driving scenarios. For instance, most EEG sensors remain uncomfortable to place and keep on the head of people who are not used to such recordings, leading to an increase of artifacts and the corruption of the recorded brain signals. Factors such as attention, memory load, and competing cognitive processes (Gonçalves et al. 2006; Käthner et al. 2014; Calhoun and Adali 2016), as well as user’s individual characteristics such as lifestyle, gender, and age (Kasahara et al. 2015) influence brain dynamics producing significant intra- and inter-subject variability (Saha and Baumert 2020; Saha et al. 2021). The low signal-to-noise ratio returned by raw EEG data requires a range of conceptually very different and computationally expensive algorithms to extract significant temporal and frequency EEG features (Müller et al. 2004; Lotte et al. 2007, 2018; Krusienski et al. 2011; Bellotti et al. 2019). Hence, the computing hardware and software must warrant a sufficiently high performance and low latency to preserve the earliness of prediction. For these reasons, it is not always useful to rely on EEG-based predictions alone (Wöhrle et al. 2017). These aspects motivate the need to identify other robust physiological features tackling increased noise due to environmental characteristics and the interaction among neural processes (Lohani et al. 2019). The final aim is to foster the utilization of neurophysiological measurements in experimental settings closer to everyday life activities.

In this sense, surface electromyography (EMG) provides a non-intrusive way of measuring muscle activation and is an appropriate technique when assessing active steering systems (De Luca 1997; Ahlström et al. 2019). It opened new perspectives in ergonomics and provided new tools for analyzing the neuromuscular system in working environments. It is experiencing a growing interest in medical and research applications thanks to the recent availability of novel low-end commercial products increasing the wearability relative to EEG sensors allowing to perform longer recording sessions more comfortably (Gazzoni et al. 2016; Milosevic et al. 2017). Previously, EMG recordings have been used to assess the function of the upper limb muscles during car driving (Jonsson and Jonsson 1975; Liu et al. 2012; Gao et al. 2014). The main findings are that the prime movers are primarily a consequence of steering direction while the stabilizing or fixating muscles are primarily constant, returning that the key muscles correlated to steering are the triceps brachii, the deltoids, pectoralis major, and infraspinatus (Pick and Cole 2006; Liu et al. 2012; Gao et al. 2014). In particular, it is well known that the whole deltoid muscle acts in the abduction of the arm, and there is a synergy between the anterior portion of the muscle and the contralateral posterior portion when moving the steering wheel. More specifically, the anterior portion serves to rotate the steering wheel contralaterally and the posterior portion to rotate it ipsilaterally (Jonsson and Jonsson 1975). EMG can be a reliable method for action prediction because it has limited sensitivity to environmental disturbance (Bi et al. 2019). Compared to EEG, surface EMG can be easily acquired and processed and provide useful information on the movement that the person is executing. Despite the advantages mentioned above, EEG and EMG exhibit different temporal properties concerning a movement onset: changes in the EMG can only be observed close to the actual movement onset, while it is possible to detect EEG motor biomarkers earlier (Bai et al. 2011; Wöhrle et al. 2017; Trigili et al. 2019).

It has been demonstrated that voluntary movements generate changes in the alpha and beta ranges of the EEG spectrum (Pfurtscheller and Lopes da Silva 1999) and that such rhythms are in turn related to modulations of the EMG activity (Hari and Salenius 1999). Recent works have focused on the synchronization between rhythmical activity in the motor cortex and muscular activity employing cortico-muscular coherence, which is usually observed during periods of muscular contraction and has been reported in several studies involving both EEG and MEG (Conway et al. 1995; Boonstra et al. 2009; Cheyne 2013; Rizzo et al. 2020). Current approaches to cortico-muscular coordination focus on associations and synchronous activation between individual brain rhythms at specific cortical areas (e.g., motor cortex, hippocampus), and peripheral muscle activity during specific movement tasks or exercises in ecological conditions (e.g., walking and running) (van Wijk et al. 2012; Cui et al. 2017; Rendeiro and Rhodes 2018; Li et al. 2019a, 2020; Fauvet et al. 2019). In such scenarios, the neuromuscular signals are noisier than those collected in standard laboratory conditions due to uncontrolled environmental settings, and there is also the issue of addressing the simultaneous presence of concurrent cerebral processes. Finally, on the user side, there is the need to create comfortable experimental settings to not pollute the neuromuscular signals with undesired components due to the experienced fatigue of wearing biomedical sensors on the head and several parts of the body. To overcome issues with both EEG- and EMG-based control methods, a combination of both systems, building on each signal’s advantages and diminishing the limitations of each, might be a promising strategy (Lalitharatne et al. 2013). It would then be optimal to identify the cerebral features of interest in standard laboratory settings and then look for neuromuscular invariants in more ecological settings with higher recording complexity. In this way, offline analysis performed on data collected in standard settings would inform the online pipeline to implement to extract neuromuscular features of interest.

Following this reasoning, we propose a hybrid method to distinguish left from right steering in a driving simulator based on EEG and EMG signals collected during steering actions. While using EEG and EMG signals in steering decoding would certainly add predictive power relative to unimodal recordings, such a strategy is hurdled by the complexity of combining the two acquisitions during real-life driving. However, a potential solution could be to first identify in a preliminary recording the steering-related EEG features and then use such information during real driving to increase the predictive power of the steering action based solely on the EMG acquisition.

We used two experimental scenarios to demonstrate this possibility: first, participants were required to perform stand-alone steering wheel movements; successively, we used a driving simulator that represents a step forward towards recording the continuous EEG signals in the real car in natural traffic conditions. This difference between the experimental conditions led us to adopt the terms “non-ecological” and “ecological” steering task. We hypothesized that the electrophysiological correlates of the non-ecological steering could assist the steering action detection in the ecological condition.

Starting from these premises, we first investigated brain and muscular activity underlying steering behavior during the non-ecological steering task. This procedure allowed us to identify the EEG correlates of steering without confounding effects. Then, we correlated such features with the EMG activity collected during a session of driving simulation to extend the validity of the non-ecological cerebral signatures to a more ecological steering task. This double task approach enabled to (i) take advantage of multimodal recordings exploiting the information carried by both neural and muscular data, and (ii) solve ergonomic issues related to the simultaneous acquisition of different signals.

Therefore, we aimed to characterize the EEG–EMG coupling associated with the natural and self-initiated execution of steering actions while driving. To this end, we took advantage of the independent component analysis (ICA), which separates mixed EEG signals into maximally independent activities, each characterized by a precise scalp topography and a corresponding generator pattern, typically modeled as patches of cortical pyramidal cells (Delorme et al. 2012). This strategy also allowed us to avoid field spread caused by the large distance between sensors and neural sources and by the spatial blurring effect of the skull on the scalp’s potential distribution of EEG signals (Schoffelen and Gross 2009). Thus, we expect to identify reliable, independent components whose topography indicates the involvement of motor circuits, whose reactivity precedes the steering action, and is correlated with the muscular activity of the deltoids. Should the EEG–EMG coupling be modulated across the two types of action participants perform (i.e., left and right steering), this would prove that such a hybrid method can be used to discriminate and predict motor intentions associated with natural driving behavior.

In the present study, we report that the modulation of the EEG mu rhythm observed during the motor preparation of non-ecological steering predicts the muscular activity of deltoids, thus anticipating subject steering behavior. The reactivity of such rhythm measured across sensorimotor areas during the non-ecological steering preparation anticipates the corresponding action. This result paves the way for using such a cerebral feature to discriminate steering actions in the ecological task. We report the increase of EMG activity of the deltoid anticipating the contralateral steering in non-ecological and ecological steering tasks. These results show an asymmetric muscular activity of the deltoids beginning before the action onset remaining steady during the steering execution, i.e., the coordinated increase of power of the right deltoid and the corresponding decrease of power of the left deltoid is associated with the steering action on the left side. Such findings show that monitoring the two deltoids’ muscular activity makes it possible to discriminate the steering side before the action onset while driving in non-ecological and ecological scenarios. Strikingly, the identified non-ecological EEG feature correlates with the ecological EMG activity of the deltoids, providing an improvement of the discrimination power of the steering side during driving simulation. The comparison between the results related to the EMG analysis and those concerning the EEG–EMG cross-correlation returned larger relative timing in favor of the latter related to the succession of steering events. In fact, on one side, there is the issue of distinguishing the anticipatory components due to EEG predicting power from those which are merely due to computational accounts, e.g., windowing implied in the cross-correlation calculation. However, we observe a clear anticipatory pattern returned by cross-correlation values related to the succession between left and right steering events. This change of cross-correlation values is detectable in advance relative to the single EMG signal.

The timing of the EEG response identified in this study is compatible with patterns of event-related de-synchronization (ERD) which are reported to be predictive of the upcoming action between 1.5 and 1 s before the movement onset (Neuper et al. 2006). ERD reflects non-phase-locked EEG changes of oscillatory activity within the alpha and beta frequency due to sensorimotor events (Pfurtscheller and Lopes da Silva 1999; Savić et al. 2020). The EEG component identified during the non-ecological task arises from the motor areas and may reflect the preparation of steering actions. The scalp topography that we reported is reminiscent of the mu rhythm observed in motor and premotor regions (Gastaut et al. 1952; Pfurtscheller et al. 1997). Such electrical activity produces somatotopically organized de-synchronization during execution, observation, and imagination of actions (Pineda 2005; Pfurtscheller et al. 2006; Arnstein et al. 2011; Avanzini et al. 2012). These regions perform several functions other than body movements control, such as sensory-motor transformation, action understanding, decision-making regarding execution and initiation of action, preparation, and planning of complex movements (Roland 1984; Rizzolatti and Luppino 2001). Recent literature reports ERD in contralateral sensorimotor cortices during movement preparation of visually cued movements (Li et al. 2018; Little et al. 2019). Therefore, we argue that the identified EEG mu rhythm modulations regulate the motor preparation of the right and left deltoids for steering actions. In particular, we report that the mu de-synchronization over left sensorimotor regions is related to the right deltoid’s activity during left steering. This would show that left steering involves larger neural computation than right steering, despite the absence of significant difference in the steering wheel angle, as already reported in a previous study (Oka et al. 2015). Another study also reports the de-synchronization of the alpha rhythm across sensorimotor regions related to relative steering angle compensation (Brooks and Kerick 2015), thus showing the relation between such electroencephalographic feature and steering response as already observed in more simple tasks (Pfurtscheller and Neuper 1994; Stancák and Pfurtscheller 1996).

Analyzing the EMG in the time–frequency domain, we observe that the maximum muscle activity changes significantly due to different steering wheel angles and turning directions in non-ecological and ecological scenarios. It is known that the anterior and middle portions of the deltoid muscle work intermittently during car driving and that their functioning regulates the contralateral rotation of the steering wheel, being activated for a duration of around 50% since the initial stage of the action (Jonsson and Jonsson 1975; Pick and Cole 2006; Liu et al. 2012; Gao et al. 2014). Specifically, muscle activity is relatively small when the steering wheel is near its center, but it increases rapidly as the wheel starts to turn (Gao et al. 2014). Although EMG was successfully used to identify the muscles involved in generating and predicting torque at the steering wheel (Pick and Cole 2006), EMG alone has low power in predicting the steering as we report to be limited to a few hundred milliseconds. Thus, we exploited the EEG scalp feature to improve steering detection by computing the cross-correlation between the mu rhythm de-synchronization retrieved during the non-ecological task and the EMG activity elicited in the ecological scenario. This procedure allowed us to enlarge the time window in which it is possible to discriminate the steering action. We showed that this EEG–EMG coupling is specific for steering actions and their directionality, returning better results relative to single EMG signals. Similar findings were recently reported concerning the use of a hybrid human–machine interface for gait decoding (Tortora et al. 2020), motor rehabilitation of stroke patients (Sarasola-Sanz et al. 2017), movement detection of hand-paralyzed patients (Lóopez-Larraz et al. 2018). All these studies suggested an increase in the classification accuracy and the number of commands for human–machine-interfaces (Hong and Khan 2017).

Here we performed two consecutive recording sessions to demonstrate the feasibility of exploiting EEG correlations of steering behaviour collected in a standard highly controlled environment to detect steering action in a more ecological setting such as a driving simulator. In the first non-ecological recording session, we extracted ERD as a neural correlate of the EMG activity of the deltoids predicting the upcoming steering behavior. In the following ecological recording session performed at the driving simulator, we collected the EMG activity of the deltoids. We showed that the ERD resulting from the previous session is indeed informative concerning the steering action in this more naturalistic scenario. As expected, a large variability characterized the EEG collected during the ecological steering task, and the statistical analysis did not return any significant result related to the cerebral steering component. Indeed, steering actions were reliably discriminated through the analysis of the cortical correlates associated with the non-ecological task. This evidence suggests the usefulness of exploiting cortical correlates of motor preparation of non-ecological actions to predict the same action in ecological conditions. The non-significant results of the EEG collected during the ecological steering task indeed highlight the usefulness of exploiting cortical correlates associated with the non-ecological task for steering detection in a more natural condition. From a methodological perspective, we used the ERSP to investigate EMG correlates of steering for directly assessing the coupling of cerebral and muscular activations. EMG features in the time domain, such as Root Mean Square and Mean Absolute Value, could also be useful for future analyses implementing online classifiers.

Several recent studies on driving addressed the issue of steering classification using different EEG features. In particular, it was assessed the possibility to decode self-generated actions detecting whether the driver would perform a lane change shortly in a simulated highway (Gheorghe et al. 2013). Authors report slow negative EEG deflections across central areas consistent with the movement-related potentials 500 ms before the lane change, yielding a classification accuracy of 79% with an average detection time of 613 ms before the actual steering action. In another series of studies performed with both driving simulators and real cars (Zhang et al. 2015), error-related brain potentials were analyzed to investigate the possibility of using an external device to be adapted to the driver’s goal, i.e., assisting in the upcoming steering action. This strategy was enacted by showing the drivers a visual stimulus indicating their inference about the next turning direction when approaching an intersection. Authors report differences in the EEG response over fronto-central areas when the directional stimulus does not match the driver’s intention. Statistical differences between error and correct conditions were observed between 200 and 600 ms after feedback, yielding a mean accuracy of the event-related decoding of 0.68, which indicates the possibility of extracting meaningful information about the driver’s need for assistance.

Several studies demonstrated that the combination of EEG and EMG can improve the reliability of movement prediction based on a single modality (Vecchiato 2021; Di Liberto et al. 2021). For example, the prediction of movement onset based on EEG analysis can be improved by designing hybrid systems monitoring at the same time additional peripheral signals depending on the context requirements. EEG and EMG signals can reliably predict movements before the action onset, showing that both can potentially control an electronic device (Kirchner et al. 2014). Different measures could be combined for different movement stages, e.g., movement planning, the start of the movement, and movement execution (Novak et al. 2013). Other investigations were conducted to predict voluntary movements before their occurrence (Bai et al. 2011) and vehicle steering (Gomez-Gil et al. 2011) using hybrid EEG–EMG BCIs. The hybrid strategy was initially introduced in the Brain Computer Interfaces (BCIs) to exploit the advantages of different physiological signals and computational approaches to achieve specific goals better than a conventional EEG-based system (Pfurtscheller et al. 2010; Li et al. 2019b). A hybrid system might predict user intention with higher accuracy, thus improving the whole system’s performance or reducing the rate of false positives (Usakli et al. 2009, 2010; Ma et al. 2015; Hong and Khan 2017). Today such a methodology is used to boost motor rehabilitation of stroke patients (Sarasola-Sanz et al. 2017), for the online movement prediction (Kirchner et al. 2014; Wöhrle et al. 2017), gait decoding (Tortora et al. 2020), and its application is investigated in other several domains of bio-robotics (Lalitharatne et al. 2013).

The existence of preparatory electrophysiological activity elicited before the onset of steering action allows us to infer the upcoming driving actions in advance. The reported EEG–EMG coupling is a proof of concept for utilizing hybrid systems for the detection and online prediction of driving actions, exemplifying how it might be possible to complement information from behavioral, physiological, and external sources to control the level of assistance needed by the driver in that context (Chavarriaga et al. 2018). The predictive power of the EEG–EMG coupling demonstrated in a car simulator could be further investigated in larger sets of actions to extend the validity of this neurophysiological mechanism beyond driving.