Automated sleep scoring: a comparative reliability study of two algorithms

doi:10.1016/0013-4694(87)90214-8

Electroencephalography and Clinical Neurophysiology

Volume 66, Issue 4, April 1987, Pages 448-456

https://doi.org/10.1016/0013-4694(87)90214-8 Get rights and content

Abstract

In the present study, deterministic and stochastic sleep staging (DSS and SSS) methods were compared with expert visual analysis in order to provide reliability estimates under strict conditions of comparison. Thirty polygraphic records (15 controls, 15 patients) have been investigated, including artefacts and doubtful periods. Average agreement rates of both methods compared to expert visual scoring were very similar, although a few specifics occasionally appeared for partial sleep stages. The comparison of more than 40,000 sleep decisions (on 20 sec epochs) yielded 75% absolute reliability for normal controls and 70% for pathological cases. However, if the agreement rate obtained for routine visual scoring (82%) in our sleep laboratory is considered as satisfactory, our system is then 90% satisfactory. Finally, complementary aspects outlined in the two automatic scoring systems suggested the development of a unique algorithm on the basis of these methods. Keeping in mind the size of the test sample and the strict procedure of comparison, the two automated staging systems described in this study can be used with reasonable confidence for large scale investigations of sleep in man.

Résumé

Dans cet article, nous avons comparé une méthode déterministre et une méthode stochastique d'analyse du sommeil à l'interprétation visuelle d'un expert de référence. Le but de l'expérience consistait à estimer les taux d'accord des deux méthodes en respectant des conditions de comparaison définies de manière stricte. L'effectif de base comprenait 30 enregistrements polygraphiques (15 contrôles et 15 patients) traités globalement, y compris les périodes de moindre qualité d'enregistrement et même les artéfacts. Les taux moyens d'accord obtenus par rapport à un expert de référence étaient semblables pour les deux méthodes, bien que des spécificités apparussent au niveau de la reconnaissance de certains stades. La comparaison de plus de 40.000 décisions de stades sur des périodes de 20 sec a conduit à un taux de fiabilité globale absolue de 75% chez les sujets normaux et 70% pour les patients, en considérant une analyse visuelle soignée. Néanmoins, en se référant au taux d'accord de l'analyse visuelle effectuée en routine dans notre laboratoire (82%), notre système était considéré comme satisfaisant à 90%. Enfin, la mise en évidence d'aspects complémentaires des deux méthodes d'analyse automatique nous a permis d'envisager de développer un nouvel algorithme plus fiable sur base de ces deux méthodes. Etant donné la taille de l'échantillon sur lequel nous avons testé les deux systèmes, et la stricte méthodologie de comparaison, nous pouvons raisonnablement estimer que ces méthodes d'analyse automatique peuvent être exploitées dans le cadre d'investigations à grande échelle sur le sommeil chez l'homme.

References (21)

J.S. Barlow
EEG transient detection by matched inverse digital filtering
Electroenceph. clin. Neurophysiol.
(1980)
R. Broughton et al.
A phase locked loop device for automatic detection of sleep spindles and stage 2
Electroenceph. clin. Neurophysiol.
(1978)
J.D. Frost
An automatic sleep analyser
Electroenceph. clin. Neurophysiol.
(1970)
J.M. Gaillard et al.
Analyse automatique des enregistrements polygraphiques de sommeil
Electroenceph. clin. Neurophysiol.
(1971)
J.M. Gaillard et al.
Analyse automatique du sommeil par un système hybride: nouveaux résultats
Electroenceph. clin. Neurophysiol.
(1972)
I. Gath et al.
Computerized method for scoring of polygraphic sleep recordings
Comput. Prog. Biomed.
(1980)
L. Johnson et al.
Spectral analysis of the EEG of alpha subjects during waking and sleeping
Electroenceph. clin. Neurophysiol.
(1969)
B.P. Lacroix et al.
On-line automatic sleep scoring system involving Bayesian filtering
Measurement
(1984)
J.R. Smith et al.
EEG sleep stage scoring by an automatic hybrid system
Electroenceph. clin. Neurophysiol.
(1971)
P. Bech et al.
The role of training courses in multicenter trials: WHO experiences
Biol. Psychiat.
(1985)

There are more references available in the full text version of this article.

Cited by (55)

A study on sleep stage classification based on a single-channel EEG signal
2021, Electronic Devices, Circuits, and Systems for Biomedical Applications: Challenges and Intelligent Approach
Automated sleep stage scoring can deliver a clinical report for analyzing patients with sleep abnormalities. Traditional polysomnography or visual scoring is unfit for observing the large populations since visual scoring is extremely conservative and depends on skillful knowledge based on Rechtschaffen and Kales score and American Academy of Sleep Medicine rules. This chapter presents a study on single-channel EEG (electroencephalogram) signals for finding the fittest approach in terms of preprocessing, extraction of features, and classification for sleep abnormalities analysis. It focuses on the various applications regarding the importance of the prediction of sleep stage scoring. Analysis of the sleep stage based on the qualitative method along with traditional quantitative methods is presented. The datasets for sleep abnormality analysis based on EEG signals are introduced in this chapter. Also, the employed expert rules and relevant character of the EEG signal dataset play significant roles in the study as well. In this work, various state-of-the-art methods are comprehensively reviewed based on all of the significant features discussed here. Each step in the sleep stage scoring is briefly explained, and the relevant investigations are presented.
A novel, fast and efficient single-sensor automatic sleep-stage classification based on complementary cross-frequency coupling estimates
2018, Clinical Neurophysiology
Citation Excerpt :
Sleep stage scoring is the gold standard for the analysis of human sleep (Agarwal and Gotman, 2001). While many sleep laboratories use the traditional manual sleep stage scoring of neurophysiological recordings to produce the hypnogram, recent years have witnessed a burst of proposed methods for automatic or semi-automatic sleep staging (e.g., Agarwal and Gotman, 2001, 2002; Becq et al., 2005; Berthomier et al., 2007; Ma et al., 2011; Itil et al., 1969; Koley and Dey, 2012; Krakovská and Mezeiová, 2011; Larsen and Walter, 1970; Schaltenbrand et al., 1996; Sheng-Fu et al., 2012; Pan et al., 2012; Stanus et al., 1987; Gudmundsson et al., 2005; Šušmáková and Krakovská, 2008; Huang et al., 2002). The different methods for ASSC generally extract features from the signals to analyse each temporal segment (epoch) and use classification algorithms to detect/predict the sleep stages (Güneş et al., 2010; Zoubek et al., 2007; Sousa et al., 2015).
Limitations of the manual scoring of polysomnograms, which include data from electroencephalogram (EEG), electro-oculogram (EOG), electrocardiogram (ECG) and electromyogram (EMG) channels have long been recognized. Manual staging is resource intensive and time consuming, and thus considerable effort must be spent to ensure inter-rater reliability. As a result, there is a great interest in techniques based on signal processing and machine learning for a completely Automatic Sleep Stage Classification (ASSC).
In this paper, we present a single-EEG-sensor ASSC technique based on the dynamic reconfiguration of different aspects of cross-frequency coupling (CFC) estimated between predefined frequency pairs over 5 s epoch lengths. The proposed analytic scheme is demonstrated using the PhysioNet Sleep European Data Format (EDF) Database with repeat recordings from 20 healthy young adults. We validate our methodology in a second sleep dataset.
We achieved very high classification sensitivity, specificity and accuracy of 96.2 ± 2.2%, 94.2 ± 2.3%, and 94.4 ± 2.2% across 20 folds, respectively, and also a high mean F1 score (92%, range 90–94%) when a multi-class Naive Bayes classifier was applied. High classification performance has been achieved also in the second sleep dataset.
Our method outperformed the accuracy of previous studies not only on different datasets but also on the same database.
Single-sensor ASSC makes the entire methodology appropriate for longitudinal monitoring using wearable EEG in real-world and laboratory-oriented environments.
Automatic detection of periods of slow wave sleep based on intracranial depth electrode recordings
2017, Journal of Neuroscience Methods
Citation Excerpt :
Notably, SWS is not unique to humans: it has been reported in mammals (Vorster and Born, 2015), birds (Libourel and Herrel, 2015), and reptiles (Shein-Idelson et al., 2016). Automated methods to exclusively identify SWS have been few (Benington et al., 1994; Shein-Idelson et al., 2016; Stanus et al., 1987), and the inter-observer agreement can be as low as 67.2% (Nizam et al., 2013). Here, we present a new method for the automated detection of points of time during which a person is in SWS from intracranial depth electrode recordings alone.
An automated process for sleep staging based on intracranial EEG data alone is needed to facilitate research into the neural processes occurring during slow wave sleep (SWS). Current manual methods for sleep scoring require a full polysomnography (PSG) set-up, including electrooculography (EOG), electromyography (EMG), and scalp electroencephalography (EEG). This set-up can be technically difficult to place in the presence of intracranial EEG electrodes. There is thus a need for a method for sleep staging based on intracranial recordings alone.
Here we show a reliable automated method for the detection of periods of SWS solely based on intracranial EEG recordings. The method utilizes the ratio of spectral power in delta, theta, and spindle frequencies relative to alpha and beta frequencies to classify 30-s segments as SWS or not.
We evaluated this new method by comparing its performance against visually scored patients (n = 9), in which we also recorded EOG and EMG simultaneously. Our method had a mean positive predictive value of 64% across all nights. Also, an ROC analysis of the performance of our algorithm compared to manually labeled nights revealed a mean average area under the curve of 0.91 across all nights.
Our method had an average kappa score of 0.72 when compared to visual sleep scoring by an independent blinded sleep scorer.
This shows that this simple method is capable of differentiating between SWS and non-SWS epochs reliably based solely on intracranial EEG recordings.
A hierarchical classification method for automatic sleep scoring using multiscale entropy features and proportion information of sleep architecture
2017, Biocybernetics and Biomedical Engineering
Citation Excerpt :
NREM is further divided into four stages: S1, S2, S3 and S4. Sometimes S3 and S4 are merged into a single stage, known as slow wave sleep (SWS) [3,4]. In sleep medicine, sleep staging is mainly conducted by highly trained experts.
Sleep scoring is a critical step in medical researches and clinical applications. Traditional visual scoring method is based on the processing of physiological signals, such as electroencephalogram (EEG), electrooculogram (EOG) and electromyogram (EMG), which is a time consuming and subjective procedure. It is an urgent task to develop an effective method for automatic sleep scoring.
This paper presents a hierarchical classification method for automatic sleep scoring by combining multiscale entropy features with the proportion information of the sleep architecture. Based on a three-layer classification scheme, sleep is categorized into five stages (Awake, S1, S2, SWS and REM). Specifically, the first layer is a standard SVM which performs classification between Awake and Sleep, while the second and third layers are implemented by combining probabilistic output SVM with proportion-based clustering. Multiscale entropy (MSE) from electroencephalogram (EEG) is extracted to represent signal characteristics in multiple temporal scales.
The proposed method is evaluated with 20 sleep recordings, including 10 subjects with mild difficulty falling asleep and 10 healthy subjects. The overall accuracy of the proposed method is 91.4%. Compared with traditional methods, the classification accuracy of the proposed method is more balanced and the global performance is much better. The dataset includes both healthy subjects and subjects with sleep disorders, which means the presented method has good generalization capacity. Experimental results demonstrate the feasibility of the attempt to introduce proportion information into automatic sleep scoring.
Learning machines and sleeping brains: Automatic sleep stage classification using decision-tree multi-class support vector machines
2015, Journal of Neuroscience Methods
Citation Excerpt :
Both manuals propose to use EEG derivations (2 in the R&K manual, and 3 in the AASM one), 2 EOG electrodes and one EMG electrode. While visual scoring remains the gold-standard, recent years have witnessed a surge in method developments for automatic or semi-automatic sleep staging (e.g. Agarwal and Gotman, 2001, 2002; Becq et al., 2005; Berthomier et al., 2007; Ma et al., 2011; Itil et al., 1969; Koley and Dey, 2012; Krakovska and Kristina, 2011; Larsen and Walter, 1970; Schaltenbrand et al., 1996; Sheng-Fu et al., 2012; Shing-Tai et al., 2012; Stanus et al., 1987; Steinn et al., 2005; Huang et al., 2014). Although these results obtained so far are promising, there is still room and a need for improvement, especially given the time-consuming and tedious nature of visual sleep scoring.
Sleep staging is a critical step in a range of electrophysiological signal processing pipelines used in clinical routine as well as in sleep research. Although the results currently achievable with automatic sleep staging methods are promising, there is need for improvement, especially given the time-consuming and tedious nature of visual sleep scoring.
Here we propose a sleep staging framework that consists of a multi-class support vector machine (SVM) classification based on a decision tree approach. The performance of the method was evaluated using polysomnographic data from 15 subjects (electroencephalogram (EEG), electrooculogram (EOG) and electromyogram (EMG) recordings). The decision tree, or dendrogram, was obtained using a hierarchical clustering technique and a wide range of time and frequency-domain features were extracted. Feature selection was carried out using forward sequential selection and classification was evaluated using k-fold cross-validation.
The dendrogram-based SVM (DSVM) achieved mean specificity, sensitivity and overall accuracy of 0.92, 0.74 and 0.88 respectively, compared to expert visual scoring. Restricting DSVM classification to data where both experts’ scoring was consistent (76.73% of the data) led to a mean specificity, sensitivity and overall accuracy of 0.94, 0.82 and 0.92 respectively.
The DSVM framework outperforms classification with more standard multi-class “one-against-all” SVM and linear-discriminant analysis.
The promising results of the proposed methodology suggest that it may be a valuable alternative to existing automatic methods and that it could accelerate visual scoring by providing a robust starting hypnogram that can be further fine-tuned by expert inspection.
Automatic sleep stage recurrent neural classifier using energy features of EEG signals
2013, Neurocomputing
Citation Excerpt :
Based on the R&K classification rules, sleep recordings of adults can be divided into the following six sleep stages: wakefulness (WA), non-rapid eye-movement sleep stage 1 (NREM 1), NREM sleep stage 2 (NREM 2), NREM sleep stage 3 (NREM 3), NREM sleep stage 4 (NREM 4), and rapid eye-movement (REM). Sometimes, NREM 3 and NREM 4 are pooled into a single stage, slow wave sleep (SWS) [5,24]. In the area of sleep medicine, sleep hypnograms obtained by visually scoring the recordings from PSG are always used to present the recordings of the brain wave activity from EEG signals during a period of sleep [5,29].
This paper presents a recurrent neural classifier for automatically classifying sleep stages based on energy features from the EEG signal of the F_pz−C_z channel. The energy features were extracted from characteristic waves of EEG signals which were then used to classify different sleep stages. The recurrent neural classifier, utilizing energy features extracted from EEG signals, assigned each 30-s epoch to one of five possible sleep stages: wakefulness, NREM 1, NREM 2, SWS, and REM. Eight sleep recordings obtained from the Sleep-EDF database, which is available from the PhysioBank, were utilized to validate the proposed method. Using the features extracted by our research, classification performance of a feedforward neural network (FNN) and a probabilistic neural network (PNN) were compared to that of the proposed recurrent neural classifier. The classification rate of the recurrent neural classifier was found to be better (87.2%) than those of the two neural classifiers (81.1% for FNN and 81.8% for PNN). The result demonstrates that the proposed recurrent neural classifier using the energy features extracted from characteristic waves of EEG signals can classify sleep stages more efficiently and accurately using only a single EEG channel.

View all citing articles on Scopus

^☆: This work was supported by the following institutions: Les Ministères de la Région Bruxelloise et de la Communauté Française, l'Association Belge pour l'Etude de la Santé Mentale, le Service d'Electronique et Techniques Numériques de la FPMs, le Service d'Automatique de l'ULB, le Fond National de la Recherche Scientifique, la CGER et la Loterie Nationale.

^∗: We are grateful to S. Lejeune, P. Linkowski, G. Hoffman, Drs. Owen, B. Jacques and H. Haverals for their collaboration and advice.

View full text

Technical sectionAutomated sleep scoring: a comparative reliability study of two algorithmsAnalyse automatique du sommeil: fiabilité comparée de deux algorithmes☆

Abstract

Résumé

Electroenceph. clin. Neurophysiol.

Electroenceph. clin. Neurophysiol.

Electroenceph. clin. Neurophysiol.

Electroenceph. clin. Neurophysiol.

Electroenceph. clin. Neurophysiol.

Comput. Prog. Biomed.

Electroenceph. clin. Neurophysiol.

Measurement

Electroenceph. clin. Neurophysiol.

The role of training courses in multicenter trials: WHO experiences

Biol. Psychiat.

Technical section
Automated sleep scoring: a comparative reliability study of two algorithmsAnalyse automatique du sommeil: fiabilité comparée de deux algorithmes☆