Generalisation and overfitting
Can one neural network score sleep across any clinic's data?
Open access · cc by · source: Europe PMC
A single network trained on very diverse sleep recordings scored sleep about as well as the best of five human experts, even on datasets it never saw, without retraining.
Study at a glance
- Design
- Computational / modelling — U-Net-style fully convolutional network trained on pooled polysomnography datasets with random EEG/EOG channel sampling; evaluated on held-out subjects, held-out datasets and consensus-scored data.
- N
- N=15660 · 15,660 participants contributing 19,924 PSG records across 21 datasets; consensus comparisons used 25 healthy subjects (DOD-H) and 55 OSA patients (DOD-O).
- Population
- Polysomnography recordings from healthy people and sleep-disordered patients in 16 clinical studies, mainly from North America and Europe.
- Outcome
- Per-stage and mean F1 agreement with expert sleep stage labels; F1 of a classifier separating OSA patients from controls using high-frequency stages.
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
Across 21 datasets the mean F1 was 0.79, with stage N1 hardest to score (about 0.53) and Wake and REM easiest. On 25 healthy subjects the model matched the best human expert (F1 0.79), and on 55 apnea patients it scored 0.76, higher than every expert, though neither difference from the best expert was statistically significant. Majority voting across channel combinations was always at least as good as the average channel. Features from high-frequency stage outputs raised apnea-versus-healthy classification F1 from 0.60 to a peak of 0.94.
Methodology
The authors trained U-Sleep, a fully convolutional encoder-decoder network inspired by U-Net, on 19,924 overnight recordings from 15,660 participants in 16 clinical studies. During training each example used a randomly chosen EEG and EOG channel pair, so the model had to work regardless of electrode placement, hardware or sampling rate. Eight datasets were kept entirely out of development to test generalisation, and two consensus-scored datasets let them compare the model with five clinical experts. They also tested whether the model's very high-frequency stage outputs could help a simple classifier separate sleep apnea patients from healthy people.
Limitations
Almost all data came from Europe and North America and mostly from healthy people or apnea patients, so performance on people with highly abnormal brain activity (stroke, neurodegeneration, psychiatric illness) is unknown. The authors did not tune the architecture, and limited demographic information means some biases cannot be ruled out. The apnea classification with high-frequency stages is a small proof-of-concept on two datasets, not a validated diagnostic test. Ground truth labels are themselves noisy human scores, which caps how accuracy can be judged.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Training on many heterogeneous datasets can yield a model that holds up across cohorts.
A single sleep-staging network trained on 16 clinical studies reached a mean F1 of 0.79 across 21 datasets and scored comparably to the best human expert on held-out cohorts, although N1 sleep stayed hard (F1 about 0.53).
Evidence for the claim as stated.
Training on many heterogeneous datasets can yield a model that holds up across cohorts.
A single sleep-staging network trained on 16 clinical studies reached a mean F1 of 0.79 across 21 datasets and scored comparably to the best human expert on held-out cohorts, although N1 sleep stayed hard (F1 about 0.53).
Scope note — Data were mostly from Europe and North America and from healthy or apnea populations; other patient groups are untested.
Limits the claim's scope: a different population, assay, or outcome.
Related papers in this topic
Same topic cluster — not a recommendation engine.