Generalisation and overfitting
How much data do you need before ML predictions can be trusted?
Open access · cc by · source: Europe PMC
With a few hundred users, cross-validation made models look much better than they really were, and even a useless model could often look good.
Study at a glance
- Design
- Computational / modelling — Learning curves: six classifiers trained on subsamples from 100 up to 3,654 users, each scored by cross-validation and on a fixed held-out test set.
- N
- N=3654 · 3,654 users of online eating-disorder prevention programmes; 731 of them form the fixed test set.
- Population
- Adult women using German digital eating-disorder prevention programmes (everyBody study)
- Outcome
- AUC for predicting programme dropout (fewer than four sessions completed), gap between cross-validation and test AUC, variance across folds, and where performance stops improving
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
At 300 users or fewer, cross-validation scores exceeded true test scores by up to 0.12 AUC, and fold-to-fold scores swung widely (standard deviation 0.20 AUC at 100 users). With a feature set that carried no real signal, picking the best model by cross-validation still produced scores above 0.61 AUC in up to 70% of cases. Flexible tree-based models overfitted most on small data but did best on large data; simpler models stopped improving around 750 users, tree models around 1,000 and the neural network around 1500. The authors propose at least 500 users to limit overfitting and about 1000 when comparing richer feature sets.
Methodology
The authors took 3,654 users of online eating-disorder prevention programmes and tried to predict who would drop out. They trained six model types (from Naive Bayes to random forests and a small neural network) on subsamples ranging from 100 users to the full data set, using six feature groups that ranged from nearly useless questionnaire items to rich app-usage logs. Each model was scored both by 10-fold cross-validation and on the same set-aside test set of 731 users.
Limitations
This is one data set from one kind of programme with self-referred women only, so the thresholds may not transfer to other conditions, outcomes or populations. Only internal validation was done; nobody tested the models on data from a different service. Even the full data set may have been too small for the largest feature group, whose curve was still rising, and the neural network used was a shallow one rather than a modern deep model.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Small datasets make cross-validation scores optimistic and noisy.
With 300 or fewer users, 10-fold cross-validation overstated held-out test AUC by up to 0.12 (0.05 on average) when predicting dropout from digital mental-health programmes, and fold-to-fold scores had a standard deviation of 0.20 AUC at 100 users.
Evidence for the claim as stated.
Model selection on small data can manufacture apparent signal from noise.
Picking the best model by cross-validation on a feature set with no real signal still produced scores above 0.61 AUC in up to 70% of cases.
Evidence for the claim as stated.
More data reduces variance, but cannot rescue features that carry little signal.
Classification accuracy varied widely on small subsets and stabilised as samples grew (for arrhythmia data, variation shrank sharply beyond about 120 samples); datasets with tiny effect sizes stayed below 80% accuracy whatever classifier was used.
Evidence for the claim as stated.
The proposed minimum sample sizes differ: one study suggests at least 500 users (about 1,000 when comparing richer feature sets) for dropout prediction, while another proposes criteria based on effect size of 0.5 or more and stable accuracy above 80%, rather than a fixed N.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
The proposed minimum sample sizes differ: one study suggests at least 500 users (about 1,000 when comparing richer feature sets) for dropout prediction, while another proposes criteria based on effect size of 0.5 or more and stable accuracy above 80%, rather than a fixed N.
Discoveries this paper informs or conflicts with
- On a few hundred samples, cross-validation flatters models and deep features lose to linear baselines
This paper informs this development.
Related papers in this topic
Same topic cluster — not a recommendation engine.