How much data do you need before ML predictions can be trusted?
With a few hundred users, cross-validation made models look much better than they really were, and even a useless model could often look good.
Source
Estimation of minimal data sets sizes for machine learning predictions in digital mental health interventions
Study at a glance
- Design
- Computational / modelling — Learning curves: six classifiers trained on subsamples from 100 up to 3,654 users, each scored by cross-validation and on a fixed held-out test set.
- N
- N=3654 · 3,654 users of online eating-disorder prevention programmes; 731 of them form the fixed test set.
- Population
- Adult women using German digital eating-disorder prevention programmes (everyBody study)
- Outcome
- AUC for predicting programme dropout (fewer than four sessions completed), gap between cross-validation and test AUC, variance across folds, and where performance stops improving
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors took 3,654 users of online eating-disorder prevention programmes and tried to predict who would drop out. They trained six model types (from Naive Bayes to random forests and a small neural network) on subsamples ranging from 100 users to the full data set, using six feature groups that ranged from nearly useless questionnaire items to rich app-usage logs. Each model was scored both by 10-fold cross-validation and on the same set-aside test set of 731 users.
What they found
At 300 users or fewer, cross-validation scores exceeded true test scores by up to 0.12 AUC, and fold-to-fold scores swung widely (standard deviation 0.20 AUC at 100 users). With a feature set that carried no real signal, picking the best model by cross-validation still produced scores above 0.61 AUC in up to 70% of cases. Flexible tree-based models overfitted most on small data but did best on large data; simpler models stopped improving around 750 users, tree models around 1,000 and the neural network around 1500. The authors propose at least 500 users to limit overfitting and about 1000 when comparing richer feature sets.
The limits
What it doesn't show
This is one data set from one kind of programme with self-referred women only, so the thresholds may not transfer to other conditions, outcomes or populations. Only internal validation was done; nobody tested the models on data from a different service. Even the full data set may have been too small for the largest feature group, whose curve was still rising, and the neural network used was a shallow one rather than a modern deep model.
Key terms
- Learning curve
- A plot of model performance against training-set size, used to see whether more data still helps.
- Cross-validation (CV)
- Splitting training data into folds, training on some and scoring on the rest in turn, to estimate performance without the test set.
- Overfitting
- When a model learns noise in its training data, so its training or CV score is higher than its score on new data.
- AUC
- Area under the ROC curve; 0.5 means no better than chance and 1.0 means perfect ranking of cases.
- Nested cross-validation
- An inner CV loop tunes hyperparameters while an outer loop estimates performance, so tuning choices do not leak into the score.
Flashcards
0 of 12 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the main outcome the models tried to predict?
Common questions
Why would a model's cross-validation score be higher than its test score?
When data are scarce, the choice of model and hyperparameters is tuned to the same small folds used for scoring, so lucky patterns in those folds get rewarded and then fail to repeat on fresh data.
Does this mean simple models are always better?
No. Simple models overfit less on small data but plateau early; flexible models such as random forests and adaBoost overtook them once there were around a thousand or more users with informative features.
What should a researcher with only 200 participants do?
Treat CV results with caution, separate model selection from evaluation (for example with nested CV), keep a reasonably sized test set, report a learning curve, and ideally validate on an external data set.
More on Generalisation and overfitting