Concept
Generalization and sample size
4 studies1 discoveryEvidence last moved Sep 27, 2026
Generalization is how well a trained model performs on data it has never seen, as opposed to the data used to fit or tune it. The evidence here comes from computational studies of biomedical prediction: learning-curve experiments on dropout prediction and on simulated and clinical tabular data, a four-hospital surgical mortality model, and a sleep-staging network tested across 21 datasets.
A model's reported score is only as trustworthy as the way it was evaluated. These studies show that small data and in-sample evaluation can make useless models look good, and that moving to a new hospital or dataset is the real test.
Studies
4
Findings
5
6 supporting · 0 challenging · 1 qualifying citations
Open tensions
1
Latest change
Concept page published
Generalization and sample size
Currently
What we know
- Small datasets make cross-validation scores optimistic and noisy.
- Model selection on small data can manufacture apparent signal from noise.
- More data reduces variance, but cannot rescue features that carry little signal.
- External validation across sites reveals what internal testing cannot, and training-set size helps transfer.
- Training on many heterogeneous datasets can yield a model that holds up across cohorts.
Largest unresolved question
The proposed minimum sample sizes differ: one study suggests at least 500 users (about 1,000 when comparing richer feature sets) for dropout prediction, while another proposes criteria based on effect size of 0.5 or more and stable accuracy above 80%, rather than a fixed N.
Common misconceptions
A high cross-validation score means the model will work on new data.
On small data, cross-validation that also drives model selection can exceed true test performance by a wide margin, even for a model built on uninformative features. A separate test set, nested cross-validation and external sites are needed.
There is one universal minimum sample size for machine learning.
The data needed depends on how strong the signal is and how flexible the model is: simpler models plateaued around 750 users while a neural network kept improving until about 1,500, and weak-effect data never reached high accuracy.
Related
Claim ledger
What the evidence shows
Drawn from 4 studies in this library. Mix labels say which citation roles are present; they are not a strength score. Supports means evidence for a finding; Challenges means evidence against a stated position; Qualifies marks scope.
Small datasets make cross-validation scores optimistic and noisy.
With 300 or fewer users, 10-fold cross-validation overstated held-out test AUC by up to 0.12 (0.05 on average) when predicting dropout from digital mental-health programmes, and fold-to-fold scores had a standard deviation of 0.20 AUC at 100 users.
Model selection on small data can manufacture apparent signal from noise.
Picking the best model by cross-validation on a feature set with no real signal still produced scores above 0.61 AUC in up to 70% of cases.
More data reduces variance, but cannot rescue features that carry little signal.
Classification accuracy varied widely on small subsets and stabilised as samples grew (for arrhythmia data, variation shrank sharply beyond about 120 samples); datasets with tiny effect sizes stayed below 80% accuracy whatever classifier was used.
- Can effect size tell you if your ML data set is big enough?
- How much data do you need before ML predictions can be trusted?
Study Role Design N Population Outcome Can effect size tell you if your ML data set is big enough? Supports Computational / modellingRepeated random subsampling (100 repeats per size, from 16 samples upward) of simulated data and three real biomedical data sets, measuring effect size and accuracy of five classifiers. No single N: simulated sets of 16 to 2500 samples; a heart-attack set trimmed to 276 patients; an ECG arrhythmia set of 5000 beats; a sleep-diary set of 3000 samples. Simulated two-class data; UCI Cleveland heart-disease patients; MIT-BIH arrhythmia heartbeats; women's menstrual-cycle sleep diaries Classification accuracy of SVM, logistic regression, decision tree, neural network and Naive Bayes, and 'average' versus 'grand' Cohen's d, as sample size grows How much data do you need before ML predictions can be trusted? Supports Computational / modellingLearning curves: six classifiers trained on subsamples from 100 up to 3,654 users, each scored by cross-validation and on a fixed held-out test set. N=3654 · 3,654 users of online eating-disorder prevention programmes; 731 of them form the fixed test set. Adult women using German digital eating-disorder prevention programmes (everyBody study) AUC for predicting programme dropout (fewer than four sessions completed), gap between cross-validation and test AUC, variance across folds, and where performance stops improving External validation across sites reveals what internal testing cannot, and training-set size helps transfer.
In a retrospective study of about 454,000 surgical patients at four Korean hospitals, models trained at the hospitals with the most data transferred best to other sites (AUROC 0.941 when the largest hospital's model was tested elsewhere), and the most important predictor differed between hospitals.
Training on many heterogeneous datasets can yield a model that holds up across cohorts.
A single sleep-staging network trained on 16 clinical studies reached a mean F1 of 0.79 across 21 datasets and scored comparably to the best human expert on held-out cohorts, although N1 sleep stayed hard (F1 about 0.53).
- Can one neural network score sleep across any clinic's data?— Data were mostly from Europe and North America and from healthy or apnea populations; other patient groups are untested.
Debates
Tensions and limits
Some items are genuine disagreements on the same question. Others mark different assays, populations, or outcomes.
The proposed minimum sample sizes differ: one study suggests at least 500 users (about 1,000 when comparing richer feature sets) for dropout prediction, while another proposes criteria based on effect size of 0.5 or more and stable accuracy above 80%, rather than a fixed N.
The proposed minimum sample sizes differ: one study suggests at least 500 users (about 1,000 when comparing richer feature sets) for dropout prediction, while another proposes criteria based on effect size of 0.5 or more and stable accuracy above 80%, rather than a fixed N.
- How much data do you need before ML predictions can be trusted?
- Can effect size tell you if your ML data set is big enough?
Study Role Design N Population Outcome How much data do you need before ML predictions can be trusted? Supports Computational / modellingLearning curves: six classifiers trained on subsamples from 100 up to 3,654 users, each scored by cross-validation and on a fixed held-out test set. N=3654 · 3,654 users of online eating-disorder prevention programmes; 731 of them form the fixed test set. Adult women using German digital eating-disorder prevention programmes (everyBody study) AUC for predicting programme dropout (fewer than four sessions completed), gap between cross-validation and test AUC, variance across folds, and where performance stops improving Can effect size tell you if your ML data set is big enough? Supports Computational / modellingRepeated random subsampling (100 repeats per size, from 16 samples upward) of simulated data and three real biomedical data sets, measuring effect size and accuracy of five classifiers. No single N: simulated sets of 16 to 2500 samples; a heart-attack set trimmed to 276 patients; an ECG arrhythmia set of 5000 beats; a sleep-diary set of 3000 samples. Simulated two-class data; UCI Cleveland heart-disease patients; MIT-BIH arrhythmia heartbeats; women's menstrual-cycle sleep diaries Classification accuracy of SVM, logistic regression, decision tree, neural network and Naive Bayes, and 'average' versus 'grand' Cohen's d, as sample size grows
PaperFren reads this as a limit on how far one study travels — different assays, populations, or outcomes — not a forced fight between papers.
Timeline
How understanding moved
Study years are when the paper was published. Evidence edits are dated changes to this page's claims. Explanations are when PaperFren added a Discovery — not a claim that the science happened that day.
2026
- On a few hundred samples, cross-validation flatters models and deep features lose to linear baselines
Concept page published
Generalization and sample size
Change log
What changed
Dated edits to this page's evidence: studies added or removed from a claim, claims added or withdrawn, and new explanations tagged here. Rewordings are not listed.
- Concept page published
Papers
4 studies in this library bear on Generalization and sample size, ordered by citations.
- Can one neural network score sleep across any clinic's data?
A single network trained on very diverse sleep recordings scored sleep about as well as the best of five human experts, even on datasets it never saw, without retraining.
- Can effect size tell you if your ML data set is big enough?
When the two classes differ strongly, classifiers reach high, stable accuracy even on modest samples; when they barely differ, adding data helps little.
- How much data do you need before ML predictions can be trusted?
With a few hundred users, cross-validation made models look much better than they really were, and even a useless model could often look good.
- Does a surgery-death prediction model work at other hospitals?
A lightweight gradient-boosted model using only routine lab values and basic surgical details predicted 30-day post-surgery death well and kept that accuracy when moved to other hospitals.
Compare studies
Select 2–10 studies. Design and N are labels, not a ranking.
Nothing selected yet.
Questions
What is still open
The proposed minimum sample sizes differ: one study suggests at least 500 users (about 1,000 when comparing richer feature sets) for dropout prediction, while another proposes criteria based on effect size of 0.5 or more and stable accuracy above 80%, rather than a fixed N.
Ask PaperFren about Generalization and sample size
Study this conceptflashcards and short-answer questions
Why can a model trained on 200 participants report an AUC of 0.65 even when its features carry no information?
When the same small folds are used both to choose among models and hyperparameters and to score them, chance patterns get rewarded. A learning-curve study of dropout prediction found that cross-validation exceeded held-out test AUC by up to 0.12 at 300 users or fewer, and that choosing the best model on an uninformative feature set gave scores above 0.61 in up to 70% of cases. Separating tuning from evaluation and using a held-out test set exposes this.
What does multi-site validation add beyond a single-hospital test set?
It checks whether a model transfers to different patients, practices and recording habits. In a four-hospital surgical mortality study, models from larger hospitals transferred best and the most important predictor varied by site. Because all sites were in one country and the design was retrospective, it still does not show benefit to patients or transfer to other health systems.
Flashcards
0 of 8 answers reviewed