Sample size · Overfitting · Baselines
On a few hundred samples, cross-validation flatters models and deep features lose to linear baselines
Save this development or follow its topic to track what changes.
Short answer
With only a few hundred labelled samples, simple regularised models are the right baseline, and cross-validated scores should be treated as optimistic.
What happened
A benchmark of 24 classification and 26 survival tasks on RNA-seq data found that features from PCA, autoencoders and variational autoencoders lowered out-of-sample performance compared with a regularised linear model on log-ratio-normalised genes. A study of 3,654 users of eating-disorder prevention programmes found that at 300 users or fewer, cross-validation overstated test AUC by up to 0.12, and a feature set with no real signal still scored above 0.61 AUC in up to 70% of cases. A third paper found that accuracy depended more on how strongly the classes differed than on the classifier.
Why it matters
Students often learn that deeper models and more features are the way forward, and that cross-validation is an honest estimate. On the dataset sizes common in medicine, both assumptions can fail together: the flexible model overfits, and the validation scheme hides it.
Evidence
- Study type
- Computational benchmarks and learning-curve analyses
- Sample
- Supervised RNA-seq tasks of typically a few hundred samples; 3,654 programme users; simulated and real sets from 16 to 5,000 samples
- Journal
- BMC Bioinformatics · peer reviewed
- Replication
- Consistent direction across three independent groups and data types, but each proposes its own threshold
- Limitations
- Only specific architectures and embedding sizes were tested; the sample-size thresholds come from one programme dataset and three real datasets and were not validated on new data; all validation was internal.
What this connects to
Sources
The 3 studies this explanation is built from, by the role each plays. Every source links to PaperFren’s explanation of it and to the original paper.
Primary study
- Do deep learned features help predict disease from gene data?
Across dozens of prediction tasks from gene expression data, fancy learned representations did not beat a simple regularised linear model on properly normalised data.
What it does not showLimitations
The result covers bulk RNA-seq phenotype prediction with the specific architectures and 512-dimensional embeddings tested; other designs or much larger labelled datasets might change it. Supervised tasks had only a few hundred samples each, which may favour simple models. The authors suggest simple methods may already be near the limit of what bulk expression can predict, but they do not test that directly.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Supporting evidence
- How much data do you need before ML predictions can be trusted?
With a few hundred users, cross-validation made models look much better than they really were, and even a useless model could often look good.
What it does not showLimitations
This is one data set from one kind of programme with self-referred women only, so the thresholds may not transfer to other conditions, outcomes or populations. Only internal validation was done; nobody tested the models on data from a different service. Even the full data set may have been too small for the largest feature group, whose curve was still rising, and the neural network used was a shallow one rather than a modern deep model.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
- Can effect size tell you if your ML data set is big enough?
When the two classes differ strongly, classifiers reach high, stable accuracy even on modest samples; when they barely differ, adding data helps little.
What it does not showLimitations
The two criteria are rules of thumb derived from only three real data sets and simple simulations; they were not tested on new data sets to see if they predict success. The 80% accuracy threshold is borrowed by analogy from the 80% power convention in statistics rather than justified for ML. Only small, classic classifiers on tabular two-class problems were used, so the results say little about deep learning or multi-class tasks, and the authors admit the guideline cannot be generalised to all biomedical data.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Before
Learned deep representations should add value over raw features, and a cross-validated score is a fair estimate of how a model will do on new data.
Now
On small biomedical datasets, learned representations did not help and cross-validation overstated performance, with one study proposing at least 500 users before trusting results. The thresholds come from single datasets and simple rules of thumb, and much larger labelled datasets could change the picture.