Model evaluation
Can effect size tell you if your ML data set is big enough?
Open access · cc by · source: Europe PMC
When the two classes differ strongly, classifiers reach high, stable accuracy even on modest samples; when they barely differ, adding data helps little.
Study at a glance
- Design
- Computational / modelling — Repeated random subsampling (100 repeats per size, from 16 samples upward) of simulated data and three real biomedical data sets, measuring effect size and accuracy of five classifiers.
- N
- No single N: simulated sets of 16 to 2500 samples; a heart-attack set trimmed to 276 patients; an ECG arrhythmia set of 5000 beats; a sleep-diary set of 3000 samples.
- Population
- Simulated two-class data; UCI Cleveland heart-disease patients; MIT-BIH arrhythmia heartbeats; women's menstrual-cycle sleep diaries
- Outcome
- Classification accuracy of SVM, logistic regression, decision tree, neural network and Naive Bayes, and 'average' versus 'grand' Cohen's d, as sample size grows
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
Data with large effect sizes (above about 0.8) gave accuracies above 90%, while data with tiny effect sizes (below 0.2) stayed under 80% regardless of classifier. Small subsets gave very variable accuracy and effect sizes; for the arrhythmia data, variation shrank sharply once samples passed about 120. The sleep data never exceeded about 67% accuracy. The authors propose two checks: at least one effect size of 0.5 or more, and accuracy of 80% or more that changes by less than 10% as more samples are added.
Methodology
The authors generated simulated two-class data with either large or tiny differences between classes, and also used three real data sets (heart attack, ECG arrhythmia and sleep diaries). They drew random subsets from 16 samples upward, repeated each draw 100 times, and measured both the classifiers' accuracy and the effect size (Cohen's d) calculated two ways, an 'average' and a 'grand' version. Five standard classifiers were compared.
Limitations
The two criteria are rules of thumb derived from only three real data sets and simple simulations; they were not tested on new data sets to see if they predict success. The 80% accuracy threshold is borrowed by analogy from the 80% power convention in statistics rather than justified for ML. Only small, classic classifiers on tabular two-class problems were used, so the results say little about deep learning or multi-class tasks, and the authors admit the guideline cannot be generalised to all biomedical data.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
More data reduces variance, but cannot rescue features that carry little signal.
Classification accuracy varied widely on small subsets and stabilised as samples grew (for arrhythmia data, variation shrank sharply beyond about 120 samples); datasets with tiny effect sizes stayed below 80% accuracy whatever classifier was used.
Evidence for the claim as stated.
The proposed minimum sample sizes differ: one study suggests at least 500 users (about 1,000 when comparing richer feature sets) for dropout prediction, while another proposes criteria based on effect size of 0.5 or more and stable accuracy above 80%, rather than a fixed N.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
The proposed minimum sample sizes differ: one study suggests at least 500 users (about 1,000 when comparing richer feature sets) for dropout prediction, while another proposes criteria based on effect size of 0.5 or more and stable accuracy above 80%, rather than a fixed N.
Discoveries this paper informs or conflicts with
- On a few hundred samples, cross-validation flatters models and deep features lose to linear baselines
This paper informs this development.
Related papers in this topic
Same topic cluster — not a recommendation engine.