Can effect size tell you if your ML data set is big enough?
When the two classes differ strongly, classifiers reach high, stable accuracy even on modest samples; when they barely differ, adding data helps little.
Source
Evaluation of a decided sample size in machine learning applications
Study at a glance
- Design
- Computational / modelling — Repeated random subsampling (100 repeats per size, from 16 samples upward) of simulated data and three real biomedical data sets, measuring effect size and accuracy of five classifiers.
- N
- No single N: simulated sets of 16 to 2500 samples; a heart-attack set trimmed to 276 patients; an ECG arrhythmia set of 5000 beats; a sleep-diary set of 3000 samples.
- Population
- Simulated two-class data; UCI Cleveland heart-disease patients; MIT-BIH arrhythmia heartbeats; women's menstrual-cycle sleep diaries
- Outcome
- Classification accuracy of SVM, logistic regression, decision tree, neural network and Naive Bayes, and 'average' versus 'grand' Cohen's d, as sample size grows
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors generated simulated two-class data with either large or tiny differences between classes, and also used three real data sets (heart attack, ECG arrhythmia and sleep diaries). They drew random subsets from 16 samples upward, repeated each draw 100 times, and measured both the classifiers' accuracy and the effect size (Cohen's d) calculated two ways, an 'average' and a 'grand' version. Five standard classifiers were compared.
What they found
Data with large effect sizes (above about 0.8) gave accuracies above 90%, while data with tiny effect sizes (below 0.2) stayed under 80% regardless of classifier. Small subsets gave very variable accuracy and effect sizes; for the arrhythmia data, variation shrank sharply once samples passed about 120. The sleep data never exceeded about 67% accuracy. The authors propose two checks: at least one effect size of 0.5 or more, and accuracy of 80% or more that changes by less than 10% as more samples are added.
The limits
What it doesn't show
The two criteria are rules of thumb derived from only three real data sets and simple simulations; they were not tested on new data sets to see if they predict success. The 80% accuracy threshold is borrowed by analogy from the 80% power convention in statistics rather than justified for ML. Only small, classic classifiers on tabular two-class problems were used, so the results say little about deep learning or multi-class tasks, and the authors admit the guideline cannot be generalised to all biomedical data.
Key terms
- Cohen's d
- The difference between two group means divided by their pooled standard deviation; about 0.2 is small and 0.8 is large.
- Average vs grand effect size
- Two ways the authors compute d across many features: from averaged per-feature means and variances, or from means and variances pooled over all values.
- Subsampling
- Drawing random subsets of a data set at different sizes to see how results change with sample size.
- Statistical power
- The probability that a study detects a real effect; 80% is the usual target in sample-size planning.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What did the authors vary in their simulated data sets?
Common questions
Why does a large effect size make classification easy?
If the two classes' feature values barely overlap, almost any classifier can draw a boundary between them, so accuracy is high even with few examples.
Does more data always rescue a weak data set?
No. In the sleep data, accuracy crept up only slightly as samples grew and stayed poor, suggesting the features simply did not separate the classes; the authors say better features or a new design are needed.
Can a small sample ever be enough?
The authors argue yes, if the data are high quality with a large effect size, because accuracy and effect size then stabilise quickly.