Are dementia-prediction models less accurate for poorer patients?
Standard machine-learning models predicting dementia from medical records made more errors for people of lower socioeconomic status, and rebalancing the training data by status narrowed but did not simply remove that gap.
Source
Assessing the Impact of Sociodemographic Factors on Artificial Intelligence Models in Predicting Dementia: Retrospective Cohort Study
Study at a glance
- Design
- Computational / modelling — Retrospective EHR-based modelling: LR, SVM, random forest and naive Bayes trained on 5 years of diagnosis codes to predict dementia in the next year, with bias measured by balanced error rate across SES-sex subgroups before and after SES oversampling.
- N
- Two cohorts, not one analytic N: the Mayo Clinic Study of Aging (N=5890 enrolled) and the Rochester Epidemiology Project (N=290,528 records); analysed samples after age matching and exclusions are smaller, split 70/30 train/test.
- Population
- Adults aged 50 and over in Olmsted County, Minnesota, from a research cohort and a population medical-records cohort
- Outcome
- Balanced error rate of dementia prediction within subgroups defined by SES (national ADI, state ADI, HOUSES Index) and sex
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
Using two Minnesota cohorts, the authors built logistic regression, support vector machine, random forest and naive Bayes models that used five years of grouped diagnosis codes, age and sex to predict a dementia diagnosis in the following year. They measured fairness with the balanced error rate (the average of false-positive and false-negative rates) in subgroups defined by sex and three socioeconomic measures: national and state Area Deprivation Index and a housing-based HOUSES Index. They then oversampled low-SES patients in the training data with SMOTE-NC and retrained.
What they found
Across both cohorts and nearly all models, low-SES patients had higher balanced error rates than high-SES patients, the one exception being naive Bayes in the larger records cohort; high-SES men usually had the lowest error in the research cohort. The three SES measures overlapped poorly, for example only 12.0% of HOUSES-low patients were also low by national ADI. Oversampling on a given SES measure narrowed the gap on that measure, sometimes flipping it so low-SES patients did better, but could widen gaps on another measure (in one random forest case from 8.38% to 30.37%) and slightly reduced overall accuracy.
The limits
What it doesn't show
Both cohorts come from one mostly White Midwestern county, so the size and direction of bias may differ elsewhere. Dementia in the records cohort was identified from diagnosis codes, and poorer patients are more likely to go undiagnosed, so some of the extra error may be label noise rather than model failure. The models used only coded diagnoses, not cognitive tests or biomarkers, so they are weak predictors built to study bias, not clinical tools. The authors did not formally test whether the oversampling improvements were statistically significant, and they used only one fairness metric.
Key terms
- Balanced error rate (BER)
- The average of the false-positive and false-negative rates; useful when one class, like dementia, is rare.
- Algorithmic bias
- Systematic differences in a model's errors across groups of people, such as by income or sex.
- Area Deprivation Index
- A neighbourhood-level measure of socioeconomic disadvantage built from census data, ranked nationally or within a state.
- SMOTE-NC
- An oversampling method that creates synthetic training examples for an under-represented group, handling both numeric and categorical features.
- Fairness-accuracy trade-off
- The tendency for interventions that equalise error across groups to slightly reduce overall accuracy.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which group generally had higher balanced error rates?
Common questions
Why would a model be worse for low-SES patients?
They are fewer in the training data and have patchier medical records, so the model learns mostly from the richer, more complete records of higher-SES patients.
Why does it matter which SES measure you use?
The measures flag largely different people, so fixing bias defined by one measure can leave or even worsen bias defined by another.
Is a 7-point difference in error rate important?
At scale, yes: the authors estimate about 69 extra misclassified low-SES patients per 1000 screened, meaning delayed diagnoses or needless follow-ups.
More on Bias and fairness