Fairness · Bias · Clinical prediction
Dropping race from a model can make it less fair, and fixing one gap can widen another
Save this development or follow its topic to track what changes.
Short answer
Excluding a protected attribute does not guarantee fairness; calibration by group, the choice of group measure and the deployment site all change the answer.
What happened
In 931,186 women, removing race and ethnicity barely changed AUC (0.682 to 0.677), but risk was underestimated for Black women (expected/observed 0.61) and overestimated for Asian women (1.28); among Black women who developed advanced cancer, the share flagged fell from 75.3% to 47.5%. Dementia models made more errors for lower-SES patients, and oversampling on one SES measure could widen the gap on another. A surgeon-skill AI showed biases whose direction flipped by hospital and skill, and training it on expert explanations reduced both kinds of bias modestly.
Why it matters
A common intuition is that leaving a sensitive attribute out makes a model fair. These results show that overall accuracy can hide group-level harm, and that fairness depends on which measure and which site you check.
Evidence
- Study type
- Retrospective cohort model evaluations
- Sample
- 931,186 women and 3,294,431 mammograms; two dementia cohorts (5,890 and 290,528 records); multi-hospital surgical videos
- Journal
- npj Digital Medicine · peer reviewed
- Replication
- Separate groups in different clinical areas find group-level gaps that overall metrics hide; the specific remedies are not replicated
- Limitations
- Small subgroups with wide intervals, single-region cohorts, retrospective data only, and bias defined as any performance gap without a threshold for what matters.
What this connects to
Sources
The 3 studies this explanation is built from, by the role each plays. Every source links to PaperFren’s explanation of it and to the original paper.
Primary study
- Does removing race from a cancer risk model make it fairer?
Taking race and ethnicity out of a breast cancer risk model barely changed its overall accuracy but made its risk estimates wrong for Black and Asian women in opposite directions.
What it does not showLimitations
The model is a logistic regression, so this is a lesson about removing a protected attribute rather than about a complex machine-learning system. Some groups were small, giving wide confidence intervals, and Pacific Islander women could not be analysed separately. The study did not measure harms such as false positives from extra screening, and it cannot say which risk thresholds are best or whether using the model improves outcomes.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Supporting evidence
- Are dementia-prediction models less accurate for poorer patients?
Standard machine-learning models predicting dementia from medical records made more errors for people of lower socioeconomic status, and rebalancing the training data by status narrowed but did not simply remove that gap.
What it does not showLimitations
Both cohorts come from one mostly White Midwestern county, so the size and direction of bias may differ elsewhere. Dementia in the records cohort was identified from diagnosis codes, and poorer patients are more likely to go undiagnosed, so some of the extra error may be label noise rather than model failure. The models used only coded diagnoses, not cognitive tests or biomarkers, so they are weak predictors built to study bias, not clinical tools. The authors did not formally test whether the oversampling improvements were statistically significant, and they used only one fairness metric.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
- Can expert hints make surgical-skill AI fairer?
An AI that grades surgeons from video judged some groups more harshly than others, and teaching it which video moments experts consider important reduced those gaps while also making it more accurate.
What it does not showLimitations
Bias is defined purely as any performance gap between sub-cohorts, with no threshold for what gap matters, and was measured once on retrospective data rather than in real credentialing. Groups such as race and sex of surgeons were excluded because of small samples, and only two suturing skills were studied. The mitigation effect varied by hospital and sometimes reversed, so TWIX is not shown to be a general fix.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Before
Removing race or other sensitive attributes from a model removes its bias, and a fairness check at one site on one measure is enough.
Now
In these studies removal caused miscalibration by group without changing AUC, and mitigation on one measure could worsen another. The breast model is logistic regression, the dementia cohorts are from one mostly White county with possible label noise, and harms such as missed cancers were not measured directly.