Concept
Algorithmic fairness and bias
4 studies1 discoveryEvidence last moved Sep 27, 2026
Algorithmic fairness asks whether a model performs or errs differently for groups defined by race, socioeconomic status or other attributes, and what fixing it does. The evidence here includes a large breast-cancer risk cohort, dementia prediction models in two county cohorts, surgical-skill assessment across hospitals, and demographic-injection tests on clinical NLP models.
Popular fixes like 'just remove race' or 'rebalance the data' sound fair but can backfire. These studies show why fairness must be checked per group, per metric and per site.
Studies
4
Findings
5
6 supporting · 0 challenging · 0 qualifying citations
Open tensions
1
Latest change
Concept page published
Algorithmic fairness and bias
Currently
What we know
- 'Race-blind' is not automatically fair.
- Performance gaps by SES appeared consistently in these cohorts.
- Mitigation moves bias around unless you check multiple measures.
- Audit at each site, not once.
- Demographic words alone can sway clinical NLP outputs.
Largest unresolved question
Group definitions don't agree: three SES measures overlapped poorly (only 12.0% of one measure's low-SES patients were low by another), so 'the' fairness gap depends on which measure is chosen.
Common misconceptions
If overall AUC is unchanged, a model change is fairness-neutral.
Removing race left AUC nearly unchanged but miscalibrated risk for Black and Asian women.
Higher error for low-SES patients proves the model is biased.
The dementia study notes poorer patients are more often undiagnosed, so some extra error may be label noise in the diagnosis codes.
Related
Claim ledger
What the evidence shows
Drawn from 4 studies in this library. Mix labels say which citation roles are present; they are not a strength score. Supports means evidence for a finding; Challenges means evidence against a stated position; Qualifies marks scope.
'Race-blind' is not automatically fair.
Removing race from a risk model can harm calibration for specific groups while leaving overall accuracy unchanged: AUC barely moved (0.682 vs 0.677), but risk was underestimated for Black women (expected/observed 0.61) and the share of Black women with advanced cancer flagged fell from 75.3% to 47.5%.
Performance gaps by SES appeared consistently in these cohorts.
Models can err more for lower-socioeconomic patients: across both cohorts and nearly all models, low-SES patients had higher balanced error rates than high-SES patients.
Mitigation moves bias around unless you check multiple measures.
Fixes targeted at one group definition can shift bias elsewhere: oversampling on one SES measure narrowed that gap but could widen gaps on another (8.38% to 30.37% in one case); in surgical assessment, baseline methods reduced underskilling but worsened overskilling, while TWIX modestly improved both.
- Are dementia-prediction models less accurate for poorer patients?
- Can expert hints make surgical-skill AI fairer?
Study Role Design N Population Outcome Are dementia-prediction models less accurate for poorer patients? Supports Computational / modellingRetrospective EHR-based modelling: LR, SVM, random forest and naive Bayes trained on 5 years of diagnosis codes to predict dementia in the next year, with bias measured by balanced error rate across SES-sex subgroups before and after SES oversampling. Two cohorts, not one analytic N: the Mayo Clinic Study of Aging (N=5890 enrolled) and the Rochester Epidemiology Project (N=290,528 records); analysed samples after age matching and exclusions are smaller, split 70/30 train/test. Adults aged 50 and over in Olmsted County, Minnesota, from a research cohort and a population medical-records cohort Balanced error rate of dementia prediction within subgroups defined by SES (national ADI, state ADI, HOUSES Index) and sex Can expert hints make surgical-skill AI fairer? Supports Computational / modellingRetrospective evaluation of a video-based surgical skill classifier (trained at one hospital, deployed at two others) for sub-cohort NPV/PPV gaps, before and after adding an expert frame-importance training signal (TWIX), compared with extra-data and video-pretraining baselines. No single N reported in the main text; video-sample counts per hospital are in a table not reproduced. Metrics are averaged over ten Monte Carlo cross-validation folds. Video clips of needle handling and needle driving during robot-assisted prostatectomy from three hospitals, plus clips of medical students suturing on a model Negative and positive predictive value per surgeon sub-cohort (underskilling and overskilling bias), and overall AUC Audit at each site, not once.
Bias can be site-specific and flip direction: which surgeon group was disadvantaged changed by skill and hospital, and TWIX's gains were smaller or negative at other hospitals.
Demographic words alone can sway clinical NLP outputs.
Language models can change outputs when only demographic words change: fine-tuned models changed their social-determinants answers in 14.3% of demographically altered sentence pairs versus 21.5% for ChatGPT.
Debates
Tensions and limits
Some items are genuine disagreements on the same question. Others mark different assays, populations, or outcomes.
Group definitions don't agree: three SES measures overlapped poorly (only 12.0% of one measure's low-SES patients were low by another), so 'the' fairness gap depends on which measure is chosen.
Group definitions don't agree: three SES measures overlapped poorly (only 12.0% of one measure's low-SES patients were low by another), so 'the' fairness gap depends on which measure is chosen.
PaperFren reads this as a limit on how far one study travels — different assays, populations, or outcomes — not a forced fight between papers.
Timeline
How understanding moved
Study years are when the paper was published. Evidence edits are dated changes to this page's claims. Explanations are when PaperFren added a Discovery — not a claim that the science happened that day.
2026
- Are dementia-prediction models less accurate for poorer patients?
- Dropping race from a model can make it less fair, and fixing one gap can widen another
Concept page published
Algorithmic fairness and bias
Change log
What changed
Dated edits to this page's evidence: studies added or removed from a claim, claims added or withdrawn, and new explanations tagged here. Rewordings are not listed.
- Concept page published
Papers
4 studies in this library bear on Algorithmic fairness and bias, ordered by citations.
- Can language models find social needs hidden in doctors' notes?
Fine-tuned language models found patients' social problems such as housing or transport trouble in clinic notes far more often than billing codes did, and were less swayed by race or gender wording than ChatGPT.
- Can expert hints make surgical-skill AI fairer?
An AI that grades surgeons from video judged some groups more harshly than others, and teaching it which video moments experts consider important reduced those gaps while also making it more accurate.
- Does removing race from a cancer risk model make it fairer?
Taking race and ethnicity out of a breast cancer risk model barely changed its overall accuracy but made its risk estimates wrong for Black and Asian women in opposite directions.
- Are dementia-prediction models less accurate for poorer patients?
Standard machine-learning models predicting dementia from medical records made more errors for people of lower socioeconomic status, and rebalancing the training data by status narrowed but did not simply remove that gap.
Compare studies
Select 2–10 studies. Design and N are labels, not a ranking.
Nothing selected yet.
Questions
What is still open
Group definitions don't agree: three SES measures overlapped poorly (only 12.0% of one measure's low-SES patients were low by another), so 'the' fairness gap depends on which measure is chosen.
Ask PaperFren about Algorithmic fairness and bias
Study this conceptflashcards and short-answer questions
Why might removing race from a breast-cancer risk model be harmful?
In a large cohort, removing race barely changed overall AUC but underestimated risk for Black women and overestimated it for Asian women. Fewer Black women who later developed advanced cancer would have been flagged (75.3% to 47.5%). The model is logistic regression, and the study didn't measure harms of extra screening.
What lessons about bias mitigation come from the SES and surgical-skill studies?
Oversampling on one SES measure narrowed that gap but could widen gaps on another. In surgical assessment, baseline mitigation reduced one bias type while worsening another, and TWIX's benefits varied by hospital. Fairness must be checked across measures and sites.