Concept · artificial-intelligence
Interpretable machine learning
6 studies1 discoveryEvidence last moved Sep 27, 2026
Interpretable machine learning aims for models whose predictions humans can understand, either by building simple models (like point scores) or by designing or probing complex networks so their parts map to meaningful things (pathways, DNA motifs). The evidence here is computational studies in clinical prediction and genomics, mostly retrospective and single-dataset.
Students often assume interpretability always costs accuracy, or that a highlighted feature is a biological explanation. These papers show both assumptions are too simple.
Studies
6
Findings
4
7 supporting · 0 challenging · 1 qualifying citations
Open tensions
1
Latest change
Concept page published
Interpretable machine learning
Currently
What we know
- Interpretability can cost little accuracy on tabular clinical data.
- Knowledge-structured networks give readable parts, with modest gains.
- Same accuracy, very different internal representations.
- An explanation is a lead to check, not a finding.
Largest unresolved question
'Interpretable' means different things: point scores are interpretable by simplicity (not measured with clinicians), knowledge-guided networks by structure, and a robot navigation paper uses 'explainable' to mean reward terms have physical meaning, not that the policy was analysed.
Common misconceptions
A filter matching a known motif database proves the network learned that motif.
Many filters matched some database motif without matching the true one, so database matching was misleading.
Interpretable models always sacrifice accuracy.
AutoScore's point score matched logistic regression and LASSO, and knowledge-guided PASNet and TranSynergy slightly outperformed their baselines.
Related
Claim ledger
What the evidence shows
Drawn from 6 studies in this library. Mix labels say which citation roles are present; they are not a strength score. Supports means evidence for a finding; Challenges means evidence against a stated position; Qualifies marks scope.
Interpretability can cost little accuracy on tabular clinical data.
Simple point scores can come close to complex models: a nine-variable AutoScore reached AUC 0.780 vs 0.778 for 24-variable logistic regression, and only the full random forest was clearly more accurate (0.809) while being uninterpretable and worse calibrated.
Knowledge-structured networks give readable parts, with modest gains.
Building biology into the architecture can make networks both usable and somewhat more accurate: PASNet's pathway-constrained network had the highest mean AUC (0.6622) for glioblastoma prognosis, and TranSynergy improved drug-synergy correlations by 3–5% over DeepSynergy.
- Can wiring a neural net by biology help small-data prognosis?— AUC 0.66 is modest, one dataset, no external validation.
Study Role Design N Population Outcome Can wiring a neural net by biology help small-data prognosis? Supports Computational / modellingNew pathway-masked sparse neural network vs four baseline classifiers, stratified 5-fold cross-validation repeated ten times on TCGA glioblastoma gene expression. N=475 · Patient samples after excluding censored cases: 99 long-term survivors (past 24 months) and 376 short-term survivors; 4359 genes in 574 Reactome pathways as inputs. Glioblastoma patients' tumour gene-expression profiles from The Cancer Genome Atlas Classification of long-term vs short-term survival, measured by AUC and F1-score Can a transformer predict which cancer drug pairs work together? Supports Computational / modellingTransformer-based synergy predictor benchmarked against DeepSynergy using nested cross-validation with leave-combination-out, leave-cell-out and leave-drug-out splits, plus feature ablations. N=18553 · 18553 drug-pair-by-cell-line synergy measurements (36 drugs, 35 cancer cell lines) from the Merck screening dataset. Merck high-throughput anti-cancer drug combination screen (36 drugs, 35 cell lines). Error and correlation between predicted and measured synergy scores (MSE, Pearson, Spearman) and classification PR-AUC/ROC-AUC. Same accuracy, very different internal representations.
What a network's filters represent depends on design choices, not only on the data: with small first-layer pooling only up to 0.127 of filters matched a true motif, versus up to 0.980 with large pooling, at similar accuracy.
An explanation is a lead to check, not a finding.
Explanations are post hoc hypotheses unless tested: PASNet's authors caution active nodes don't mean activated pathways, TranSynergy's SHAP explanations were not validated in the lab, and the EHR model's attribution example was a single case.
- Can wiring a neural net by biology help small-data prognosis?
- Can a transformer predict which cancer drug pairs work together?
- Can deep learning use a whole hospital record to predict outcomes?
Study Role Design N Population Outcome Can wiring a neural net by biology help small-data prognosis? Supports Computational / modellingNew pathway-masked sparse neural network vs four baseline classifiers, stratified 5-fold cross-validation repeated ten times on TCGA glioblastoma gene expression. N=475 · Patient samples after excluding censored cases: 99 long-term survivors (past 24 months) and 376 short-term survivors; 4359 genes in 574 Reactome pathways as inputs. Glioblastoma patients' tumour gene-expression profiles from The Cancer Genome Atlas Classification of long-term vs short-term survival, measured by AUC and F1-score Can a transformer predict which cancer drug pairs work together? Supports Computational / modellingTransformer-based synergy predictor benchmarked against DeepSynergy using nested cross-validation with leave-combination-out, leave-cell-out and leave-drug-out splits, plus feature ablations. N=18553 · 18553 drug-pair-by-cell-line synergy measurements (36 drugs, 35 cancer cell lines) from the Merck screening dataset. Merck high-throughput anti-cancer drug combination screen (36 drugs, 35 cell lines). Error and correlation between predicted and measured synergy scores (MSE, Pearson, Spearman) and classification PR-AUC/ROC-AUC. Can deep learning use a whole hospital record to predict outcomes? Supports Computational / modellingRetrospective development and test of an ensemble of LSTM, attention and boosted-stump networks on raw FHIR-formatted records, compared with logistic-regression baselines, with an 80/10/10 patient split. N=216221 · 216,221 hospitalizations from 114,003 unique adult patients across two US academic hospitals. Adult inpatients (stays of at least 24 hours) at two US academic medical centres AUROC for in-hospital mortality, 30-day unplanned readmission, long length of stay, and discharge diagnosis codes
Debates
Tensions and limits
Some items are genuine disagreements on the same question. Others mark different assays, populations, or outcomes.
'Interpretable' means different things: point scores are interpretable by simplicity (not measured with clinicians), knowledge-guided networks by structure, and a robot navigation paper uses 'explainable' to mean reward terms have physical meaning, not that the policy was analysed.
'Interpretable' means different things: point scores are interpretable by simplicity (not measured with clinicians), knowledge-guided networks by structure, and a robot navigation paper uses 'explainable' to mean reward terms have physical meaning, not that the policy was analysed.
- Can machine learning build simple, readable hospital risk scores?
- Can wiring a neural net by biology help small-data prognosis?
- Can treating obstacles like electric charges teach a robot to steer?
Study Role Design N Population Outcome Can machine learning build simple, readable hospital risk scores? Supports Computational / modellingRetrospective model development on MIMIC-III ICU admissions with a 70/10/20 train/validation/test split, comparing AutoScore point scores with five baseline models. N=44918 · 44,918 ICU admission episodes from one hospital (BIDMC) in MIMIC-III; 3958 died in hospital. Adult ICU admissions aged 18-90 at Beth Israel Deaconess Medical Center, 2001-2012 Inpatient mortality; AUC, sensitivity, specificity and calibration on the held-out test set Can wiring a neural net by biology help small-data prognosis? Supports Computational / modellingNew pathway-masked sparse neural network vs four baseline classifiers, stratified 5-fold cross-validation repeated ten times on TCGA glioblastoma gene expression. N=475 · Patient samples after excluding censored cases: 99 long-term survivors (past 24 months) and 376 short-term survivors; 4359 genes in 574 Reactome pathways as inputs. Glioblastoma patients' tumour gene-expression profiles from The Cancer Genome Atlas Classification of long-term vs short-term survival, measured by AUC and F1-score Can treating obstacles like electric charges teach a robot to steer? Supports Computational / modellingReward-shaping ablation: five TD3 agents differing only in reward terms, trained in Gazebo and tested in simulated scenes (static, dynamic, cluttered) and two real-world scenes. No single N: five agent variants evaluated over multiple rollouts in several simulated test scenes and two real test scenes; exact rollout counts are in tables not included in the extracted text. Simulated and real TurtleBot3 Burger mobile robots with 2-D LiDAR Success rate, collision ratio per 1000 steps, and average final distance to goal
PaperFren reads this as a limit on how far one study travels — different assays, populations, or outcomes — not a forced fight between papers.
Timeline
How understanding moved
Study years are when the paper was published. Evidence edits are dated changes to this page's claims. Explanations are when PaperFren added a Discovery — not a claim that the science happened that day.
2026
- Dropping race from a model can make it less fair, and fixing one gap can widen another
Concept page published
Interpretable machine learning
Change log
What changed
Dated edits to this page's evidence: studies added or removed from a claim, claims added or withdrawn, and new explanations tagged here. Rewordings are not listed.
- Concept page published
Papers
6 studies in this library bear on Interpretable machine learning, ordered by citations.
- Can deep learning use a whole hospital record to predict outcomes?
Neural networks reading a patient's entire raw hospital record predicted death, readmission and long stays more accurately than standard clinical scores.
- Can a transformer predict which cancer drug pairs work together?
Describing drugs by the proteins they affect and cells by which genes they depend on let a transformer predict drug-pair synergy better than an earlier chemistry-based network, while keeping predictions explainable gene by gene.
- Can machine learning build simple, readable hospital risk scores?
An automated method produced a nine-item point score for predicting hospital death that was almost as accurate as a black-box random forest while being simple enough to add up by hand.
- Can wiring a neural net by biology help small-data prognosis?
A neural network whose first connections follow known gene pathways, and which prunes itself to stay sparse, predicted brain-tumour survival slightly better than standard classifiers while being easier to interpret.
- When do a CNN's first filters learn whole DNA motifs?
CNNs only learn whole, readable DNA motifs in their first layer when the architecture stops deeper layers from assembling motifs out of pieces.
- Can treating obstacles like electric charges teach a robot to steer?
Adding reward terms modelled on electric attraction to the goal and repulsion from obstacles helped a learning robot reach goals more often with fewer collisions, in simulation and on a real robot.
Compare studies
Select 2–10 studies. Design and N are labels, not a ranking.
Nothing selected yet.
Questions
What is still open
'Interpretable' means different things: point scores are interpretable by simplicity (not measured with clinicians), knowledge-guided networks by structure, and a robot navigation paper uses 'explainable' to mean reward terms have physical meaning, not that the policy was analysed.
Ask PaperFren about Interpretable machine learning
Study this conceptflashcards and short-answer questions
How does the genomic motif study challenge the idea that interpretation reveals what a network 'knows'?
Networks with similar accuracy learned either scattered partial motifs or whole motifs depending on max-pooling size. So the representation reflects architecture, not just data. Also, database matching gave false reassurance, and visualising a filter doesn't show how much the network relies on it.
Evaluate AutoScore as an interpretable alternative to random forests.
A nine-variable point score reached AUC 0.780, close to logistic regression and a nine-variable random forest, and was well calibrated, while the full random forest was more accurate (0.809) but poorly calibrated. The study is single-hospital and retrospective, and interpretability was argued from simplicity rather than tested with clinicians.