Representation learning
Can a pre-trained medical AI cope with a new disease using few labels?
Open access · cc by · source: Europe PMC
A model pre-trained on large amounts of unlabelled medical images and text could perform COVID-19 reporting, diagnosis and prognosis with only about 1% of the usual labelled data.
Study at a glance
- Design
- Computational / modelling — Self-supervised multimodal pre-training (contrastive image learning, language-model text objectives, knowledge-guided image-text alignment) followed by fine-tuning with 1%-100% labels, compared against published baselines on five COVID-19 and several thorax-disease datasets.
- N
- No single N: five COVID-19 datasets of very different sizes (from 96 patients to 29,986 images) plus public thorax-disease benchmarks.
- Population
- Chest X-ray and CT images with radiology reports in English, Chinese and Spanish from multiple countries.
- Outcome
- Report-generation metrics (BLEU, ROUGE-L, CIDEr), AUC for COVID-19 diagnosis and survival prognosis.
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
With 1% of labels, Med-MLLM was competitive with earlier methods trained on all labels, and with 10% it beat previous state-of-the-art models trained on 100%. At 1% labels it exceeded the best previous method by up to 21.8 points on reporting and 6.7 on diagnosis. A model trained on Spanish patients reached 90.1% AUC on Chinese patients, and removing image-only pre-training cut diagnosis AUC from 78.2% to 69.4%. It outperformed GPT-2, GPT-3 and ChatGPT on text-only diagnosis and was competitive with GPT-4.
Methodology
The authors built Med-MLLM, which first learns from unlabelled chest images (via contrastive learning, including pairs of images from the same patient), from unlabelled medical text (via a language model trained to fill masked words and reconstruct sentences), and from unpaired images and reports linked through a medical knowledge base. They then fine-tuned it for three COVID-19 tasks - writing radiology reports, classifying COVID-19 vs other pneumonia, and predicting survival - using 1% up to 100% of the labels. They tested it retrospectively, prospectively (pre-trained on Delta, evaluated on Omicron), across regions, against GPT-family models, and with ablations removing each component.
Limitations
Most result tables are not in the extracted text, so many exact scores cannot be checked here. The 'prospective' study is a simulation on already-collected Omicron records, not a live clinical deployment, and no clinicians used the tool to see whether patient care improved. Some datasets are small (one CT set has 96 patients), test sets were artificially balanced 1:1, which differs from real prevalence, and hallucination checks relied on an automatic labeller rather than physician review.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Pretrained representations shine in the low-label regime.
Pretraining helped most when labels were scarce: frozen CT foundation-model features beat all baselines at every reduced training size, and a medical multimodal model with 10% of labels beat prior models trained on all labels.
Evidence for the claim as stated.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can a small protein language model match much bigger ones?
- Can a transformer predict which drugs bind which proteins?
- Can one language model read protein sequence and structure?
- Can random negative pairs teach a better genome-sorting model?
- Can a self-taught CT model find cancer markers with little data?