Can a pre-trained medical AI cope with a new disease using few labels?
A model pre-trained on large amounts of unlabelled medical images and text could perform COVID-19 reporting, diagnosis and prognosis with only about 1% of the usual labelled data.
Source
A medical multimodal large language model for future pandemics
Study at a glance
- Design
- Computational / modelling — Self-supervised multimodal pre-training (contrastive image learning, language-model text objectives, knowledge-guided image-text alignment) followed by fine-tuning with 1%-100% labels, compared against published baselines on five COVID-19 and several thorax-disease datasets.
- N
- No single N: five COVID-19 datasets of very different sizes (from 96 patients to 29,986 images) plus public thorax-disease benchmarks.
- Population
- Chest X-ray and CT images with radiology reports in English, Chinese and Spanish from multiple countries.
- Outcome
- Report-generation metrics (BLEU, ROUGE-L, CIDEr), AUC for COVID-19 diagnosis and survival prognosis.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors built Med-MLLM, which first learns from unlabelled chest images (via contrastive learning, including pairs of images from the same patient), from unlabelled medical text (via a language model trained to fill masked words and reconstruct sentences), and from unpaired images and reports linked through a medical knowledge base. They then fine-tuned it for three COVID-19 tasks - writing radiology reports, classifying COVID-19 vs other pneumonia, and predicting survival - using 1% up to 100% of the labels. They tested it retrospectively, prospectively (pre-trained on Delta, evaluated on Omicron), across regions, against GPT-family models, and with ablations removing each component.
What they found
With 1% of labels, Med-MLLM was competitive with earlier methods trained on all labels, and with 10% it beat previous state-of-the-art models trained on 100%. At 1% labels it exceeded the best previous method by up to 21.8 points on reporting and 6.7 on diagnosis. A model trained on Spanish patients reached 90.1% AUC on Chinese patients, and removing image-only pre-training cut diagnosis AUC from 78.2% to 69.4%. It outperformed GPT-2, GPT-3 and ChatGPT on text-only diagnosis and was competitive with GPT-4.
The limits
What it doesn't show
Most result tables are not in the extracted text, so many exact scores cannot be checked here. The 'prospective' study is a simulation on already-collected Omicron records, not a live clinical deployment, and no clinicians used the tool to see whether patient care improved. Some datasets are small (one CT set has 96 patients), test sets were artificially balanced 1:1, which differs from real prevalence, and hallucination checks relied on an automatic labeller rather than physician review.
Key terms
- Self-supervised pre-training
- Learning general representations from unlabelled data by solving automatically generated tasks, before fine-tuning on a labelled task.
- Contrastive learning
- Training an encoder to place different views of the same item close together and different items far apart.
- Patient-level contrastive learning
- Treating two different X-ray views from the same patient as a positive pair so the model attends to pathology rather than image appearance.
- AUC
- Area under the ROC curve; the probability a classifier ranks a random positive case above a random negative case.
- Ablation study
- Removing one component at a time to measure how much each contributes to performance.
- Label efficiency
- How well a model performs when only a small fraction of training examples have labels.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What is Med-MLLM's main advantage?
Common questions
Why does pre-training help with a new disease?
Early in a pandemic labelled cases are scarce, but unlabelled images and reports of related diseases are plentiful; pre-training on them gives a warm start that needs few new labels.
What makes the Omicron test 'prospective'?
The model was pre-trained on Delta-era data and then adapted to Omicron data it had not seen, mimicking the arrival of a new variant.
Did bigger language models help?
Yes; a larger 8.9-billion-parameter language model improved text-only diagnosis and prognosis AUC by a few points over the base version.
More on Representation learning