Topic
Language models research, explained
7 open-access language models studies, each with a flashcard deck and a quiz.
- Does pre-training BERT on biomedical papers help it read biology?
Letting a general language model keep learning from millions of biomedical abstracts and articles made it clearly better at biomedical text tasks than the same model trained only on Wikipedia and books.
- Do bigger language models understand clinical notes better?
Scaling a clinical language model from hundreds of millions to 8.9 billion parameters, trained on tens of billions of words of real patient notes, improved performance on a range of medical language tasks, most for the hardest ones.
- Can a language model learn useful features from protein sequences?
Embeddings from a language model trained only on raw protein sequences beat other single-sequence encodings and came close to, but did not beat, the best methods that use evolutionary information, while running far faster.
- Can fine-tuned language models draft replies to patient messages?
A language model fine-tuned only on doctors' real portal replies wrote drafts no better than the doctors, but adding richer, empathetic example replies to its training made it about as good as ChatGPT.
- Can a fine-tuned open LLM assign hospital billing codes from notes?
A LLaMA model fine-tuned on discharge summaries predicted hospital diagnosis-related groups more accurately than earlier leading text classifiers, and got better with larger models and longer inputs.
- Can a general chatbot-style model be taught to spot biomedical terms?
A general-purpose open language model, fine-tuned on instructions made from existing labelled data, found diseases, chemicals and genes in biomedical text about as well as a specialist biomedical model and far better than prompted GPT models.
- Can an LLM pipeline speed up systematic reviews without losing quality?
Breaking systematic-review work into structured steps that a large language model performs and experts can check found far more relevant studies and extracted data more accurately than simply prompting GPT-4.