Language models
Can a language model learn useful features from protein sequences?
Open access · cc by · source: Europe PMC
Embeddings from a language model trained only on raw protein sequences beat other single-sequence encodings and came close to, but did not beat, the best methods that use evolutionary information, while running far faster.
Study at a glance
- Design
- Computational / modelling — Pre-train a bidirectional LSTM language model (ELMo) on UniRef50, then train small supervised networks on its embeddings for four protein prediction tasks and compare to baselines.
- N
- No single N; pre-training used UniRef50 (33 million sequences) and each downstream task used its own public benchmark set (NetSurfP-2.0 and DeepLoc data).
- Population
- Protein sequences from UniRef50 and public benchmark sets
- Outcome
- Per-residue secondary structure (Q3/Q8) and disorder (MCC); per-protein subcellular localization (Q10) and membrane vs soluble (Q2)
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
Among methods using only a single sequence, SeqVec beat the alternatives, for example reaching 68% ten-class localization accuracy versus 42% for ProtVec. It did not reach the top tools that use evolutionary alignments: DeepLoc stayed best at 78% on localization and NetSurfP-2.0 stayed best on secondary structure. The gain relative to competitors was larger for per-protein than per-residue tasks, and prediction was roughly 100-times faster than alignment-based pipelines. t-SNE plots showed the unsupervised embeddings already grouped amino acids and proteins by biochemical and structural properties.
Methodology
The authors took ELMo, a bidirectional LSTM language model from natural language processing, and trained it to predict the next amino acid across UniRef50 protein sequences, calling the result SeqVec. They then fed its embeddings into deliberately simple networks for four tasks: secondary structure, intrinsic disorder, subcellular localization and membrane-bound versus water-soluble classification. These were compared with one-hot and BLOSUM encodings, the Word2vec-style ProtVec, their own profile-based models and published state-of-the-art tools.
Limitations
It does not show that language-model embeddings can replace evolutionary information: on every task the best alignment-based method still won. The downstream networks were intentionally simple and far smaller than competitors, so the comparison mixes representation quality with model size. The language model was not evaluated on a held-out split during pre-training, and the reasons for the per-protein advantage are, as the authors say, speculation. t-SNE clusters are a qualitative picture, not a test.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Language models of proteins learn useful structure, but did not replace evolutionary information.
The same idea transfers beyond words: SeqVec, a language model trained on protein sequences, beat other single-sequence methods (68% vs 42% ten-class localisation) but did not beat alignment-based tools such as DeepLoc (78%).
Evidence for the claim as stated.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Does pre-training BERT on biomedical papers help it read biology?
- Do bigger language models understand clinical notes better?
- Can fine-tuned language models draft replies to patient messages?
- Can a fine-tuned open LLM assign hospital billing codes from notes?
- Can a general chatbot-style model be taught to spot biomedical terms?