Can a language model learn useful features from protein sequences?
Embeddings from a language model trained only on raw protein sequences beat other single-sequence encodings and came close to, but did not beat, the best methods that use evolutionary information, while running far faster.
Source
Modeling aspects of the language of life through transfer-learning protein sequences
Study at a glance
- Design
- Computational / modelling — Pre-train a bidirectional LSTM language model (ELMo) on UniRef50, then train small supervised networks on its embeddings for four protein prediction tasks and compare to baselines.
- N
- No single N; pre-training used UniRef50 (33 million sequences) and each downstream task used its own public benchmark set (NetSurfP-2.0 and DeepLoc data).
- Population
- Protein sequences from UniRef50 and public benchmark sets
- Outcome
- Per-residue secondary structure (Q3/Q8) and disorder (MCC); per-protein subcellular localization (Q10) and membrane vs soluble (Q2)
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors took ELMo, a bidirectional LSTM language model from natural language processing, and trained it to predict the next amino acid across UniRef50 protein sequences, calling the result SeqVec. They then fed its embeddings into deliberately simple networks for four tasks: secondary structure, intrinsic disorder, subcellular localization and membrane-bound versus water-soluble classification. These were compared with one-hot and BLOSUM encodings, the Word2vec-style ProtVec, their own profile-based models and published state-of-the-art tools.
What they found
Among methods using only a single sequence, SeqVec beat the alternatives, for example reaching 68% ten-class localization accuracy versus 42% for ProtVec. It did not reach the top tools that use evolutionary alignments: DeepLoc stayed best at 78% on localization and NetSurfP-2.0 stayed best on secondary structure. The gain relative to competitors was larger for per-protein than per-residue tasks, and prediction was roughly 100-times faster than alignment-based pipelines. t-SNE plots showed the unsupervised embeddings already grouped amino acids and proteins by biochemical and structural properties.
The limits
What it doesn't show
It does not show that language-model embeddings can replace evolutionary information: on every task the best alignment-based method still won. The downstream networks were intentionally simple and far smaller than competitors, so the comparison mixes representation quality with model size. The language model was not evaluated on a held-out split during pre-training, and the reasons for the per-protein advantage are, as the authors say, speculation. t-SNE clusters are a qualitative picture, not a test.
Key terms
- Transfer learning
- Reusing what a model learned on one large task (here, predicting amino acids) to help a different task with less labelled data.
- Embedding
- A learned vector of numbers that represents an item, such as an amino acid in context, for use by other models.
- Contextualized embedding
- An embedding that changes depending on the surrounding sequence, unlike Word2vec-style vectors that are fixed per token.
- Evolutionary information
- Profiles built by aligning a protein to its relatives; powerful for prediction but slow to compute.
- Q3 accuracy
- The percentage of residues whose three-state secondary structure (helix, strand, other) is predicted correctly.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What architecture did SeqVec adapt from natural language processing?
Common questions
Why train a language model on proteins at all?
Alignment searches get slower as databases grow and fail for proteins with few relatives; a pre-trained model gives a fast summary from the sequence alone.
Did SeqVec set a new state of the art?
No. It beat other single-sequence encodings but remained below the best tools that use evolutionary profiles on all four tasks.
Where did the embeddings help most?
On per-protein tasks such as localization, which have far smaller training sets, where it came second-best alongside another method.
More on Language models