Representation learning
Can a small protein language model match much bigger ones?
Open access · cc by · source: Europe PMC
A compact protein model pretrained to both fill in masked amino acids and predict protein functions performs close to far larger models across many protein tasks.
Study at a glance
- Design
- Computational / modelling — Self-supervised pretraining on UniRef90 sequences plus GO annotations, then fine-tuning and testing on nine protein benchmarks with pretraining and ablation comparisons
- N
- Pretraining used roughly 106M proteins; evaluation used nine benchmarks of different sizes, so there is no single analytic N
- Population
- Protein sequences from UniProtKB/UniRef90 and benchmark datasets (four from TAPE, five new)
- Outcome
- Benchmark performance (e.g. secondary structure, remote homology, fluorescence, stability) versus TAPE Transformer, LSTM and ProtT5 models
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
With about 16 million parameters, ProteinBERT performed comparably to, and sometimes better than, larger models such as the 38-million-parameter TAPE Transformer, though the 3-billion-parameter ProtT5 did better on some tasks. Pretraining strongly helped many tasks, and harder tasks like secondary structure and remote homology kept improving with more pretraining. The function-annotation task helped some benchmarks, and performance dropped only modestly on proteins much longer than those used in training.
Methodology
The authors designed ProteinBERT, a BERT-like model that keeps separate per-residue (local) and whole-protein (global) representations linked by a linear-cost global attention mechanism. They pretrained it for four weeks on one GPU on about 106 million non-redundant proteins, with two tasks at once: recovering corrupted amino acids and predicting corrupted Gene Ontology function labels. They then fine-tuned the same pretrained model on nine benchmarks and compared it to published TAPE models and ProtT5, also testing no pretraining, different pretraining durations, and removing the function-label task.
Limitations
Direct comparisons were possible only on four TAPE benchmarks, and other leading models such as ESM lacked comparable published results, so the ranking is incomplete. The paper reports results relative to published numbers rather than re-running all baselines under identical conditions. The attention-map analysis looks at just two example proteins and the authors note attention values can change between training runs, so it is illustrative rather than a systematic interpretability test.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Language-model pretraining captures structural information in protein sequences.
Pretrained protein language models transferred to structure-related tasks: ProteinBERT (16M parameters) performed comparably to larger models and kept improving on secondary structure and remote homology with more pretraining, and ProstT5's sequence-to-3Di translation approached experimental-structure search sensitivity (0.45 vs 0.49 ROC-AUC).
Evidence for the claim as stated.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can a transformer predict which drugs bind which proteins?
- Can one language model read protein sequence and structure?
- Can random negative pairs teach a better genome-sorting model?
- Can a self-taught CT model find cancer markers with little data?
- Can a network trained on everyday photos search pathology slides?