Representation learning
Can one language model read protein sequence and structure?
Open access · cc by · source: Europe PMC
Teaching a protein language model to translate between amino-acid sequences and a letter-code for 3D shape let it find distant structural relatives almost as well as real structures, thousands of times faster than predicting structures first.
Study at a glance
- Design
- Computational / modelling — ProtT5 (3B parameters) extended with 20 lower-case 3Di structure tokens, further pre-trained on both modalities and then trained to translate AA->3Di and 3Di->AA; embeddings and translations compared with ProtT5, ESM, Ankh, Foldseek, MMseqs2 and ProteinMPNN on public benchmarks.
- N
- No single N: training set of 17M AlphaFold2-predicted proteins (34M bidirectional samples); 474 held-out test proteins for inverse folding; separate benchmark sets (SCOPe40, CATH, CASP12/14, NEW364, DeepLoc) for other tasks.
- Population
- Protein sequences and AlphaFold2-predicted structures from the clustered AlphaFold Protein Structure Database, plus standard protein benchmarks.
- Outcome
- Remote-homology sensitivity (ROC-AUC), CATH classification accuracy, secondary-structure Q3 and other per-residue/per-protein task performance, inverse-folding structural similarity (lDDT, TM-score, RMSD) and runtime.
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
3Di strings predicted from sequence reached a superfamily-level ROC-AUC of 0.45 in the Foldseek benchmark, close to experimental structures (0.49) and far above sequence-only search (0.06), and a faster encoder-plus-CNN version reached 0.47. Predicting 3Di for a 1787-protein proteome took under a minute on a GPU versus 48 hours for structure prediction. ProstT5 embeddings improved CATH classification over ProtT5, ESM-1b and Ankh, but general tasks such as subcellular location got worse. Inverse-folded sequences shared only about 21% identity with natives yet were predicted to fold similarly (lDDT 72), below the specialised ProteinMPNN (77).
Methodology
The authors encoded protein 3D structures as strings of 20 structural letters (Foldseek's 3Di alphabet), so a text model could read them. Starting from the existing ProtT5 protein language model, they continued training on 17 million AlphaFold2-predicted proteins in both amino-acid and 3Di form, then trained it to translate in both directions, calling the result ProstT5. They tested predicted 3Di strings for remote-homology search, its embeddings on CATH classification and standard prediction tasks, and its reverse translation for designing new sequences that fold into a given shape.
Limitations
Training and much of the testing rely on AlphaFold2 predictions and ESMFold re-predictions rather than experimentally solved structures, so inverse-folding success is judged by one predictor agreeing with another, not by lab-made proteins. Some benchmark sets (CASP12, NEW364) overlap with AlphaFold2 training data, allowing indirect leakage. Fine-tuning caused partial forgetting, so ProstT5 is worse than its base model on tasks like subcellular location. Training was stopped before full convergence for cost reasons, and inverse folding still trails ProteinMPNN.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Language-model pretraining captures structural information in protein sequences.
Pretrained protein language models transferred to structure-related tasks: ProteinBERT (16M parameters) performed comparably to larger models and kept improving on secondary structure and remote homology with more pretraining, and ProstT5's sequence-to-3Di translation approached experimental-structure search sensitivity (0.45 vs 0.49 ROC-AUC).
Evidence for the claim as stated.
Specialising a representation can cost generality: ProstT5 improved CATH structural classification but got worse on tasks like subcellular location than its base model.
Evidence for the claim as stated.
Open questions
Tensions this paper is part of
From concept pages' “where studies disagree.” Disagreement means the same question; scope means different assays, populations, or outcomes.
Specialising a representation can cost generality: ProstT5 improved CATH structural classification but got worse on tasks like subcellular location than its base model.
Related papers in this topic
Same topic cluster — not a recommendation engine.
- Can a small protein language model match much bigger ones?
- Can a transformer predict which drugs bind which proteins?
- Can random negative pairs teach a better genome-sorting model?
- Can a self-taught CT model find cancer markers with little data?
- Can a network trained on everyday photos search pathology slides?