Can one language model read protein sequence and structure?
Teaching a protein language model to translate between amino-acid sequences and a letter-code for 3D shape let it find distant structural relatives almost as well as real structures, thousands of times faster than predicting structures first.
Source
Bilingual language model for protein sequence and structure
Study at a glance
- Design
- Computational / modelling — ProtT5 (3B parameters) extended with 20 lower-case 3Di structure tokens, further pre-trained on both modalities and then trained to translate AA->3Di and 3Di->AA; embeddings and translations compared with ProtT5, ESM, Ankh, Foldseek, MMseqs2 and ProteinMPNN on public benchmarks.
- N
- No single N: training set of 17M AlphaFold2-predicted proteins (34M bidirectional samples); 474 held-out test proteins for inverse folding; separate benchmark sets (SCOPe40, CATH, CASP12/14, NEW364, DeepLoc) for other tasks.
- Population
- Protein sequences and AlphaFold2-predicted structures from the clustered AlphaFold Protein Structure Database, plus standard protein benchmarks.
- Outcome
- Remote-homology sensitivity (ROC-AUC), CATH classification accuracy, secondary-structure Q3 and other per-residue/per-protein task performance, inverse-folding structural similarity (lDDT, TM-score, RMSD) and runtime.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors encoded protein 3D structures as strings of 20 structural letters (Foldseek's 3Di alphabet), so a text model could read them. Starting from the existing ProtT5 protein language model, they continued training on 17 million AlphaFold2-predicted proteins in both amino-acid and 3Di form, then trained it to translate in both directions, calling the result ProstT5. They tested predicted 3Di strings for remote-homology search, its embeddings on CATH classification and standard prediction tasks, and its reverse translation for designing new sequences that fold into a given shape.
What they found
3Di strings predicted from sequence reached a superfamily-level ROC-AUC of 0.45 in the Foldseek benchmark, close to experimental structures (0.49) and far above sequence-only search (0.06), and a faster encoder-plus-CNN version reached 0.47. Predicting 3Di for a 1787-protein proteome took under a minute on a GPU versus 48 hours for structure prediction. ProstT5 embeddings improved CATH classification over ProtT5, ESM-1b and Ankh, but general tasks such as subcellular location got worse. Inverse-folded sequences shared only about 21% identity with natives yet were predicted to fold similarly (lDDT 72), below the specialised ProteinMPNN (77).
The limits
What it doesn't show
Training and much of the testing rely on AlphaFold2 predictions and ESMFold re-predictions rather than experimentally solved structures, so inverse-folding success is judged by one predictor agreeing with another, not by lab-made proteins. Some benchmark sets (CASP12, NEW364) overlap with AlphaFold2 training data, allowing indirect leakage. Fine-tuning caused partial forgetting, so ProstT5 is worse than its base model on tasks like subcellular location. Training was stopped before full convergence for cost reasons, and inverse folding still trails ProteinMPNN.
Key terms
- Protein language model (pLM)
- A transformer trained on huge numbers of protein sequences, treating amino acids like words, whose internal representations can be reused for prediction tasks.
- 3Di alphabet
- Foldseek's code that turns each residue's 3D structural neighbourhood into one of 20 letters, so structures become 1D strings.
- Remote homology detection
- Finding evolutionarily related proteins whose sequences have diverged so much that simple sequence comparison fails.
- Inverse folding
- Designing amino-acid sequences that should fold into a given 3D structure.
- Catastrophic forgetting
- When further training on new data makes a model lose abilities it learned earlier.
- Embedding
- The vector of hidden-layer values a model produces for an input, used as features for other models.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What does ProstT5 translate between?
Common questions
Why convert structures into letters at all?
Language models and fast sequence-alignment tools work on 1D strings; turning 3D structure into a 20-letter string lets the same machinery read, compare and generate structures.
If ProstT5 is better at structure, why isn't it better at everything?
Fine-tuning on structure translation partly overwrote what ProtT5 had learned about sequences, so tasks such as subcellular location prediction dropped; concatenating ProtT5 and ProstT5 embeddings compensated.
Why is the speed-up so large?
ProstT5 predicts the 3Di letters directly from sequence in one pass, skipping full 3D structure prediction, which is the slow step in the conventional pipeline.
More on Representation learning