Which uncertainty methods help ML guide protein engineering?
No uncertainty-estimation method was best across protein tasks, and using uncertainty to choose which proteins to test next never beat simply picking the highest predicted ones.
Source
Benchmarking uncertainty quantification for protein engineering
Study at a glance
- Design
- Computational / modelling — Seven UQ methods (Bayesian ridge regression, Gaussian process, and five CNN variants: dropout, ensemble, evidential, mean-variance estimation, last-layer SVI) trained on 8 FLIP train-test splits with one-hot or ESM-1b inputs, then used in simulated active learning and Bayesian optimisation.
- N
- No single N: three protein landscapes of 8,733 (GB1), 284,009 (AAV) and 27,951 (Meltome) sequences, split into 8 tasks; CNN metrics averaged over 5 random seeds.
- Population
- Protein sequence-function datasets from the FLIP benchmark (GB1 binding, AAV capsid stability, Meltome thermostability).
- Outcome
- Prediction accuracy (RMSE, Spearman rho), uncertainty quality (miscalibration area, 95% coverage, interval width, rank correlation of uncertainty with error), and active-learning / Bayesian-optimisation performance.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors compared seven ways of attaching uncertainty to machine-learning predictions of protein properties, from simple Bayesian linear regression and Gaussian processes to five kinds of convolutional neural network (dropout, ensembles, evidential, mean-variance and variational last layer). They trained each on 8 tasks from the FLIP benchmark chosen to range from random splits to strong domain shift, using either one-hot sequence codes or embeddings from the ESM-1b protein language model. They scored accuracy, calibration, coverage and interval width, then simulated active learning and Bayesian optimisation to see whether better uncertainty led to better data collection.
What they found
Accuracy fell as domain shift grew, but calibration did not follow a clear pattern, and no method won across landscapes and metrics; ensembles were often among the most accurate CNNs yet among the worst calibrated, while GPs and Bayesian ridge regression were often better calibrated. Language-model embeddings beat one-hot codes in only 21 of 51 cases for prediction ranking, helping mostly on high-shift splits. Uncertainty-based sampling often beat random sampling in later active-learning rounds, but in Bayesian optimisation no uncertainty-aware acquisition beat a greedy strategy, and standard UQ metrics did not track active-learning gains well.
The limits
What it doesn't show
All experiments are retrospective simulations on three public landscapes, so they do not show what happens in real wet-lab campaigns or on other protein properties. Some models (several Gaussian processes and one Bayesian ridge model) could not be trained for memory reasons, leaving gaps in the comparison. Uncertainties were used as output without post-hoc recalibration, and only ESM-1b with mean pooling was tried, so newer language models or calibration fixes might change the ranking. The authors suggest the landscapes may simply be easy enough for greedy search, which limits how far the Bayesian-optimisation null result generalises.
Key terms
- Uncertainty quantification (UQ)
- Estimating how unsure a model is about each prediction, not just the prediction itself.
- Calibration
- Whether stated uncertainty matches reality, e.g. true values fall inside 95% intervals about 95% of the time.
- Domain shift
- A mismatch between the data a model was trained on and the data it is tested on, such as training on single mutants and testing on many-mutation proteins.
- Active learning
- Choosing which new data points to label next, often the most uncertain ones, to improve a model with fewer measurements.
- Bayesian optimisation
- Searching for the best-scoring item by using a model's predictions plus uncertainty to balance exploring and exploiting.
- Greedy acquisition
- Always picking the candidate with the highest predicted value, ignoring uncertainty.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the overall conclusion about which UQ method is best?
Common questions
If a model is well calibrated, will it be better for active learning?
Not necessarily. The authors found UQ evaluation metrics were poorly correlated with the accuracy gained per active-learning round, so they recommend testing methods in retrospective active-learning or optimisation studies directly.
Why can't you just judge uncertainty by coverage?
A model that gives every point a huge uncertainty would cover almost all true values but be useless; good uncertainty needs high coverage and narrow intervals together.
Did protein language-model embeddings always help?
No. They improved prediction ranking in 21 of 51 comparisons, mostly on splits with strong domain shift, while one-hot encodings were slightly better on some easier splits.
More on Uncertainty and calibration