Uncertainty and calibration
Which uncertainty methods help ML guide protein engineering?
Open access · cc by · source: Europe PMC
No uncertainty-estimation method was best across protein tasks, and using uncertainty to choose which proteins to test next never beat simply picking the highest predicted ones.
Study at a glance
- Design
- Computational / modelling — Seven UQ methods (Bayesian ridge regression, Gaussian process, and five CNN variants: dropout, ensemble, evidential, mean-variance estimation, last-layer SVI) trained on 8 FLIP train-test splits with one-hot or ESM-1b inputs, then used in simulated active learning and Bayesian optimisation.
- N
- No single N: three protein landscapes of 8,733 (GB1), 284,009 (AAV) and 27,951 (Meltome) sequences, split into 8 tasks; CNN metrics averaged over 5 random seeds.
- Population
- Protein sequence-function datasets from the FLIP benchmark (GB1 binding, AAV capsid stability, Meltome thermostability).
- Outcome
- Prediction accuracy (RMSE, Spearman rho), uncertainty quality (miscalibration area, 95% coverage, interval width, rank correlation of uncertainty with error), and active-learning / Bayesian-optimisation performance.
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
Accuracy fell as domain shift grew, but calibration did not follow a clear pattern, and no method won across landscapes and metrics; ensembles were often among the most accurate CNNs yet among the worst calibrated, while GPs and Bayesian ridge regression were often better calibrated. Language-model embeddings beat one-hot codes in only 21 of 51 cases for prediction ranking, helping mostly on high-shift splits. Uncertainty-based sampling often beat random sampling in later active-learning rounds, but in Bayesian optimisation no uncertainty-aware acquisition beat a greedy strategy, and standard UQ metrics did not track active-learning gains well.
Methodology
The authors compared seven ways of attaching uncertainty to machine-learning predictions of protein properties, from simple Bayesian linear regression and Gaussian processes to five kinds of convolutional neural network (dropout, ensembles, evidential, mean-variance and variational last layer). They trained each on 8 tasks from the FLIP benchmark chosen to range from random splits to strong domain shift, using either one-hot sequence codes or embeddings from the ESM-1b protein language model. They scored accuracy, calibration, coverage and interval width, then simulated active learning and Bayesian optimisation to see whether better uncertainty led to better data collection.
Limitations
All experiments are retrospective simulations on three public landscapes, so they do not show what happens in real wet-lab campaigns or on other protein properties. Some models (several Gaussian processes and one Bayesian ridge model) could not be trained for memory reasons, leaving gaps in the comparison. Uncertainties were used as output without post-hoc recalibration, and only ESM-1b with mean pooling was tried, so newer language models or calibration fixes might change the ranking. The authors suggest the landscapes may simply be easy enough for greedy search, which limits how far the Bayesian-optimisation null result generalises.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Accuracy and calibration can come apart.
In a benchmark on three protein sequence-function landscapes, no uncertainty method was best across datasets and metrics; CNN ensembles were often among the most accurate yet among the worst calibrated, while Gaussian processes and Bayesian ridge regression were often better calibrated.
Evidence for the claim as stated.
Good uncertainty estimates do not automatically improve downstream decisions.
In the same retrospective benchmark, uncertainty-based sampling often beat random sampling in later active-learning rounds, but in Bayesian optimisation no uncertainty-aware acquisition beat a greedy strategy.
Evidence for the claim as stated.
Related papers in this topic
Same topic cluster — not a recommendation engine.