Concept · artificial-intelligence
Representation learning and pretraining
8 studies1 discoveryEvidence last moved Sep 27, 2026
Representation learning trains a model to turn raw data into useful features (embeddings), often by pretraining on large unlabelled data before a small labelled task. The evidence here is from computational studies of protein language models, a CT imaging foundation model, a medical multimodal model, contrastive metagenome binning, histopathology image search, drug-target transformers, and a head-to-head test against simple models on transcriptomics.
Pretrained embeddings are now the default in many fields. These studies show where they help most (little labelled data, related tasks) and where plain baselines still win.
Studies
8
Findings
5
7 supporting · 0 challenging · 0 qualifying citations
Open tensions
2
Latest change
Concept page published
Representation learning and pretraining
Currently
What we know
- Pretrained representations shine in the low-label regime.
- Language-model pretraining captures structural information in protein sequences.
- A learned embedding can be the key ingredient of a pipeline.
- Learned embeddings beat hand-crafted descriptors for similarity search.
- Learned representations are not always better than raw features with a simple model.
Largest unresolved question
Pretrained embeddings helped for CT imaging, protein and metagenome tasks but hurt for bulk transcriptomic phenotype prediction with a few hundred labelled samples.
Common misconceptions
A deep learned representation always beats simple baselines.
On bulk RNA-seq phenotype tasks, deep representations lowered performance, and an l2-regularised linear model on centred log-ratio data was recommended as the baseline.
Strong benchmark gains mean the model is ready for new cases.
MolTrans led on standard splits but was only competitive on unseen drugs or targets, the setting closest to real discovery; the foundation-model and multimodal studies were retrospective with no clinical deployment.
Related
Claim ledger
What the evidence shows
Drawn from 8 studies in this library. Mix labels say which citation roles are present; they are not a strength score. Supports means evidence for a finding; Challenges means evidence against a stated position; Qualifies marks scope.
Pretrained representations shine in the low-label regime.
Pretraining helped most when labels were scarce: frozen CT foundation-model features beat all baselines at every reduced training size, and a medical multimodal model with 10% of labels beat prior models trained on all labels.
- Can a self-taught CT model find cancer markers with little data?
- Can a pre-trained medical AI cope with a new disease using few labels?
Study Role Design N Population Outcome Can a self-taught CT model find cancer markers with little data? Supports Computational / modellingSelf-supervised contrastive pretraining of a convolutional encoder on CT lesions, then linear-probe versus fine-tuned evaluation against baselines on lesion-site, nodule-malignancy and lung-cancer survival tasks, with reduced-data ablations. No single N: pretraining used 11,467 lesions from 2,312 patients; downstream test sets were 1,221 lesions (use case 1), 170 nodules (use case 2) and two survival cohorts of 420 and 133 patients (use case 3). Retrospective CT scans from five datasets (DeepLesion, LUNA16, HarvardRT, LUNG1, RADIO) Balanced accuracy, mean average precision and AUC; survival stratification; test-retest stability Can a pre-trained medical AI cope with a new disease using few labels? Supports Computational / modellingSelf-supervised multimodal pre-training (contrastive image learning, language-model text objectives, knowledge-guided image-text alignment) followed by fine-tuning with 1%-100% labels, compared against published baselines on five COVID-19 and several thorax-disease datasets. No single N: five COVID-19 datasets of very different sizes (from 96 patients to 29,986 images) plus public thorax-disease benchmarks. Chest X-ray and CT images with radiology reports in English, Chinese and Spanish from multiple countries. Report-generation metrics (BLEU, ROUGE-L, CIDEr), AUC for COVID-19 diagnosis and survival prognosis. Language-model pretraining captures structural information in protein sequences.
Pretrained protein language models transferred to structure-related tasks: ProteinBERT (16M parameters) performed comparably to larger models and kept improving on secondary structure and remote homology with more pretraining, and ProstT5's sequence-to-3Di translation approached experimental-structure search sensitivity (0.45 vs 0.49 ROC-AUC).
- Can a small protein language model match much bigger ones?
- Can one language model read protein sequence and structure?
Study Role Design N Population Outcome Can a small protein language model match much bigger ones? Supports Computational / modellingSelf-supervised pretraining on UniRef90 sequences plus GO annotations, then fine-tuning and testing on nine protein benchmarks with pretraining and ablation comparisons Pretraining used roughly 106M proteins; evaluation used nine benchmarks of different sizes, so there is no single analytic N Protein sequences from UniProtKB/UniRef90 and benchmark datasets (four from TAPE, five new) Benchmark performance (e.g. secondary structure, remote homology, fluorescence, stability) versus TAPE Transformer, LSTM and ProtT5 models Can one language model read protein sequence and structure? Supports Computational / modellingProtT5 (3B parameters) extended with 20 lower-case 3Di structure tokens, further pre-trained on both modalities and then trained to translate AA->3Di and 3Di->AA; embeddings and translations compared with ProtT5, ESM, Ankh, Foldseek, MMseqs2 and ProteinMPNN on public benchmarks. No single N: training set of 17M AlphaFold2-predicted proteins (34M bidirectional samples); 474 held-out test proteins for inverse folding; separate benchmark sets (SCOPe40, CATH, CASP12/14, NEW364, DeepLoc) for other tasks. Protein sequences and AlphaFold2-predicted structures from the clustered AlphaFold Protein Structure Database, plus standard protein benchmarks. Remote-homology sensitivity (ROC-AUC), CATH classification accuracy, secondary-structure Q3 and other per-residue/per-protein task performance, inverse-folding structural similarity (lDDT, TM-score, RMSD) and runtime. A learned embedding can be the key ingredient of a pipeline.
Self-supervised contrastive learning in SemiBin2 recovered 8.3% to 21.5% more high-quality genomes than its semi-supervised predecessor on four short-read projects, and removing the learned embedding lost roughly a fifth to a third.
Learned embeddings beat hand-crafted descriptors for similarity search.
A general-purpose image embedding (SMILY) returned a matching tissue feature in the top five 62.0% of the time for prostate queries, versus 44.2% for SIFT features and 28.3% for random retrieval.
Learned representations are not always better than raw features with a simple model.
For phenotype prediction from bulk transcriptomics, PCA, autoencoder and VAE representations lowered out-of-sample performance compared with regularised linear models or random forests on log-ratio normalised data.
Debates
Tensions and limits
Some items are genuine disagreements on the same question. Others mark different assays, populations, or outcomes.
Pretrained embeddings helped for CT imaging, protein and metagenome tasks but hurt for bulk transcriptomic phenotype prediction with a few hundred labelled samples.
Evidence for
Evidence against
PaperFren reads this as a limit on how far one study travels — different assays, populations, or outcomes — not a forced fight between papers.
Specialising a representation can cost generality: ProstT5 improved CATH structural classification but got worse on tasks like subcellular location than its base model.
Specialising a representation can cost generality: ProstT5 improved CATH structural classification but got worse on tasks like subcellular location than its base model.
PaperFren reads this as a limit on how far one study travels — different assays, populations, or outcomes — not a forced fight between papers.
Timeline
How understanding moved
Study years are when the paper was published. Evidence edits are dated changes to this page's claims. Explanations are when PaperFren added a Discovery — not a claim that the science happened that day.
2026
- On a few hundred samples, cross-validation flatters models and deep features lose to linear baselines
Concept page published
Representation learning and pretraining
Change log
What changed
Dated edits to this page's evidence: studies added or removed from a claim, claims added or withdrawn, and new explanations tagged here. Rewordings are not listed.
- Concept page published
Papers
8 studies in this library bear on Representation learning and pretraining, ordered by citations.
- Can a small protein language model match much bigger ones?
A compact protein model pretrained to both fill in masked amino acids and predict protein functions performs close to far larger models across many protein tasks.
- Can a transformer predict which drugs bind which proteins?
Breaking drugs and proteins into frequently recurring sub-pieces and modelling how those pieces interact gave better drug-target predictions than earlier deep models, especially when labelled data were scarce.
- Can one language model read protein sequence and structure?
Teaching a protein language model to translate between amino-acid sequences and a letter-code for 3D shape let it find distant structural relatives almost as well as real structures, thousands of times faster than predicting structures first.
- Can random negative pairs teach a better genome-sorting model?
Treating randomly chosen pairs of DNA fragments as 'different' let a contrastive model sort fragments into genomes better and far more cheaply than using reference-database labels.
- Can a self-taught CT model find cancer markers with little data?
A CT image model pretrained without labels produced features that, with just a simple linear classifier, matched or beat fully supervised deep networks and held up best when training data were cut to a tenth.
- Can a network trained on everyday photos search pathology slides?
A neural network that had never seen tissue images could still find visually similar pathology patches far better than chance and better than a classic hand-crafted method.
- Can a pre-trained medical AI cope with a new disease using few labels?
A model pre-trained on large amounts of unlabelled medical images and text could perform COVID-19 reporting, diagnosis and prognosis with only about 1% of the usual labelled data.
- Do deep learned features help predict disease from gene data?
Across dozens of prediction tasks from gene expression data, fancy learned representations did not beat a simple regularised linear model on properly normalised data.
Compare studies
Select 2–10 studies. Design and N are labels, not a ranking.
Nothing selected yet.
Questions
What is still open
Pretrained embeddings helped for CT imaging, protein and metagenome tasks but hurt for bulk transcriptomic phenotype prediction with a few hundred labelled samples.
Specialising a representation can cost generality: ProstT5 improved CATH structural classification but got worse on tasks like subcellular location than its base model.
Ask PaperFren about Representation learning and pretraining
Study this conceptflashcards and short-answer questions
When is pretraining most likely to pay off? Use at least two studies.
When labelled data are scarce and the downstream task resembles what the model learned from. Frozen CT foundation-model features beat baselines at every reduced training size, and a multimodal medical model with 10% of labels beat models trained on all labels. By contrast, deep representations hurt transcriptomic phenotype prediction, where the semi-supervised version helped only on tasks similar to its training data.
Why might a specialised protein language model be worse at some tasks than its base model?
Fine-tuning on a new objective can overwrite earlier knowledge. ProstT5, trained to translate between sequence and 3Di structure tokens, improved structural classification but lost ground on general tasks such as subcellular location, which its authors attribute to partial forgetting.