Can a network trained on everyday photos search pathology slides?
A neural network that had never seen tissue images could still find visually similar pathology patches far better than chance and better than a classic hand-crafted method.
Source
Similar image search for histopathology: SMILY
Study at a glance
- Design
- Computational / modelling — Image-retrieval benchmark on pathologist-annotated TCGA slides comparing a pretrained deep-ranking embedding with SIFT and random retrieval, plus a blinded rating study by pathologists.
- N
- Database of 127,000 patches from 45 slides; query set of 22,500 patches from 15 other slides; three pathologists rated a subsample.
- Population
- Digitised breast, prostate and colon tissue slides from The Cancer Genome Atlas.
- Outcome
- Top-5 retrieval accuracy for matching histologic feature, organ and Gleason pattern; pathologist match ratings.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The team used a deep ranking network, trained on about 500,000,000 everyday photos to place similar images close together, to turn small patches of pathology slides into 128-number embeddings. A query patch is matched to the nearest embeddings in a database built from 127,000 patches from 45 slides. They scored retrieval against pathologist annotations and also had three pathologists, blinded to the source, rate SMILY results against random results.
What they found
For prostate queries, at least one of the top five results showed the same tissue feature 62.0% of the time, versus 44.2% for SIFT features and 28.3% for random retrieval. For prostate cancer grade, SMILY beat SIFT at 73.1% versus 62.1%. In the blinded study pathologists judged SMILY results as matching the tissue feature 62.1% of the time versus 26.8% for random results; matches were often right on tissue type but wrong on organ.
The limits
What it doesn't show
Annotations were coarse and incomplete, so some correct matches were scored as errors, and only three pathologists rated results with no measurement of their agreement. The test used a modest number of slides from one public archive and only a few organs and similarity axes. A general-purpose search tool is likely less accurate than a model trained for one specific task, and the study does not show any effect on diagnoses.
Key terms
- Embedding
- A fixed-length vector of numbers that summarises an image so that similar images have nearby vectors.
- Content-based image retrieval
- Searching a database using an image as the query, rather than text tags.
- Top-5 score
- The fraction of queries where at least one of the first five results is a correct match.
- Transfer learning
- Reusing a model trained on one kind of data (everyday photos) for a different domain (tissue images).
- SIFT
- Scale-invariant feature transform, a traditional hand-designed method for describing local image features.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was SMILY's embedding network trained on?
Common questions
Why is it surprising that this works?
The network never saw tissue during training, yet its general notion of visual similarity carried over well enough to beat a hand-crafted baseline.
Why include random results in the pathologist study?
As a blinded control, so raters could not simply score everything highly; SMILY had to beat random results they could not tell apart by source.
Why did matches fail on organ more than on tissue type?
Similar structures like fat or blood vessels look alike across organs, so the embedding grouped by appearance rather than by organ of origin.
More on Representation learning