Which vision models represent objects like the brain's IT cortex?
Only a deep network trained with a million labelled images came close to the category structure of inferior temporal cortex; unsupervised and hand-engineered models did not.
Source
Deep supervised, but not unsupervised, models may explain IT cortical representation
Study at a glance
- Design
- Computational / modelling — Representational similarity analysis comparing model RDMs with existing human fMRI and monkey cell-recording RDMs for 96 object images, plus cross-validated feature reweighting
- N
- 37 model representations were compared on a set of 96 images. The brain data were reused from earlier human fMRI and monkey recording studies, and the monkey data came from two animals.
- Population
- Computational vision models, compared with human IT (fMRI) and monkey IT (cell recordings)
- Outcome
- Kendall tau-a correlation between model and IT representational dissimilarity matrices, and a categoricality index
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors tested 37 model representations, including neuroscience-inspired models like HMAX, classic computer-vision features like SIFT and GIST, and every layer of a deep convolutional network trained on ImageNet. They used representational similarity analysis, comparing how each model and the brain separate the same 96 object images, against human IT fMRI and monkey IT recordings. They also tried reweighting and remixing model features with cross-validation to see whether a better IT match could be built.
What they found
The not-strongly-supervised models all correlated only weakly with IT, and none showed IT's strong animate/inanimate split or its face cluster spanning humans and animals. Human IT had a categoricality index of 0.4, while all these models scored below 0.16. Layer 7 of the deep supervised network matched human IT better (tau-a 0.24) than the best combination of other models (0.17). Reweighting the deep network's layers together with category-discriminant features produced a model (tau-a 0.38) that reached the noise ceiling for human IT.
The limits
What it doesn't show
The stimulus set is small (96 images of isolated objects) and was split equally between animate and inanimate objects, which shapes the categoricality measure. Only one deep supervised network was tested, so supervision is confounded with that architecture and its large training set. The best-fitting model was partly fitted to the IT data, though with cross-validation. The noise ceiling could not be estimated for monkey IT because data came from only two animals.
Key terms
- Representational similarity analysis (RSA)
- Comparing systems by how similarly they separate the same stimuli, using matrices of pairwise response dissimilarities.
- Representational dissimilarity matrix (RDM)
- A table listing, for every pair of images, how different a system's responses to them are.
- Noise ceiling
- The best correlation any model could achieve given the noise in the brain data.
- Categoricality index
- The proportion of RDM variance explained by category divisions such as animate versus inanimate.
- Supervised learning
- Training a model with labelled examples, here category labels for images.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which model best explained human IT before any reweighting?
Common questions
Why use RSA instead of predicting each neuron directly?
The models have up to millions of units. Fitting a linear map from each unit to each brain response would need far more data. RSA compares representational geometry directly without fitting that map.
Does this prove the brain learns by supervision?
No. It shows that features learned with heavy category supervision match IT better than the unsupervised models tested, which is consistent with, but not proof of, category-driven learning.
Why did the final readout layer fit worse than layer 7?
The readout layer holds one unit per ImageNet class. It is tied to the specific training labels, while layer 7 is a richer continuous representation.
More on Neural network models of the brain