Neuroscience · Vision · Convolutional networks
Deep networks trained on object labels predict visual cortex better than hand-built and unsupervised models
Save this development or follow its topic to track what changes.
Short answer
Networks trained on labelled object recognition currently give the best predictive models of visual cortex responses among those compared, from V1 to IT.
What happened
Comparing 37 model representations on 96 images, only a network trained on a million labelled images approached human IT's animate/inanimate split; its layer 7 matched IT better (tau-a 0.24) than the best combination of other models (0.17), and a reweighted version reached the noise ceiling. In 166 V1 neurons from two monkeys, a layer of the pretrained VGG network explained 51.6% of explainable variance, against 45.6% for a Gabor filter bank and 16.3% for a linear-nonlinear model.
Why it matters
It suggests that training a network on a demanding visual task produces internal representations resembling the brain's, even in early visual cortex, which is why such networks are now used as working models of vision.
Evidence
- Study type
- Model comparison against recorded brain responses
- Sample
- 96 images with human fMRI and data from two monkeys; 166 V1 neurons from two monkeys
- Journal
- PLoS Computational Biology · peer reviewed
- Replication
- Two independent groups at different stages of the visual hierarchy report the same direction
- Limitations
- Small stimulus sets, few animals, brief feedforward response windows, and no account of which computations the network features correspond to.
What this connects to
Sources
The 2 studies this explanation is built from, by the role each plays. Every source links to PaperFren’s explanation of it and to the original paper.
Primary study
- Which vision models represent objects like the brain's IT cortex?
Only a deep network trained with a million labelled images came close to the category structure of inferior temporal cortex; unsupervised and hand-engineered models did not.
What it does not showLimitations
The stimulus set is small (96 images of isolated objects) and was split equally between animate and inanimate objects, which shapes the categoricality measure. Only one deep supervised network was tested, so supervision is confounded with that architecture and its large training set. The best-fitting model was partly fitted to the IT data, though with cross-validation. The noise ceiling could not be estimated for monkey IT because data came from only two animals.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Supporting evidence
- Do deep neural networks predict early visual neurons best?
Deep convolutional networks predicted monkey V1 responses to natural images far better than classic Gabor-style models, which suggests V1 computation involves several nonlinear steps.
What it does not showLimitations
The data come from only two monkeys, and responses were limited to brief, mostly feedforward windows after images flashed for 60 ms, so recurrent and feedback processing is not captured. The best models still leave almost half of the explainable variance unexplained. They also do not tell us which nonlinearities (for example divisive normalisation) they capture, because the CNNs are hard to interpret. Better prediction shows that the representation is similar to V1's, not that V1 is trained on object recognition.
PaperFren explanationStudy with cards and a quizOriginal paper (DOI)cc by
Before
Early visual cortex is well described by simple Gabor-like filters, and high-level categories need hand-designed or unsupervised features to model.
Now
Task-trained deep networks predicted both V1 and IT better in these studies. They still leave nearly half of V1's explainable variance unexplained, only one supervised network was tested for IT so supervision is confounded with architecture and data, and the data come from small image sets and few animals.