Do deep neural networks predict early visual neurons best?
Deep convolutional networks predicted monkey V1 responses to natural images far better than classic Gabor-style models, which suggests V1 computation involves several nonlinear steps.
Source
Deep convolutional models improve predictions of macaque V1 responses to natural images
Study at a glance
- Design
- Computational / modelling — Model comparison on held-out images: pretrained VGG-19 features with a regularised GLM readout, CNNs fitted directly to spikes, and LNP and Gabor filter bank baselines
- N
- N=166 · 166 V1 neurons, selected from 262 isolated in 17 sessions, went into the models. The neurons came from two monkeys.
- Population
- V1 neurons in two awake, fixating rhesus macaques viewing natural images and synthesised textures
- Outcome
- Fraction of explainable variance explained (FEV) in spike counts for held-out images
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors recorded spiking from V1 neurons in two monkeys while the animals viewed rapid sequences of natural images and textures synthesised from them. They then fitted four model types: a linear-nonlinear Poisson (LNP) model, a Gabor filter bank, a GLM readout from each layer of the ImageNet-trained VGG-19 network (the goal-driven approach), and CNNs trained directly on the neural data (the data-driven approach). Each model was scored on images it had not seen in training.
What they found
The best VGG layer was conv3_1, the fifth of its 16 convolutional layers, and it explained 51.6% of explainable variance. The data-driven CNN reached 49.8%, the Gabor filter bank 45.6% and the LNP only 16.3%. Careful sparsity-based regularisation of the readout was essential. The VGG model kept its performance with only a fifth of the training data, while the data-driven CNN needed the full dataset. The CNNs' gains over the Gabor model did not depend on whether cells were simple or complex, or on how sharply they were tuned.
The limits
What it doesn't show
The data come from only two monkeys, and responses were limited to brief, mostly feedforward windows after images flashed for 60 ms, so recurrent and feedback processing is not captured. The best models still leave almost half of the explainable variance unexplained. They also do not tell us which nonlinearities (for example divisive normalisation) they capture, because the CNNs are hard to interpret. Better prediction shows that the representation is similar to V1's, not that V1 is trained on object recognition.
Key terms
- Goal-driven model
- A network trained on a task such as object recognition, whose internal features are reused to predict neural responses.
- Data-driven model
- A network whose parameters are all fitted directly to the recorded neural responses.
- Fraction of explainable variance explained (FEV)
- The share of the stimulus-driven, repeatable part of a neuron's response that a model predicts, ignoring trial-to-trial noise.
- Linear-nonlinear Poisson (LNP) model
- A classic neuron model that applies a linear filter to the image, passes the result through a nonlinearity, and treats the output as a Poisson firing rate.
- Gabor filter bank
- A hand-designed set of oriented wavelet filters and energy terms that models V1 simple and complex cells.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which model best predicted V1 responses on average?
Common questions
Why didn't the deepest VGG layers predict V1 best?
V1 is an early stage of visual processing, so intermediate features matched it best. Performance rose up to the fifth layer and then fell off as features became more abstract and receptive fields grew.
Could the best layer just reflect the chosen image size?
The authors checked this by rescaling the input images. The best layer shifted somewhat, but conv3_1 at the original resolution still gave the best overall performance.
Why does the pretrained model need less data?
Its features are already learned from ImageNet, so only a sparse readout has to be fitted. The data-driven CNN must learn its whole feature space from the limited neural data.
More on Neural network models of the brain