Could brain-like local learning rules approximate backpropagation?
A predictive coding network that changes each synapse using only the activity of the two neurons it connects can learn almost exactly like backpropagation, reaching the same accuracy on handwritten digits.
Source
An Approximation of the Error Backpropagation Algorithm in a Predictive Coding Network with Local Hebbian Synaptic Plasticity
Study at a glance
- Design
- Computational / modelling — Mathematical analysis plus simulations comparing weight updates and learning curves of a predictive coding network and a standard backprop network (toy 1-1-1 net, 784-600-600-10 MNIST net, bidirectional linear association task).
- N
- No single N; MNIST learning curves are averaged over 10 simulations per model, and toy tasks use randomly generated samples.
- Population
- Simulated artificial neural networks trained on synthetic data and MNIST handwritten digits
- Outcome
- Angle between predictive coding and backprop weight updates; MNIST training and validation error; prediction error in both input-to-output and output-to-input directions
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors built a supervised predictive coding network in which separate error nodes signal the mismatch between each layer and the prediction from the layer above, and weights change by a simple Hebbian product of pre- and postsynaptic activity. They proved when its updates match backpropagation and checked this in a tiny three-layer network, then trained a 784-600-600-10 version on MNIST and compared it with an ordinary backprop network. They also tested an architecture where inputs and outputs both enter at the bottom, letting the network predict in both directions.
What they found
In prediction mode the network settles to exactly the same activities as the equivalent artificial network. Its weight updates point in nearly the same direction as backprop near a good solution, and converge to backprop as the variance parameter on the output layer is made large. On MNIST all networks reached 0.00% training error and 1.7% to 1.8% validation error, with learning curves similar to backprop. With equal noise assumed on inputs and outputs, the network predicted reasonably well in both directions, whereas backprop is optimal only for input-to-output prediction.
The limits
What it doesn't show
Exact equivalence with backprop happens only in a limit where error-node activity becomes tiny, which the authors admit may be hard for spiking neurons to signal reliably. The model assumes symmetric forward and backward weights, one-to-one pairing of value and error neurons, and error neurons that can go negative, none of which is established in cortex. MNIST learning rates were not tuned and only small fully connected networks were tested, so it says nothing about scaling to modern deep architectures. It is a simulation study with no neural recordings, so it shows biological feasibility, not that the brain actually learns this way.
Key terms
- Backpropagation
- The standard deep-learning algorithm that sends output errors backward through layers to compute how each weight should change.
- Predictive coding
- A theory in which each brain level predicts activity at the level below and dedicated neurons encode the prediction error.
- Hebbian plasticity
- A local learning rule where a synapse changes according to the activity of the two neurons it connects.
- Local computation
- The requirement that each neuron and synapse only use information physically available to it, not global signals.
- Variance parameter
- A scaling term on each error node; making it large at the output layer makes the network's learning converge to backprop.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What makes the predictive coding learning rule more biologically plausible than backprop?
Common questions
Why is backprop thought to be biologically implausible?
Each weight update depends on errors and weights from many distant neurons, whereas real synapses only see the activity of the neurons on either side.
Does this prove the brain uses backpropagation?
No. The authors actually argue the brain probably does not implement backprop exactly, and that predictive coding with balanced noise may suit real learning problems better.
What is gained by learning in both directions?
The same trained network can infer a likely input from an output, like recalling a letter shape from its sound, which a feedforward backprop net cannot do.
More on Neural network models of the brain