How should a learner speed up learning when the world keeps changing?
A simple extension of the Kalman filter that also tracks how fast the world is changing approximates optimal learning more accurately than the popular HGF and explains most people's choices better.
Source
A simple model for learning in volatile environments
Study at a glance
- Design
- Computational / modelling — New Bayesian learning algorithm (VKF) tested in simulations against ground truth, a particle filter and the HGF, then fitted to two existing human choice datasets with hierarchical model comparison.
- N
- Two reanalysed human datasets: 44 participants in the first task and 161 analysed (of 174) in the second; simulations used 1000 or 500 generated sequences.
- Population
- Simulated volatile environments; adult participants in two published probabilistic reversal-learning tasks.
- Outcome
- Tracking error relative to a near-optimal particle filter; learning-rate/volatility relationship; model evidence for fitting human choices.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors built a generative model in which both a hidden reward value and its rate of change (volatility) drift over time, and derived a cheap approximate learning rule for it called the volatile Kalman filter (VKF), plus a version for binary win/lose outcomes. They compared its estimates with a near-exact particle filter and with the hierarchical Gaussian filter (HGF) on simulated data. They then fitted VKF, HGF, a plain Kalman filter and a Rescorla-Wagner model to choices from two existing human reversal-learning datasets.
What they found
Measured as extra error over the particle-filter benchmark, the VKF was off by only a few percent while the HGF was off by about a fifth, and the HGF sometimes crashed with impossible negative variances. In the binary version, VKF's learning rate rose with its volatility estimate (as Pearce-Hall theory predicts), whereas the binary HGF showed the opposite relationship in almost all simulations. VKF was the best-fitting model for 37 of 44 people in the first dataset and 102 of 161 in the second, and most people's fitted learning rates rose after contingency switches.
The limits
What it doesn't show
The human tests reuse two existing datasets rather than a task designed to discriminate the models, and in the second task about 30% of people were best fit by a plain Kalman filter, so volatility tracking is not universal. VKF only handles two levels and gives up the HGF's ability to stack deeper hierarchies. The observation noise is treated as a fixed parameter, whereas real learners may need to estimate it too, and parameter recovery showed the volatility parameters were harder to estimate reliably. Better model fit does not prove the brain literally runs this algorithm.
Key terms
- Kalman filter
- An optimal update rule for tracking a hidden quantity that drifts at a known constant rate, where the learning rate depends on current uncertainty.
- Volatility
- How quickly the hidden state of the environment is changing; in volatile periods a learner should update beliefs faster.
- Learning rate
- The fraction of a prediction error used to update a belief on each trial.
- Pearce-Hall model
- A classic associative-learning theory in which surprising outcomes increase how much is learned from later trials.
- Particle filter
- A sampling-based method that approximates exact Bayesian inference using many simulated hypotheses; accurate but computationally expensive.
- Hierarchical Gaussian filter (HGF)
- A widely used multi-level model of learning under volatility that relies on Taylor approximations.
Flashcards
0 of 12 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What extra quantity does the VKF track beyond the ordinary Kalman filter?
Common questions
Why not just use the exact Bayesian answer?
Exact inference in this two-level model is intractable; sampling methods like particle filters approximate it but are too slow to be a plausible brain algorithm or to fit many participants easily.
How is VKF different from the HGF?
VKF assumes volatility changes by positive multiplicative noise, which lets its update be solved exactly under a variational approximation; the HGF needs an extra Taylor approximation that can produce negative variances.
What does this mean for psychology?
It gives a normative justification for hybrid Rescorla-Wagner/Pearce-Hall learning, and its parameters can serve as measures of individual differences such as anxiety-related learning deficits.
More on Uncertainty and calibration