Can a predictive map explain planning with simple learning?
Learning values on top of a map of which states tend to follow which lets an agent show flexible, planning-like behaviour using the same prediction-error rule linked to dopamine, and adding offline replay closes the remaining gaps.
Source
Predictive representations can link model-based reinforcement learning to model-free mechanisms
Study at a glance
- Design
- Computational / modelling — Grid-world simulations of SR-TD, SR-MB and SR-Dyna agents (plus model-free and Dyna-Q foils) on latent learning, detour and policy revaluation tasks.
- N
- No sample size: results are simulated agent choices on the first test trial after each task manipulation; the Methods section is not in the available text.
- Population
- Simulated reinforcement-learning agents in grid-world mazes modelled on Tolman's rodent experiments
- Outcome
- Whether each agent immediately chooses the correct new shortest path after a reward change, a transition change, or a policy-relevant reward change
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors framed classic Tolman-style rat experiments as grid-world tasks and simulated agents that learn a successor representation (expected future visits to each state) and then learn reward weights with ordinary temporal-difference learning. They compared three ways of building that representation: learning it by TD from experience (SR-TD), recomputing it at decision time from a learned one-step transition model (SR-MB), and updating it offline by replaying stored transitions (SR-Dyna). Each agent was tested on the first trial after a reward change, a blocked path, or a new reward the old policy never led to.
What they found
SR-TD handled reward revaluation such as latent learning but failed the detour task, because its cached predictions only change along paths it actually re-experiences. SR-MB solved both latent learning and detours but failed a new policy revaluation task, because its map is tied to the policy it previously followed. With enough offline replay, SR-Dyna solved all three tasks like full model-based value iteration, and with too little replay it fell back to SR-TD rather than to purely model-free behaviour. A foil with only one step of look-ahead failed every revaluation task.
The limits
What it doesn't show
All results are simulations in small tabular grid worlds; no animal or human data were fitted, so the paper shows what each algorithm can do in principle rather than what brains actually do. The tests are categorical (pass or fail on the first trial) and the authors say they checked robustness across parameters but do not report quantitative fits. Replay was modelled with a naive uniform sampling scheme that the authors say does not reflect realistic hippocampal replay, and scaling to large environments would need better replay prioritisation. The proposed mapping onto dorsomedial versus dorsolateral striatum is a hypothesis, not a tested result.
Key terms
- Successor representation (SR)
- A table of how often, discounted over time, an agent expects to visit each future state from each current state under its policy.
- Temporal-difference (TD) learning
- Learning predictions by nudging them toward the reward just received plus the prediction for the next state; its error signal is linked to phasic dopamine.
- Reward revaluation
- A test where the value of an outcome changes (for example a new reward appears) and the agent must adjust choices without re-experiencing the full path.
- Transition revaluation (detour)
- A test where the layout changes, such as a blocked corridor, and the agent must replan around it.
- On-policy vs off-policy
- On-policy estimates assume the agent keeps behaving as before; off-policy estimates assume it will pick the best action, even one it has rarely taken.
- Dyna
- An approach where stored experiences are replayed offline, for example during rest, to update learned quantities without new real experience.
Flashcards
0 of 11 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which task does SR-TD fail but SR-MB pass?
Common questions
Why can't a normal model-free learner pass latent learning?
It stores a single cached value per state, so learning that a location is now rewarding changes nothing about distant states until the agent walks paths from them to the reward again.
If SR-Dyna matches full planning, why propose three algorithms?
Each makes different, testable errors: SR-TD fails detours, SR-MB fails policy revaluation, and SR-Dyna depends on having time to replay, so behavioural experiments can tell them apart.
Does this paper prove dopamine supports model-based learning?
No. It shows a mechanism by which the standard dopamine prediction-error rule could produce model-based-looking behaviour, consistent with prior lesion and imaging findings, but it does not test this with new neural data.
More on Reinforcement learning