Reinforcement learning
Can a predictive map explain planning with simple learning?
Open access · cc by · source: Europe PMC
Learning values on top of a map of which states tend to follow which lets an agent show flexible, planning-like behaviour using the same prediction-error rule linked to dopamine, and adding offline replay closes the remaining gaps.
Study at a glance
- Design
- Computational / modelling — Grid-world simulations of SR-TD, SR-MB and SR-Dyna agents (plus model-free and Dyna-Q foils) on latent learning, detour and policy revaluation tasks.
- N
- No sample size: results are simulated agent choices on the first test trial after each task manipulation; the Methods section is not in the available text.
- Population
- Simulated reinforcement-learning agents in grid-world mazes modelled on Tolman's rodent experiments
- Outcome
- Whether each agent immediately chooses the correct new shortest path after a reward change, a transition change, or a policy-relevant reward change
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
SR-TD handled reward revaluation such as latent learning but failed the detour task, because its cached predictions only change along paths it actually re-experiences. SR-MB solved both latent learning and detours but failed a new policy revaluation task, because its map is tied to the policy it previously followed. With enough offline replay, SR-Dyna solved all three tasks like full model-based value iteration, and with too little replay it fell back to SR-TD rather than to purely model-free behaviour. A foil with only one step of look-ahead failed every revaluation task.
Methodology
The authors framed classic Tolman-style rat experiments as grid-world tasks and simulated agents that learn a successor representation (expected future visits to each state) and then learn reward weights with ordinary temporal-difference learning. They compared three ways of building that representation: learning it by TD from experience (SR-TD), recomputing it at decision time from a learned one-step transition model (SR-MB), and updating it offline by replaying stored transitions (SR-Dyna). Each agent was tested on the first trial after a reward change, a blocked path, or a new reward the old policy never led to.
Limitations
All results are simulations in small tabular grid worlds; no animal or human data were fitted, so the paper shows what each algorithm can do in principle rather than what brains actually do. The tests are categorical (pass or fail on the first trial) and the authors say they checked robustness across parameters but do not report quantitative fits. Replay was modelled with a naive uniform sampling scheme that the authors say does not reflect realistic hippocampal replay, and scaling to large environments would need better replay prioritisation. The proposed mapping onto dorsomedial versus dorsolateral striatum is a hypothesis, not a tested result.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Habit vs plan may be a spectrum built from shared parts.
Intermediate or hierarchical mechanisms can explain behaviour sitting between the two systems: people repeated whole rewarded action sequences (hierarchical models beat flat ones, exceedance probability ≈0.99), and successor-representation agents with enough offline replay solved revaluation tasks like a full planner but fell back to cached behaviour with too little replay.
Evidence for the claim as stated.
Matching behaviour is a hypothesis about the brain, not a measurement of it.
Algorithms can be matched to behavioural signatures without neural data: successor-representation agents reproduce which revaluation tasks animals pass or fail, and the proposed striatal mapping is presented as a hypothesis.
Evidence for the claim as stated.
Related papers in this topic
Same topic cluster — not a recommendation engine.