Can habits look like planning in the two-step task?
Agents that never plan can still produce the behavioural pattern usually taken as proof of planning, so the standard test for 'model-based' choice can be fooled.
Source
Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task
Study at a glance
- Design
- Computational / modelling — Simulated model-free, model-based, reward-as-cue and latent-state agents on the original and a reduced two-step task, analysed with stay-probability regressions, lagged regressions and likelihood model comparison.
- N
- No participants; results come from large simulated datasets of agent choices.
- Population
- Simulated reinforcement-learning agents.
- Outcome
- Fraction of rewarded trials; regression loading on the transition-by-outcome interaction; model-fit likelihood.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors simulated reinforcement-learning agents on the popular two-step task and on a reduced version designed for animals, with more common transitions (probability raised from 0.7 to 0.8), no second-step choice and blocks of clearly good and bad options. They compared a simple model-free agent, a planning (model-based) agent, and two agents that treat where the last reward came from as a cue, including one that infers a hidden 'which side is good' state. They analysed choices with the usual stay-probability test, an extended regression and model fitting.
What they found
On the original task, planning barely paid: the model-free and model-based agents earned rewards on about 0.558 and 0.557 of trials. On the reduced task planning paid more (0.649 versus 0.594), but the model-free agent now showed the transition-by-outcome interaction thought to signal planning, because its starting action values correlated with later trial events; adding a 'repeat correct choice' predictor removed this artefact. The latent-state agent looked almost identical to a planner on regression tests and could only be told apart by likelihood model comparison.
The limits
What it doesn't show
Everything is simulation; the paper does not show that real humans or animals actually use reward-as-cue or latent-state strategies. The authors argue the artefact is very weak in the original task, so existing human findings are probably not undermined. Model comparison worked on huge simulated datasets, but real datasets are much smaller, real subjects may mix strategies, and fitted models will not match them exactly.
Key terms
- Model-based reinforcement learning
- Choosing actions by using a learned model of what each action leads to, planning ahead to evaluate options.
- Model-free reinforcement learning
- Learning the value of actions directly from reward prediction errors, without representing their consequences.
- Two-step task
- A choice task where a first choice usually (but not always) leads to one of two second-stage states that give rewards with changing probabilities.
- Stay probability
- The chance of repeating the same first-step choice on the next trial, split by whether the last transition was common or rare and whether it was rewarded.
- Latent state
- A hidden feature of the world, here which second-step state currently has the higher reward probability, that must be inferred from evidence.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which pattern is classically taken as evidence of model-based control?
Common questions
Why did the model-free agent show a 'planning' signature?
Its action values at the start of a trial were larger when it chose correctly, and correct choices make rewarded-common and unrewarded-rare trials more likely, creating an interaction without using transition information.
Does this mean human two-step studies are wrong?
Not necessarily; the authors say the artefact is very weak in the original task and humans are told the task structure and do few trials.
How can we tell a latent-state agent from a planner?
Simple regressions look alike, but fitting both models and comparing how well each explains the choices can separate them, given enough data.
More on Reinforcement learning