When is it worth stopping to think before acting?
A model in which the brain only plans when the expected gain from better information beats the reward lost by waiting explains why well-practised actions become fast, inflexible habits.
Source
Speed/accuracy trade-off between the habitual and the goal-directed processes
Study at a glance
- Design
- Computational / modelling — Simulations of an agent combining Kalman model-free learning with model-based value iteration and a cost-benefit arbitrator, run on formalised versions of published animal and human tasks.
- N
- No participants or single dataset; simulated agents are compared qualitatively with published behavioural results.
- Population
- Simulated reinforcement-learning agents modelled on rats and humans.
- Outcome
- Response rates after outcome devaluation, deliberation time, and choice reaction time patterns.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors built an agent with two systems: a fast habitual (model-free) learner that keeps uncertainty estimates for action values, and a slow but accurate goal-directed (model-based) planner. Before each choice, an arbitrator compares the value of perfect information about an action with the cost of deliberating, taken as time multiplied by the average reward rate. They simulated the agent on outcome-devaluation experiments after moderate (40 trials) versus extensive (240 trials) training, a two-lever concurrent schedule, a reversal-learning reaction-time task and choice reaction-time tasks with varying numbers of options.
What they found
After moderate training the simulated agent reduced responding for a devalued outcome, but after extensive training it kept responding, matching the classic finding that overtraining creates habits; deliberation time fell to zero after about 100 training trials. With two equally valued outcomes available concurrently, the agent stayed goal-directed even after long training, as reported in rats. The model also reproduced slower reaction times while searching for a rule than while applying it, and reaction times that grow with the number of options early in training but not later.
The limits
What it doesn't show
The fits to data are qualitative comparisons with published figures, not quantitative fits to raw data. The model assumes the planner has perfect knowledge of values, which the authors admit fails in tasks like reversal learning, and it predicts a linear rather than the observed logarithmic growth of reaction time with the number of choices. The proposed role of tonic dopamine as the average reward signal is a hypothesis, not tested here.
Key terms
- Habitual (model-free) system
- A fast system that caches learned action values from past rewards but adapts slowly when outcomes change.
- Goal-directed (model-based) system
- A slower system that plans using knowledge of what actions lead to, so it adapts quickly to changed outcomes.
- Outcome devaluation
- Making a reward undesirable (for example by pairing it with illness) and testing whether the animal still works for it.
- Value of perfect information
- The expected improvement in choices from knowing an action's true value, computed from the habit system's uncertainty.
- Opportunity cost of time
- The reward forgone while deliberating, estimated here from the average reward rate.
Flashcards
0 of 9 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
In this model, when does the agent call on the goal-directed system?
Common questions
Why do habits form with extensive training in this model?
As habitual value estimates become confident and well separated, extra information adds little, so the cost of deliberating outweighs its benefit and the agent acts habitually.
Why did extended training not create habits in the two-lever task?
When both outcomes are equally valuable, their value estimates keep overlapping, so information about them stays valuable and planning remains worthwhile.
What does dopamine have to do with it?
The model suggests tonic dopamine could signal the average reward rate, which sets the cost of time and therefore how often the agent deliberates.
More on Reinforcement learning