Does planning ahead actually earn more reward in lab tasks?
In the most widely used planning-versus-habit task, planning earns essentially no extra reward, but a redesigned task makes planning pay off and people plan more in it.
Source
When Does Model-Based Control Pay Off?
Study at a glance
- Design
- Human experiment — Monte Carlo simulations of hybrid model-based/model-free RL agents across task variants, then an online experiment assigning participants to the original Daw two-step task or a new task, with RL model fitting.
- N
- N=381 · 381 Mechanical Turk participants analysed after exclusions, split between the novel task and the original Daw task; simulations used synthetic agents.
- Population
- Adult online participants recruited through Amazon Mechanical Turk; simulated reinforcement-learning agents.
- Outcome
- Relationship between the model-based weighting parameter w and reward rate; average w in each task.
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors simulated agents that mix model-free learning (repeat what was rewarded) and model-based planning (use a map of the task), varying the mixing weight w, learning rate and choice randomness, on the Daw two-step task and several variants. They then changed task features one at a time to find what makes planning rewarding, and combined all of them into a new task. Finally, online participants played either the original task or the new task for 125 trials each, and hybrid RL models were fitted to each person's choices.
What they found
Across nearly the whole parameter range, more model-based control did not raise reward in the original task or two published variants. Five features restricted the benefit: low-contrast reward probabilities, slow drift, rare transitions, a choice at the second stage, and noisy binary rewards; fixing all five together boosted the planning-reward link by a factor of about 230. In people, w correlated with reward in the new task (r = 0.55) but not the original (r = 0.10), and median w was higher in the new task (0.48 vs 0.27).
The limits
What it doesn't show
The human result is correlational between two task groups, so the higher planning in the new task could reflect other features such as deterministic transitions or negative rewards rather than a cost-benefit calculation, as the authors note. The paper does not measure effort or cognitive cost directly, only reward. Participants were online workers with limited trials, and model comparison for the original task partly favoured a pure model-free model.
Key terms
- Model-free learning
- Learning action values directly from past rewards without representing how the environment works; fast but inflexible, like a habit.
- Model-based learning
- Learning a model of how actions lead to states and rewards, then planning with it; flexible but computationally demanding.
- Two-step task
- A sequential choice task where a first choice leads probabilistically to one of two second-stage states with changing rewards, used to separate planning from habit.
- Weighting parameter w
- In the hybrid model, the share of choice driven by model-based values (1 = pure planning, 0 = pure habit).
- Accuracy-demand trade-off
- The idea that effortful planning should be worth it only if it earns more reward than cheaper habits.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What was the main simulation finding for the original two-step task?
Common questions
If planning doesn't pay in the original task, are its past results useless?
No. The task still measures how much people plan, and that measure relates to working memory, age and psychiatric symptoms; it just cannot test whether people plan more when planning pays.
Why did no single change fix the task?
Each change helped only a little alone; the effects were superadditive, so all five were needed for a strong trade-off.
Why fit models rather than just look at stay/switch rates?
In the new task, raw stay probabilities look similar for model-free and mixed agents, so model fitting across all trials is needed to estimate w.
More on Reinforcement learning