How We Learn Value While Making Quick Decisions
Decisions between high-value options are faster because people lower their threshold for making a choice, not because they process information faster.
Source
A reinforcement learning diffusion decision model for value-based decisions
What they did
Researchers tested 32 participants in a learning task where they chose between pairs of symbols with different hidden reward values. Over 240 trials, participants received immediate feedback on their choices, while the researchers measured their accuracy and reaction times. The study used these data to test a new mathematical model that combines reinforcement learning with an evidence-accumulation decision process.
What they found
Participants became faster and more accurate as they learned the values of different options. When choosing between two highly rewarding options, participants made decisions faster than when choosing between two low-reward options, even though their accuracy remained the same. This speed increase occurred because participants lowered their decision threshold for attractive options, a process best explained by a model linking learned value directly to the decision boundary.
The limits
What it doesn't show
The study only tested binary choices, meaning the model cannot yet explain how people make decisions when presented with multiple options. It also relied on a relatively small sample of 27 analyzed participants, who were mostly young university students. Finally, the study cannot prove the underlying neural mechanisms behind these decisions, as it only collected behavioral data.
Key terms
- Reinforcement learning
- A framework describing how agents learn the value of actions by updating their expectations based on reward or punishment feedback.
- Diffusion decision model
- A model of decision-making that assumes evidence is accumulated over time until a pre-set threshold is reached.
- Drift rate
- The speed at which evidence is accumulated in a decision process, representing the ease of the decision.
- Decision threshold
- The amount of evidence required to make a decision, which balances speed and accuracy.
- Magnitude effect
- The phenomenon where people make faster decisions when choosing between two highly valuable options compared to two low-value options.
- Prediction error
- The difference between the received reward and the expected reward, used to update expectations.
Flashcards
Want these cards to stick?
Save the deck to NoteFren and study it with spaced repetition.
Quiz yourself
Standard reinforcement learning (RL) computational models are typically structured using which two components?
Common questions
Why can't standard reinforcement learning models predict reaction times?
Standard reinforcement learning models only predict the final choice probabilities based on learned values, lacking a mechanism to represent the time-dependent process of gathering evidence.
How does the proposed RLDDM improve upon older models?
It combines reinforcement learning with a diffusion model, allowing it to predict both what a person will choose and how long they will take, while updating these predictions trial-by-trial as the person learns.
What is the difference between the difficulty effect and the magnitude effect?
The difficulty effect makes people slower and less accurate when options have similar values, whereas the magnitude effect makes people faster (but not less accurate) when both options have high overall values.
Did the researchers find differences in how people learn from positive versus negative feedback?
Yes, the best-fitting models used separate learning rates for positive and negative prediction errors, indicating that people update their expectations differently depending on whether feedback is better or worse than expected.
More on Decision making