Item

Abstract

Aim: The nature of attention, and how it interacts with learning and choice processes in the
context of reinforcement learning, is still unclear. Probabilistic accounts of associative
learning, as well as approximately optimal solutions of the exploration-exploitation dilemma,
suggest that both learned value and uncertainty about those values (i.e. reducible or
estimation uncertainty) are important for learning and choice. This implies that both factors
should jointly guide attention. Our main goal was to test this prediction. Our secondary goal
was to examine whether the relation between attention and reinforcement learning is
bidirectional, whether attention also influences or biases what we learn and how we choose.
There are some tests of this direction of influence; however, the role of estimation uncertainty
has not previously been addressed.
Method: Participants (N=36) completed two games in which they repeatedly chose between
six options. Each game was a multi-armed bandit task where rewards for each option were
drawn from Gaussian distributions, differing in both their means and variances. The
participants' goal was to maximize the cumulative sum of rewards in each game. To do this,
they needed to explore the options in the choice set in order to learn which option had the
highest average reward, and subsequently exploit this knowledge. We monitored participants'
attention using eye tracking while they performed the tasks, operationalizing attention as the
proportion of time spent fixating on each of the options before making a choice.
Results: We relied on computational modeling to garner evidence for our two questions. To
address our main question, we modeled attention with a combination of a Bayesian (Kalman
filter) learning component and two types of choice rules: one that relies only on learned value (softmax) and one that additionally uses estimation uncertainty to assign an "exploration
bonus" to the options (upper confidence bound rule). Model evidence showed that Kalman
filter learning with the exploration bonus described overt attention best, providing evidence
that trial-by-trial learned values and estimation uncertainty jointly guide visual attention. For
our secondary question, we used the same models to model choices, but allowing measured
attention to affect the choice process by increasing the probability of choosing attended
options and decreasing it for unattended options. Attention was also allowed to modulate the
magnitude of updates in the learning process. Again, we found that Kalman filter learning with
exploration bonus was the best model, showing that estimation uncertainty plays an
independent role in determining choice, over and above its effect on attention.
Conclusions: In summary, the interaction between attention, learning, and decision making,
extends further than previously found. Our results provide support for probabilistic associative
learning accounts that ground attention in efficient computations rather than constraints, and
establish a relation with approximately optimal resolutions of the exploration-exploitation trade-off.