Abstract
Humans and other animals are able to discover underlying statistical structure in their environ-
ments and exploit it to achieve efficient and effective performance. However, the largest scale structures
such as ‘world models’ are often difficult to learn and use because they are obscure, involving long-range temporal dependencies. Here, we analyzed behavioral data from a lengthy experiment with rats, showing
that subjects discovered such hidden structure, using it to respond more quickly to rewarding states whilst
responding more slowly, or not at all, to unrewarding states. We also identified surprising occasions where
subjects responded rapidly to unrewarding states, despite the structure of the task seemingly having been
learned. We attributed these instances to immediate inferential imperfections caused by the partial observ-
ability of hidden states. To describe this process statistically, we built a hidden Markov model (HMM) of
the subjects’ models of the experiment, describing overall behavior as integrating recent observations with
the recollections of an imperfect memory. Over the course of training, we found that subjects came to
track their progress through the task more accurately, indicating an improved ability to infer state. Model
fits attributed this improvement to decreased forgetting of the previous state. This ‘learning to remember’
decreased reliance on more recent observations, which can be misleading, in favor of a more dependable
memory.
