Help Privacy Policy Disclaimer
  Advanced SearchBrowse





A Local Temporal Difference Code for Distributional Reinforcement Learning


Dayan,  P       
Department of Computational Neuroscience, Max Planck Institute for Biological Cybernetics, Max Planck Society;

Fulltext (restricted access)
There are currently no full texts shared for your IP range.
Fulltext (public)
There are no public fulltexts stored in PuRe
Supplementary Material (public)
There is no public supplementary material available

Tano, P., Dayan, P., & Pouget, A. (2021). A Local Temporal Difference Code for Distributional Reinforcement Learning. Poster presented at Computational and Systems Neuroscience Meeting (COSYNE 2021).

Cite as: https://hdl.handle.net/21.11116/0000-0007-F694-C
Recent theoretical and experimental results suggest that the dopamine system implements distributional temporal difference backups, allowing learning of the entire distributions of the long-run values of states rather than just their expected values. However, the distributional codes explored so far rely on a complex imputation step which crucially relies on spatial non-locality: in order to compute reward prediction errors, units must know not only their own state but also the states of the other units. It is far from clear how these steps could be implemented in realistic neural circuits. Here, we introduce the Laplace code: a local temporal difference code for distributional reinforcement learning that is representationally powerful and computationally straightforward. The code decomposes value distributions and prediction errors across three separated dimensions: reward magnitude (related to distributional quantiles), temporal discounting (related to the Laplace transform of future rewards) and time horizon (related to eligibility traces). Besides lending itself to a local learning rule, the decomposition recovers the temporal evolution of the immediate reward distribution, indicating all possible rewards at all future times.This increases representational capacity and allows for temporally-flexible computations that immediately adjust to changing horizons or discount factors.