English
 
Help Privacy Policy Disclaimer
  Advanced SearchBrowse

Item

ITEM ACTIONSEXPORT
 
 
DownloadE-Mail
 Local TagsRelease HistoryDetailsSummary
  A local temporal difference code for distributional reinforcement learning

Tano, P., Dayan, P., & Pouget, A. (2021). A local temporal difference code for distributional reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, & H.-T. Lin (Eds.), Advances in Neural Information Processing Systems 33 (pp. 13662-13673). Red Hook, NY, USA: Curran.

Item is

 

show all hide all

Basic

show hide
Item Permalink: https://hdl.handle.net/21.11116/0000-0007-34C1-4 Version Permalink: https://hdl.handle.net/21.11116/0000-0008-E6D2-7
Genre: Conference Paper

Files

show Files

Locators

show
hide
Locator:
https://papers.nips.cc/paper/2020/file/9dd16e049becf4d5087c90a83fea403b-Paper.pdf (Any fulltext)
Description:
-
OA-Status:

Creators

show
hide
 Creators:
Tano, P, Author
Dayan, P1, 2, Author           
Pouget, A, Author
Affiliations:
1Department of Computational Neuroscience, Max Planck Institute for Biological Cybernetics, Max Planck Society, ou_3017468              
2Max Planck Institute for Biological Cybernetics, Max Planck Society, Spemannstrasse 38, 72076 Tübingen, DE, ou_1497794              

Content

show
hide
Free keywords: -
 Abstract: Recent theoretical and experimental results suggest that the dopamine system implements distributional temporal difference backups, allowing learning of the entire distributions of the long-run values of states rather than just their expected values. However, the distributional codes explored so far rely on a complex imputation step which crucially relies on spatial non-locality: in order to compute reward prediction errors, units must know not only their own state but also the states of the other units. It is far from clear how these steps could be implemented in realistic neural circuits. Here, we propose a local temporal difference code for distributional reinforcement learning that is representationally powerful and computationally straightforward. The code decomposes value distributions and prediction errors across three completely separated dimensions: reward magnitude (related to distributional quantiles), time horizon (related to eligibility traces) and temporal discounting (related to the Laplace transform of future immediate rewards). Besides lending itself to a local learning rule, the decomposition can be exploited by model-based computations, for instance allowing immediate adjustments to changing horizons or discount factors. Finally, we show that our code can be computed linearly from an ensemble of successor representations with multiple temporal discounts which, according to a recent proposal, might be implemented in the hippocampus.

Details

show
hide
Language(s):
 Dates: 2020-122021-07
 Publication Status: Issued
 Pages: -
 Publishing info: -
 Table of Contents: -
 Rev. Type: -
 Identifiers: -
 Degree: -

Event

show
hide
Title: 34th Conference on Neural Information Processing Systems (NeurIPS 2020)
Place of Event: -
Start-/End Date: 2020-12-06 - 2020-12-12

Legal Case

show

Project information

show

Source 1

show
hide
Title: Advances in Neural Information Processing Systems 33
Source Genre: Proceedings
 Creator(s):
Larochelle, H, Editor
Ranzato, M, Editor
Hadsell, R, Editor
Balcan, M-F, Editor
Lin, H-T, Editor
Affiliations:
-
Publ. Info: Red Hook, NY, USA : Curran
Pages: - Volume / Issue: - Sequence Number: - Start / End Page: 13662 - 13673 Identifier: ISBN: 978-1-7138-2954-6