Fast Parametric Learning with Activation Memorization

Rae, J; Dyer, C; Dayan, P; Lillicrap, C

Item

ITEM ACTIONSEXPORT

Add to Basket

Local TagsRelease HistoryDetailsSummary

Released

Conference Paper

Fast Parametric Learning with Activation Memorization

MPS-Authors

There are no MPG-Authors in the publication available

External Resource

http://proceedings.mlr.press/v80/rae18a/rae18a.pdf
(Publisher version)

Fulltext (restricted access)

There are currently no full texts shared for your IP range.

Fulltext (public)

There are no public fulltexts stored in PuRe

Supplementary Material (public)

There is no public supplementary material available

Citation

Rae, J., Dyer, C., Dayan, P., & Lillicrap, C. (2018). Fast Parametric Learning with Activation Memorization. In J. Dy, & A. Krasue (Eds.), International Conference on Machine Learning, 10-15 July 2018 (pp. 4228-4237). Madison, WI, USA: International Machine Learning Society.

Cite as: https://hdl.handle.net/21.11116/0000-0002-6847-A

Abstract

Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non-parametric model which stores recent activations and class labels into an external memory. We explore a simplified architecture where we treat a subset of the model parameters as fast memory stores. This can help retain information over longer time intervals than a traditional memory, and does not require additional space or compute. In the case of image classification, we display faster binding of novel classes on an Omniglot image curriculum task. We also show improved performance for word-based language models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia articles (WikiText-103) - the latter achieving a state-of-the-art perplexity of 29.2.