English
 
Help Privacy Policy Disclaimer
  Advanced SearchBrowse

Item

ITEM ACTIONSEXPORT
Add to Basket
 Local TagsRelease HistoryDetailsSummary
  Fast Parametric Learning with Activation Memorization

Rae, J., Dyer, C., Dayan, P., & Lillicrap, C. (2018). Fast Parametric Learning with Activation Memorization. In J. Dy, & A. Krasue (Eds.), International Conference on Machine Learning, 10-15 July 2018 (pp. 4228-4237). Madison, WI, USA: International Machine Learning Society.

Item is

 

show all hide all

Basic

show hide
Item Permalink: https://hdl.handle.net/21.11116/0000-0002-6847-A Version Permalink: https://hdl.handle.net/21.11116/0000-0002-6848-9
Genre: Conference Paper

Files

show Files

Locators

show
hide
Locator:
http://proceedings.mlr.press/v80/rae18a/rae18a.pdf (Publisher version)
Description:
-
OA-Status:

Creators

show
hide
 Creators:
Rae, J, Author
Dyer, C, Author
Dayan, P1, Author           
Lillicrap, C, Author
Affiliations:
1External Organizations, ou_persistent22              

Content

show
hide
Free keywords: -
 Abstract: Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non-parametric model which stores recent activations and class labels into an external memory. We explore a simplified architecture where we treat a subset of the model parameters as fast memory stores. This can help retain information over longer time intervals than a traditional memory, and does not require additional space or compute. In the case of image classification, we display faster binding of novel classes on an Omniglot image curriculum task. We also show improved performance for word-based language models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia articles (WikiText-103) - the latter achieving a state-of-the-art perplexity of 29.2.

Details

show
hide
Language(s):
 Dates: 2018-07
 Publication Status: Issued
 Pages: -
 Publishing info: -
 Table of Contents: -
 Rev. Type: -
 Identifiers: -
 Degree: -

Event

show
hide
Title: 35th International Conference on Machine Learning (ICML 2018)
Place of Event: Stockholm, Sweden
Start-/End Date: 2018-07-10 - 2018-07-15

Legal Case

show

Project information

show

Source 1

show
hide
Title: International Conference on Machine Learning, 10-15 July 2018
Source Genre: Proceedings
 Creator(s):
Dy, J, Editor
Krasue, A, Editor
Affiliations:
-
Publ. Info: Madison, WI, USA : International Machine Learning Society
Pages: - Volume / Issue: - Sequence Number: - Start / End Page: 4228 - 4237 Identifier: -

Source 2

show
hide
Title: PMLR Proceedings of Machine Learning Research
Source Genre: Series
 Creator(s):
Affiliations:
Publ. Info: -
Pages: - Volume / Issue: 80 Sequence Number: - Start / End Page: - Identifier: -