English
 
Help Privacy Policy Disclaimer
  Advanced SearchBrowse

Item

ITEM ACTIONSEXPORT
  Towards Strong Pruning for Lottery Tickets with Non-Zero Biases

Fischer, J., & Burkholz, R. (2021). Towards Strong Pruning for Lottery Tickets with Non-Zero Biases. Retrieved from https://arxiv.org/abs/2110.11150.

Item is

Files

show Files
hide Files
:
arXiv:2110.11150.pdf (Preprint), 304KB
Name:
arXiv:2110.11150.pdf
Description:
File downloaded from arXiv at 2021-12-29 13:43
OA-Status:
Visibility:
Public
MIME-Type / Checksum:
application/pdf / [MD5]
Technical Metadata:
Copyright Date:
-
Copyright Info:
-

Locators

show

Creators

show
hide
 Creators:
Fischer, Jonas1, Author           
Burkholz, Rebekka2, Author
Affiliations:
1Databases and Information Systems, MPI for Informatics, Max Planck Society, ou_24018              
2External Organizations, ou_persistent22              

Content

show
hide
Free keywords: Computer Science, Learning, cs.LG,Computer Science, Artificial Intelligence, cs.AI
 Abstract: The strong lottery ticket hypothesis holds the promise that pruning randomly
initialized deep neural networks could offer a computationally efficient
alternative to deep learning with stochastic gradient descent. Common parameter
initialization schemes and existence proofs, however, are focused on networks
with zero biases, thus foregoing the potential universal approximation property
of pruning. To fill this gap, we extend multiple initialization schemes and
existence proofs to non-zero biases, including explicit 'looks-linear'
approaches for ReLU activation functions. These do not only enable truly
orthogonal parameter initialization but also reduce potential pruning errors.
In experiments on standard benchmark data sets, we further highlight the
practical benefits of non-zero bias initialization schemes, and present
theoretically inspired extensions for state-of-the-art strong lottery ticket
pruning.

Details

show
hide
Language(s): eng - English
 Dates: 2021-10-212021
 Publication Status: Published online
 Pages: 16 p.
 Publishing info: -
 Table of Contents: -
 Rev. Type: -
 Identifiers: arXiv: 2110.11150
URI: https://arxiv.org/abs/2110.11150
BibTex Citekey: Fischer_arXiv2110.11150
 Degree: -

Event

show

Legal Case

show

Project information

show

Source

show