English
 
Help Privacy Policy Disclaimer
  Advanced SearchBrowse

Item

ITEM ACTIONSEXPORT

Released

Paper

Plant 'n' Seek: Can You Find the Winning Ticket?

MPS-Authors
/persons/resource/persons229482

Fischer,  Jonas
Databases and Information Systems, MPI for Informatics, Max Planck Society;

External Resource
No external resources are shared
Fulltext (restricted access)
There are currently no full texts shared for your IP range.
Fulltext (public)

arXiv:2111.11153.pdf
(Preprint), 912KB

Supplementary Material (public)
There is no public supplementary material available
Citation

Fischer, J., & Burkholz, R. (2021). Plant 'n' Seek: Can You Find the Winning Ticket? Retrieved from https://arxiv.org/abs/2111.11153.


Cite as: https://hdl.handle.net/21.11116/0000-0009-B124-6
Abstract
The lottery ticket hypothesis has sparked the rapid development of pruning
algorithms that perform structure learning by identifying a sparse subnetwork
of a large randomly initialized neural network. The existence of such 'winning
tickets' has been proven theoretically but at suboptimal sparsity levels.
Contemporary pruning algorithms have furthermore been struggling to identify
sparse lottery tickets for complex learning tasks. Is this suboptimal sparsity
merely an artifact of existence proofs and algorithms or a general limitation
of the pruning approach? And, if very sparse tickets exist, are current
algorithms able to find them or are further improvements needed to achieve
effective network compression? To answer these questions systematically, we
derive a framework to plant and hide target architectures within large randomly
initialized neural networks. For three common challenges in machine learning,
we hand-craft extremely sparse network topologies, plant them in large neural
networks, and evaluate state-of-the-art lottery ticket pruning methods. We find
that current limitations of pruning algorithms to identify extremely sparse
tickets are likely of algorithmic rather than fundamental nature and anticipate
that our planting framework will facilitate future developments of efficient
pruning algorithms, as we have addressed the issue of missing baselines in the
field raised by Frankle et al.