Signal Propagation in Transformers: Theoretical Perspectives and the Role of 
Rank Collapse

Noci, Lorenzo; Anagnostidis, Sotiris; Biggio, Luca; Orvieto, Antonio; Singh, Sidak Pal; Lucchi, Aurelien

Item

ITEM ACTIONSEXPORT

Add to Basket

Local TagsRelease HistoryDetailsSummary

Released

Conference Paper

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse

MPS-Authors

Singh, Sidak Pal
Dept. Empirical Inference, Max Planck Institute for Intelligent Systems, Max Planck Society;
External Organizations;

External Resource

https://proceedings.neurips.cc/paper_files/paper/2022/hash/ae0cba715b60c4052359b3d52a2cff7f-Abstract-Conference.html
(Publisher version)

https://openreview.net/forum?id=FxVH7iToXS
(Any fulltext)

https://doi.org/10.48550/arXiv.2206.03126
(Preprint)

Fulltext (restricted access)

There are currently no full texts shared for your IP range.

Fulltext (public)

There are no public fulltexts stored in PuRe

Supplementary Material (public)

There is no public supplementary material available

Citation

Noci, L., Anagnostidis, S., Biggio, L., Orvieto, A., Singh, S. P., & Lucchi, A. (2022). Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Advances in Neural Information Processing Systems 35 (pp. 27198-27211). Red Hook, NY: Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2022/hash/ae0cba715b60c4052359b3d52a2cff7f-Abstract-Conference.html.

Cite as: https://hdl.handle.net/21.11116/0000-000F-E964-A

Abstract

There is no abstract available