STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules
Sparse computation can scale the parametric capacity of LLMs without proportionally increasing active computation, but pretraining sparse architectures remains expensive and data-hungry.
arXiv cs.LG2 min read