DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation
Sparse expert execution does not remove the stored states needed to adapt a mixture-of-experts language model. We present DuplexTune, a framework for replaying coupled hidden states through pretrained attention and expert modules.
arXiv cs.LG2 min read