Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass
Variable-length training creates conflicting parallelism requirements across the layers of a hybrid-attention model. The longest samples require wide sequence-parallel (SP) groups of 16 or 32 ranks for full attention, whose cost grows quadratically with sequence length.
By Jiahao Zhang, Junwei Feng, Dongliang Huang, Xiaoling Zhou, Yiqiao Liao, Chengru SongarXiv cs.LG
Read the full story
Originally published by arXiv cs.LG on Oct 28, 2026.
Comments
Sign in to join the discussion.