Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction
The corpus-mean direction of a transformer's residual stream is a component shared across all inputs, and is commonly removed by mean-centering before representational analysis.
arXiv - cs.CL (NLP)3 min read