Opening up Attention: A QK Transcoder Dashboard for Feature Pairs
Multi Head Attention in Transformers continues to be an elusive topic in mechanistic interpretability. One way of trying to decompose MHA into interpretable parts is applying SDL (Sparse Dictionary Learning) methods to the attention mechanism.
LessWrong12 min read