CoWindow and MassAlloc Attention: collective causal coverage and distribution-adaptive compute [R]
I'm one of the authors of two recent papers exploring different sources of redundant computation in attention. I'd like to share the ideas and hear feedback from people working on long-context models and attention kernels.
Reddit - r/ML2 min read