Simple beats complicated: We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training. Huge thanks to my collaborators Rhea Sukthanker, Pashmina Cameron, and Emy Gervais. Paper: arxiv.org/abs/2608.28444
Sliding-window beats linear attention
Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and val...
arxiv.org
