Attention Sinks Enable Streaming Language Models Over Endless Input

This episode examines "Efficient Streaming Language Models with Attention Sinks" (ICLR 2024), which asks how a model trained on a fixed attention window can run over an endless input stream at constant memory and constant per-token cost, with no fine-tuning. It explains why plain window attention collapses once the first tokens are evicted from the KV cache (perplexity in the thousands), and why recomputing every window works but is slow. The key argument is that softmax forces attention to land somewhere, so models learn to park surplus attention on the initial tokens, which act as "sinks". On Llama-2-13B, keeping just the first four tokens plus a rolling window brings perplexity from about 5,158 down to 5.40, close to the 5.43 of full recomputation. Replacing those four tokens with newlines still works, which shows that absolute position matters and content doesn't. The discussion also stresses that the method delivers stable generation, not long-term memory, and it ends on how positions are assigned by cache slot rather than by original text position.

Sources:

1. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

http://arxiv.org/abs/2309.17453

2. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023 (ICLR 2024)

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

3. Massive Activations in Large Language Models — Mingjie Sun, Xinlei Chen, J. Zico Kolter, Zhuang Liu, 2024

https://scholar.google.com/scholar?q=Massive+Activations+in+Large+Language+Models

4. Vision Transformers Need Registers — Timothée Darcet, Maxime Oquab, Julien Mairal, Piotr Bojanowski, 2023

https://scholar.google.com/scholar?q=Vision+Transformers+Need+Registers

5. Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing — Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort, 2023

https://scholar.google.com/scholar?q=Quantizable+Transformers%3A+Removing+Outliers+by+Helping+Attention+Heads+Do+Nothing