Language Models Learn to Fetch Only What They Need

This episode examines "Language Models Can Control Their Own Attention" from KAIST AI, which tackles the memory bottleneck of long-context inference: at a million tokens, generating each token requires hauling roughly 15 gigabytes of key-value cache through memory, comparable to reloading the entire model's active parameters. The discussion traces prior fixes—StreamingLLM's attention-sink heuristic, H2O's cumulative attention scoring, and Quest's query-aware page selection—all grouped as "extrinsic scoring" methods that still require a full pass over context statistics before discarding anything. The paper's proposed alternative, Declarative Attention, repurposes chain-of-thought so the model states in its own reasoning which parts of the context it needs, letting the inference engine skip loading the rest instead of relying on an external scorer. The hosts debate whether self-reported relevance is trustworthy compared to an independent extrinsic estimate, since errors here directly create blind spots in what the model can see rather than showing up as recoverable noise. Listeners interested in long-context efficiency, KV-cache management, or the mechanics behind sparse attention will find the back-and-forth over whether this approach is elegant or quietly risky especially engaging.

Sources:

1. Language Models Can Control Their Own Attention — Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos, 2026

http://arxiv.org/abs/2609.02737

2. Generating Long Sequences with Sparse Transformers — Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever, 2019

https://scholar.google.com/scholar?q=Generating+Long+Sequences+with+Sparse+Transformers

3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

4. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023/2024

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024

https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference