AttnComp: Attention-Guided Adaptive Context Compression for RAG

This episode examines AttnComp, a method that uses a language model's own middle-layer attention, lightly fine-tuned, to compress retrieved documents in retrieval-augmented generation. It covers why large retrieved contexts hurt: noisy passages can push accuracy below no retrieval, key facts get lost in the middle, and every extra token adds cost. It contrasts slow abstractive compressors with extractive ones that use fixed budgets and score each sentence alone, which misses bridge documents in multi-hop questions. The discussion explains how attention acts as a shared budget, so document scores compete jointly, and how a nucleus-sampling-style rule keeps as many documents as the attention requires, including none. The attention share on the instruction and first token serves as a signal for how relevant the context is. The episode also flags caveats, including attention sinks and the warning that attention is not always an explanation.

Sources:

1. AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation — Lvzhou Luo, Yixuan Cao, Ping Luo, 2025

http://arxiv.org/abs/2509.17486

2. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation — Fangyuan Xu, Weijia Shi, Eunsol Choi, 2023

https://scholar.google.com/scholar?q=RECOMP%3A+Improving+Retrieval-Augmented+LMs+with+Compression+and+Selective+Augmentation

3. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (and LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression) — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu (Microsoft), 2023

https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models+%28and+LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression%29

4. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

5. Making Retrieval-Augmented Language Models Robust to Irrelevant Context — Ori Yoran, Tomer Wolfson, Ori Ram, Jonathan Berant, 2024 (ICLR)

https://scholar.google.com/scholar?q=Making+Retrieval-Augmented+Language+Models+Robust+to+Irrelevant+Context