LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning

This episode examines LongAttnComp, a method that uses a small trained scorer to pick about 16,000 relevant tokens out of roughly 115,000. The goal is for other companies' models to find code bugs about as well as with the full input. It explains why long prompts are costly (prefill dominates time-to-first-token) and how extractive compression differs from summarization. It also ties the approach to Speculative Prefill, AttnComp, and SnapKV, which score chunks by query-aligned attention, and frames compression as the retrieval half of a retrieve-then-read pipeline. The method freezes the first 13 layers of Llama-3.1-8B-Instruct, adds one trainable cross-attention layer, and trains it on synthetic needle-in-a-haystack data. It keeps the top-scoring chunks under a token budget and sends them to the target model in their original order. Throughout, the discussion stays skeptical of the claim that compression "exceeds" full context. The baseline is middle truncation, which cuts the region where models already read poorly. The paper also never measures latency itself, and the training data is synthetic, so transfer to real inputs is open.

Sources:

1. LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning — Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu, 2026

http://arxiv.org/abs/2606.01336

2. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023

https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models

3. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (ACL 2024)

https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression

4. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation — Jingyu Liu, Beidi Chen, Ce Zhang, 2025

https://scholar.google.com/scholar?q=Speculative+Prefill%3A+Turbocharging+TTFT+with+Lightweight+and+Training-Free+Token+Importance+Estimation

5. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation — Fangyuan Xu, Weijia Shi, Eunsol Choi, 2023

https://scholar.google.com/scholar?q=RECOMP%3A+Improving+Retrieval-Augmented+LMs+with+Compression+and+Sele