LLMLingua: Compressing Prompts with Small-Model Token Pruning

This episode examines LLMLingua, a method for compressing prompts five to twenty times by using a small local language model to delete low-information tokens, so that a black-box model such as GPT-3.5 can still answer well. It explains why prompts get so long (chain-of-thought demonstrations, retrieved documents, conversation history). It also explains why quantization, pruning, and learned soft-prompt compression don't work on closed APIs. The discussion ties token surprisal and perplexity to Shannon's redundancy argument and the view of language modeling as compression. It then shows why the Selective-Context baseline fails: it scores every token independently, so a phrase repeated across demonstrations can be deleted entirely. The episode walks through the paper's components, including aligning the small model to the target and a budget controller that barely trims the instruction and question but cuts the demonstrations heavily. Along the way it flags caveats: predictable tokens aren't always unimportant, scoring is left-to-right only, and the alignment step adds under a point on GSM8K.

Sources:

1. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023

http://arxiv.org/abs/2310.05736

2. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (EMNLP 2023)

https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models

3. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (ACL 2024)

https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression

4. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, et al., 2024 (ACL Findings 2024)

https://scholar.google.com/scholar?q=LLMLingua-2%3A+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression

5. Compressing Context to Enhance Inference Efficiency of Large Language Models (Selective Context) — Yucheng Li, Bo Dong, Frank Guerin, Chenghua Lin, 2023 (EMNLP 2023)

https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Ef