Speculative Prefill: Training-Free Token Importance for Faster TTFT

This episode examines "Speculative Prefill," an ICML 2025 method that cuts time to first token by having a small model (Llama-3.1-8B) estimate which prompt tokens matter, so a large model (Llama-405B) prefills only those. It explains why prefill dominates in real traffic, where prompts are often about ten times longer than outputs. It also explains why dropping tokens differs from sparse-attention approaches like MInference, since it also removes MLP and tensor-parallel communication costs. The discussion places the work among related approaches (SnapKV, LazyLLM, GemFilter, SwiftKV, LLMLingua) as a training-free method that doesn't rely on the big model to pick tokens. It also spells out the two bets behind it: that token importance transfers between models, and that the big model tolerates a gappy prompt at a given keep rate. The episode cautions that the headline 7.66x TTFT and 7x QPS figures come from different experiments and best-case conditions. It also notes that the importance score is an attention heuristic rather than a formal information-theoretic measure.

Sources:

1. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation — Jingyu Liu, Beidi Chen, Ce Zhang, 2025

http://arxiv.org/abs/2502.02789

2. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, et al., 2024

https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation

3. LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference — Qichen Fu et al., 2024

https://scholar.google.com/scholar?q=LazyLLM%3A+Dynamic+Token+Pruning+for+Efficient+Long+Context+LLM+Inference

4. Not All Heads Matter / Quest, and RetrievalAttention-style dynamic sparse attention — Jiaming Tang et al. (Quest); Di Liu et al. (RetrievalAttention), 2024

https://scholar.google.com/scholar?q=Not+All+Heads+Matter+%2F+Quest%2C+and+RetrievalAttention-style+dynamic+sparse+attention

5. GemFilter: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction — Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty, 2024

https://scholar.google.com/scholar?q=GemFilter%3A+Discovering+the+Gems+in+Early+Layers%3A+Accelerating+Long-Context+LLMs+with+1000x+Input+Token+Reduction