In-Context Autoencoder: Compressing Long Contexts into Memory Slots

This episode examines "In-Context Autoencoder for Context Compression in a Large Language Model" (ICLR 2024), which uses a LoRA-adapted copy of an LLM to squeeze a context into 128 learned memory slots that an unmodified, frozen decoder reads in place of the original tokens, for roughly 4x compression. It covers the two-stage training: self-supervised pretraining on the Pile with autoencoding and language-modeling objectives, then instruction tuning on a GPT-4-generated prompt-and-answer dataset. The discussion contrasts this design with Gist tokens and AutoCompressors, and with architectural fixes for long context such as sparse attention and state-space models. It also draws on the "Lost in the Middle" findings as motivation. The hosts question whether the paper's evidence supports its "long context" framing, since compressing a single passage is a different claim from handling truly long inputs. They also flag that the synthetic training data and the GPT-4 judge deserve scrutiny. Listeners get a clear explanation of how soft-prompt-style memory slots work and a critical lens on what the results do and don't show.

Sources:

1. In-context Autoencoder for Context Compression in a Large Language Model — Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, Furu Wei, 2023

http://arxiv.org/abs/2307.06945

2. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

3. Adapting Language Models to Compress Contexts — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023

https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts

4. Compressive Transformers for Long-Range Sequence Modelling — Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Timothy P. Lillicrap, 2019

https://scholar.google.com/scholar?q=Compressive+Transformers+for+Long-Range+Sequence+Modelling

5. Perceiver: General Perception with Iterative Attention — Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira, 2021

https://scholar.google.com/scholar?q=Perceiver%3A+General+Perception+with+Iterative+Attention