Product Keys Give Transformers Million-Slot Memory Layers

This episode examines "Large Memory Layers with Product Keys," which replaces a transformer's feed-forward network with a sparse memory of over a million trainable slots. The layer's central trick is splitting each query in half and scoring it against two small codebooks of sub-keys. This avoids scoring every key and cuts search cost by roughly a thousandfold. The hosts work through a toy example and the argument for why taking the top-k of each half still recovers the exact top-k. They also cover the motivation for training on the 28-billion-word CC-News corpus rather than One Billion Word, which overfits. Throughout, they stay skeptical of the headline claim that a 12-layer model with memory beats a 24-layer one at nearly twice the speed. They note that the paper doesn't test directly against approximate-nearest-neighbor memory or mixture-of-experts baselines, which makes it a useful listen for anyone wondering what "free capacity" actually costs.

Sources:

1. Product Keys Give Transformers Million-Slot Memory Layers

https://proceedings.neurips.cc/paper_files/paper/2019/file/9d8df73a3cfbf3c5b47bc9b50f214aff-Paper.pdf

2. Large Memory Layers with Product Keys — Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou, 2019

https://scholar.google.com/scholar?q=Large+Memory+Layers+with+Product+Keys

3. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011

https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search

4. Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes — Jack W. Rae, Jonathan J. Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, Timothy Lillicrap, 2016

https://scholar.google.com/scholar?q=Scaling+Memory-Augmented+Neural+Networks+with+Sparse+Reads+and+Writes

5. Memory Layers at Scale — Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Ghosh, 2024

https://scholar.google.com/scholar?q=Memory+Layers+at+Scale