Memory Layers at Scale: Trainable Key-Value Lookup for Facts

This episode examines "Memory Layers at Scale" from Meta FAIR, which asks whether a language model can gain huge numbers of fact-storing parameters without adding any arithmetic per token. It explains how a memory layer works: a token's query retrieves the top-k rows from a large learned key-value table, and product keys keep the search cheap. It also covers where the idea came from, including the 2019 product-key work, which this paper scales up to billions of parameters with custom GPU kernels, and the related million-expert approach. The discussion examines what "matched FLOPs" leaves out, namely stored parameters and memory-bandwidth and communication costs. It also looks at how closed-book question answering on NaturalQuestions and TriviaQA is used to test the factual-recall claims against dense and mixture-of-experts baselines. Listeners interested in how models store facts, and in the gap between FLOP counts and real hardware cost, will find it useful.

Sources:

1. Memory Layers at Scale — Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Ghosh, 2024

http://arxiv.org/abs/2412.09764

2. Mixture of A Million Experts (PEER) — Xu Owen He, 2024

https://scholar.google.com/scholar?q=Mixture+of+A+Million+Experts+%28PEER%29

3. Large Memory Layers with Product Keys — Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou, 2019

https://scholar.google.com/scholar?q=Large+Memory+Layers+with+Product+Keys

4. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai et al., 2024

https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models

5. Scaling Laws for Fine-Grained Mixture of Experts — Jan Ludziejewski et al., 2024

https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts