LLM Unpacked: A Deep Dive into Modern AI Architectures

<p>Join us for an insightful exploration into the cutting-edge design of today&#39;s <a href="https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison" target="_blank" rel="noopener noreferer">Large Language Models</a>. Seven years on from the original GPT architecture, have we truly seen groundbreaking changes, or are we simply refining existing foundations? This podcast focuses on the <strong>architectural developments</strong> that define flagship open models in 2025, moving beyond benchmark performance or training algorithms.</p><p>In this episode, we&#39;ll unpack the key ingredients contributing to LLM performance, examining how developers are pushing the boundaries of efficiency, memory management, and training stability. Discover the evolution and intricacies of:</p><ul><li><strong>Attention Mechanisms:</strong> From Multi-Head Attention (MHA) to the more efficient Grouped-Query Attention (GQA), and innovative approaches like Multi-Head Latent Attention (MLA) used in DeepSeek-V3, which compresses key and value tensors for memory savings. We also delve into <strong>Sliding Window Attention</strong> from Gemma 3, which restricts context size for local efficiency.</li><li><strong>Normalization Layers:</strong> Explore the shift from LayerNorm to RMSNorm and the crucial placement of these layers (Pre-Norm, Post-Norm) as seen in OLMo 2 and Gemma 3, including the addition of <strong>QK-Norm</strong> for enhanced training stability.</li><li><strong>Mixture-of-Experts (MoE):</strong> Understand why this approach has seen a significant resurgence in 2025. Learn how MoE, as implemented in models like DeepSeek-V3, Llama 4, and Qwen3&#39;s sparse variants, allows for massive total parameter counts (e.g., DeepSeek-V3&#39;s 671 billion parameters) while activating only a small subset (e.g., 37 billion) per inference step for remarkable efficiency.</li><li><strong>Positional Embeddings:</strong> Discover how positional information is handled, from rotational positional embeddings (RoPE) to the radical concept of <strong>No Positional Embeddings (NoPE)</strong> in SmolLM3, which aims for better length generalization.</li></ul><p>We&#39;ll compare the structural nuances of leading models such as:</p><ul><li><strong>DeepSeek-V3:</strong> A massive 671-billion-parameter model known for MLA and MoE with a shared expert.</li><li><strong>OLMo 2:</strong> Notable for its transparency and specific RMSNorm placements for training stability.</li><li><strong>Gemma 3 &amp; 3n:</strong> Featuring sliding window attention for KV cache memory savings and unique normalization layer placements; Gemma 3n also introduces Per-Layer Embedding and MatFormer concepts.</li><li><strong>Mistral Small 3.1:</strong> Prioritizing lower inference latency through custom tokenizers and specific architectural choices.</li><li><strong>Llama 4:</strong> Adopting an MoE approach similar to DeepSeek-V3 but with its own distinct expert configuration.</li><li><strong>Qwen3:</strong> Available in both dense and MoE variants, offering flexibility for various use cases and moving away from shared experts in some MoE configurations.</li><li><strong>SmolLM3:</strong> A compact 3-billion-parameter model exploring the effectiveness of NoPE.</li><li><strong>Kimi K2:</strong> An impressive 1 trillion parameter model, building on the DeepSeek-V3 architecture with more experts and fewer MLA heads, setting new standards for open-weight performance.</li></ul><p><br></p><p>Tune in to understand the intricate design decisions driving the next generation of large language models.</p>