LLM Unpacked: A Deep Dive into Modern AI Architectures

<p>Join us for an insightful exploration into the cutting-edge design of today&#39;s <a href="https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison" target="_blank" rel="noopener noreferer">Large Language Models</a>. Seven years on from the original GPT architecture, have we truly seen groundbreaking changes, or are we simply refining existing foundations? This podcast focuses on the <strong>architectural developments</strong> that define flagship open models in 2025, moving beyond benchmark performance or training algorithms.</p><p>In this episode, we&#39;ll unpack the key ingredients contributing to LLM performance, examining how developers are pushing the boundaries of efficiency, memory management, and training stability. Discover the evolution and intricacies of:</p><ul><li><strong>Attention Mechanisms:</strong> From Multi-Head Attention (MHA) to the more efficient Grouped-Query Attention (GQA), and innovative approaches like Multi-Head Latent Attention (MLA) used in DeepSeek-V3, which compresses key and value tensors for memory savings. We also delve into <strong>Sliding Window Attention</strong> from Gemma 3, which restricts context size for local efficiency.</li><li><strong>Normalization Layers:</strong> Explore the shift from LayerNorm to RMSNorm and the crucial placement of these layers (Pre-Norm, Post-Norm) as seen in OLMo 2 and Gemma 3, including the addition of <strong>QK-Norm</strong> for enhanced training stability.</li><li><strong>Mixture-of-Experts (MoE):</strong> Understand why this approach has seen a significant resurgence in 2025. Learn how MoE, as implemented in models like DeepSeek-V3, Llama 4, and Qwen3&#39;s sparse variants, allows for massive total parameter counts (e.g., DeepSeek-V3&#39;s 671 billion parameters) while activating only a small subset (e.g., 37 billion) per inference step for remarkable efficiency.</li><li><strong>Positional Embeddings:</strong> Discover how positional information is handled, from rotational positional embeddings (RoPE) to the radical concept of <strong>No Positional Embeddings (NoPE)</strong> in SmolLM3, which aims for better length generalization.</li></ul><p>We&#39;ll compare the structural nuances of leading models such as:</p><ul><li><strong>DeepSeek-V3:</strong> A massive 671-billion-parameter model known for MLA and MoE with a shared expert.</li><li><strong>OLMo 2:</strong> Notable for its transparency and specific RMSNorm placements for training stability.</li><li><stro