84: Trust But Canary: Configuration Safety at Scale

Have you ever wondered how Meta makes config rollouts safe at scale? In this episode, Pascal sits down with Ishwari and Joe to discuss Meta's approach for propagating changes across services in seconds and discuss why speed increases the need for strong safeguards. Catch the episode to discover canarying and progressive rollouts, the health checks and monitoring signals used to catch regressions early, and how incident reviews focus on improving systems rather than blaming people. We also hear how data and early AI/ML are slashing alert noise and speeding up bisecting when something goes wrong.

Got feedback? Send it to us on Threads (https://threads.net/@metatechpod), Instagram (https://instagram.com/metatechpod) and don't forget to follow our host Pascal (https://mastodon.social/@passy, https://threads.net/@passy_). Fancy working with us? Check out https://www.metacareers.com/.

Links

FFmpeg at Meta: Media Processing at Scale - https://engineering.fb.com/2026/03/02/video-engineering/ffmpeg-at-meta-media-processing-at-scale/

Reliably Changing Configuration @ Scale - https://atscaleconference.com/reliably-changing-configuration-scale/

Timestamps

Intro 0:06

Introduction and Overview of Configuration Changes 2:31

Understanding Configurations in Distributed Systems 4:02

Meta's Configuration Management Systems 6:43

Safeguards and Incident Prevention 9:22

Deployment Mechanisms: Canary and Progressive Rollouts 12:06