Kubernetes Deep Dive: Pod Startup and Failure Modes

<p>&quot;Walk me through what happens from kubectl apply to the first request hitting your container.&quot; It sounds like a warm-up question, but how you answer it can decide whether you land a senior or staff role. In this deep dive we trace a pod from creation to production and unpack the failure modes interviewers expect you to anticipate.</p><p><br></p><p>You&#39;ll learn:</p><p>- The full startup chain: API server, etcd, controllers, scheduler, kubelet and container runtime</p><p>- Why init containers can&#39;t run in parallel, and why migrations belong in one for security</p><p>- Startup, liveness and readiness probes, and the liveness trap that turns a 10-second database blip into an outage</p><p>- Native sidecars in Kubernetes 1.29 (KEP-753) and the startup and shutdown races they fix</p><p>- HashiCorp Vault: mutating webhook injection, TokenReview and system:auth-delegator, and dynamic database credentials</p><p>- Zero trust with mTLS, SPIFFE identities and JWTs, plus the alg: none attack and audience checks</p><p>- Service DNS, ndots:5, ClusterIP &quot;phantom&quot; IPs, iptables vs IPVS, and headless services for gRPC</p><p>- Istio traps: DestinationRule ordering, empty AuthorizationPolicies and a stray YAML hyphen</p><p>- Helm secrets, migration hooks, and a five-phase rollout for breaking API changes</p><p>- A layered runbook for debugging intermittent 503s</p><p><br></p><p>Chapters</p><p>00:00 Introduction</p><p>00:22 What interviewers are really listening for</p><p>01:31 From kubectl apply to &quot;success&quot;</p><p>03:06 The scheduler and the kubelet</p><p>04:18 Init containers</p><p>05:26 Why migrations belong in an init container</p><p>06:34 Startup, liveness and readiness probes</p><p>08:09 The liveness probe thundering herd trap</p><p>10:04 What a liveness probe should check</p><p>10:53 Startup probes for slow-booting apps</p><p>11:40 The sidecar startup and shutdown race</p><p>12:52 Native sidecars in Kubernetes 1.29</p><p>13:39 Vault injection with a mutating webhook</p><p>14:48 Vault&#39;s init and sidecar agents</p><p>15:34 Vault authentication and TokenReview</p><p>16:43 Dynamic database credentials</p><p>17:55 A Postgres quoting trap</p><p>18:44 Zero trust and mutual TLS</p><p>19:52 SPIFFE identities in Istio policies</p><p>20:17 mTLS vs JWT</p><p>20:39 Istio policy traps</p><p>21:46 The alg: none attack and audience checks</p><p>23:20 Service DNS and the phantom ClusterIP</p><p>24:31 iptables vs IPVS and headless services</p><