Kafka in Production: Failure Scenarios for Staff-Level Interviews

<p>A single misconfigured setting in Kafka doesn&#39;t just slow a page down. It can silently vaporize financial transactions. In this deep dive we walk through real production failure scenarios and the exact reasoning interviewers look for in staff-level system design interviews.</p><p><br></p><p>You&#39;ll learn:</p><p>- Why acks=1 loses data, and the four configuration layers for zero data loss</p><p>- Consistency vs availability: why a healthy cluster refuses writes on purpose</p><p>- Pushing from 400,000 to 2 million events/s with batching, linger.ms and compression</p><p>- How retries reorder a debit and a credit, and how idempotent producers fix it</p><p>- Rebalance storms, cooperative sticky assignment and static membership</p><p>- Hot partitions and the irreducible trade-off between ordering and scale</p><p>- How deleting and recreating a topic silently skips hours of data</p><p>- A 3 AM role-play: 85 under-replicated partitions, a disk at 96%, and why you never restart the broker</p><p>- Exactly-once with Kafka transactions, and exactly-once into PostgreSQL</p><p>- Tiered retry topics and dead letter queues</p><p>- Split brain, KRaft quorums and vanishing tombstones</p><p>- System design at scale: multi-tenant topics, quotas, event-driven sagas and change data capture with Debezium</p><p><br></p><p>Ideal for backend and platform engineers preparing for senior and staff-level interviews.</p><p><br></p><p>Chapters</p><p>00:00 Distributed systems have no X-ray</p><p>01:08 Why Kafka interviews matter</p><p>02:19 acks=1 and the 200 missing payments</p><p>03:27 Four layers of zero data loss</p><p>05:21 The CAP theorem trade-off</p><p>06:31 Throughput: taxis vs buses</p><p>07:42 batch.size, linger.ms and compression</p><p>09:38 When retries reorder messages</p><p>11:38 Idempotent producers</p><p>12:48 Message keys and partition ordering</p><p>13:12 Flash sale: buffer exhausted</p><p>14:22 Decoupling with a local buffer and circuit breaker</p><p>16:18 Rebalance storms</p><p>18:13 Eager vs cooperative sticky rebalancing</p><p>19:25 Static group membership</p><p>20:12 Hot partitions and the hot key problem</p><p>23:18 Silent data loss after recreating a topic</p><p>25:20 auto.offset.reset and topic versioning</p><p>26:08 3 AM: 85 under-replicated partitions</p><p>28:54 Why you never restart the broker</p><p>29:42 Disk at 96%: the 15-minute response</p><p>31:36 Throttled reassignment, Cruise Control, tiered storage</p><p>33:36 Duplicates in consume-transfor