System Design Interview: Payment Settlement Batch Processing

<p>Design a batch processing system for end-of-day payment settlement at a payments company that processes <strong>50 million transactions per day</strong>. The system must net merchant positions, calculate fees, and initiate fund transfers to merchant bank accounts within a strict bank cutoff window. Walk me through your design, covering reliability, scalability, and how you&#39;d handle failures.</p><p><br></p><p>Key Takeaways</p><ol><li>The <strong>outbox pattern</strong> is the canonical solution to the dual-write problem. Know it cold.</li><li><strong>Partition keys define ordering and parallelism</strong> — choose with intention, usually around the natural aggregation boundary (here, merchant ID).</li><li><strong>Throttling must be distributed</strong> when you have multiple workers — Redis-backed buckets or a sidecar.</li><li><strong>Idempotency is non-negotiable</strong> in payments. Design for at-least-once and dedupe.</li><li><strong>Retries are tiered</strong>: in-process for transient, delay-queue for slower-resolving, DLQ for terminal.</li><li><strong>Backpressure beats dropping</strong> — use Kafka lag as your buffer when downstream is slow.</li><li><strong>Reconciliation closes the loop</strong> — you don&#39;t know it worked until ground truth confirms.</li><li><strong>Corrections are new events</strong> — never rewrite history in a financial system.</li></ol>