Scope the requirements
The trap here is treating it as a scaling question. The volume is tiny by design; the interviewer is testing whether you can make money movement correct across services that fail, retry, and deliver webhooks out of order. Scope it to a marketplace: buyers pay in, sellers get paid out, and a PSP handles the cards.
Functional — what it must do
- Pay-in: a buyer pays for a checkout; funds route to the platform via a PSP (Stripe/Adyen) — we never touch raw card numbers, only PSP tokens.
- Pay-out: settle seller balances to their bank accounts on a schedule.
- Every money movement is recorded in a double-entry ledger — the auditable source of truth.
- Merchants can query payment status and receive webhooks on state changes.
- Nightly reconciliation against the PSP's settlement file, with mismatches routed to finance ops.
- Explicitly cut: fraud scoring internals, FX and multi-currency, chargebacks — I name them as ledger extensions, not v1.
Non-functional — the numbers that shape the design
- Correctness over throughput: effectively exactly-once processing — no double charge, no lost payment, ever.
- Scale is modest: 1M transactions/day ≈ 10 TPS average — the difficulty is consistency, not QPS.
- 99.99% availability on payment acceptance; a down payment system is lost revenue by the minute.
- Full auditability: every state transition and money movement is reconstructable, retained for years.
- PCI-DSS by delegation: card data lives at the PSP; we store tokens only.
I want to say the quiet part first: at 10 TPS this is not a scaling problem. The whole interview is about what happens when a charge times out, a webhook arrives twice, or a retry fires — so I'll spend my time on idempotency, the ledger, and reconciliation.
Back-of-envelope estimates
The arithmetic takes one minute and its job is to kill the scaling conversation early: one Postgres handles this comfortably, which frees the whole interview for correctness.
| Quantity | Estimate | How you got there |
|---|---|---|
| Transactions | ~10/s | 1M/day ÷ 86,400 s; peak ~10× at ~100/s. Any single relational DB shrugs at this. |
| Ledger writes | ~30/s | each transaction produces 2+ ledger entries (debit + credit, plus fees) — still trivial. |
| Ledger storage | ~150 GB/yr | ~2M entries/day × ~200 B ≈ 400 MB/day, append-only, retained forever for audit. |
| Idempotency keys | ~1M/day, ~100 MB | UUID + cached response per attempt; keep 7 days then archive — a rounding error. |
| Reconciliation batch | 1M records/night | settlement file vs ledger diff; a single-node batch job finishes in minutes. |
| Money representation | integer minor units | cents as integers (or exact decimal) with currency alongside — never float. |
Everything fits in one Postgres for years, so I get ACID transactions where money is involved — and that's a feature I'm choosing deliberately, not a limitation I'm tolerating.
Sketch the API
A small surface, and one header carries most of the design: the idempotency key. The other interesting decision is that the synchronous response is never the final word — the webhook is.
- The Idempotency-Key contract: the client generates a UUID per payment attempt and reuses it on every retry. Server-side it's a unique constraint — first request processes and stores the response, replays return the stored response without re-executing. A double-clicked "Pay" button produces exactly one charge, by construction.
- Status is an enum, never booleans:
pending → executing → succeeded | failedwith explicit transitions. Ais_paidflag can't represent "we don't know yet", and "we don't know yet" is the state this system lives in.
Data model
Four tables carry the whole design, and one of them — the ledger — is where the interviewer decides whether you have financial-domain sense.
Postgres, without hesitation — money wants ACID, and 10 TPS needs nothing exotic. I'd say explicitly that NoSQL is the wrong default here: I'm trading effortless horizontal scale I don't need for transactions I absolutely do. If volume grew 100×, I'd shard by merchant before I'd give up transactional writes to the ledger.
High-level design
A payment service that owns the state machine, an executor that talks to the PSP, and wallet/ledger updates decoupled behind Kafka so every hop is independently retryable. Reconciliation sits outside the flow as the safety net.
- Buyer submits payment → gateway → payment service checks the idempotency key: seen before, return the stored response; new, persist the
payment_orderrows inpendingand hand off to the executor. - Executor calls the PSP with a PSP-side nonce (so even our retry to them can't double-charge) and records the synchronous response — but treats it as provisional.
- The PSP webhook arrives with the authoritative outcome. The handler is idempotent: it checks the current state, applies the transition once, and ignores duplicates and stale reorderings.
- On success the payment service emits a
paidevent to Kafka; the wallet/ledger consumer appends the double-entry ledger rows and updates the merchant balance in one local transaction, then marks the order settled. Failures retry with backoff; poison events go to a dead-letter queue. - Nightly, reconciliation diffs the PSP settlement file against the ledger. Matches confirm; mismatches become adjustment tickets for finance ops — the system's last line of defence.
Deep dives — where the interview is won
Exactly-once is a lie you construct from at-least-once plus idempotency
No network gives you exactly-once delivery, so the system fakes it: retry everything (at-least-once) and make every operation safe to repeat (idempotent). The dedupe lives at every hop, not one place. Client → API: the Idempotency-Key header hits a unique constraint; the first insert wins, replays get the cached response. API → PSP: the PSP accepts a nonce per charge attempt, so our timeout-and-retry can't create two charges. Kafka consumers: dedupe on event id before applying, because Kafka will redeliver.
The scenario the interviewer will run is the nasty one: we call the PSP and the connection drops after they charged but before we heard. The wrong answer is to retry the charge blind. The right answer is that our state is executing, and from executing we never re-charge — we query the PSP's status API with our nonce, learn the truth, and transition accordingly. Timeout means unknown, and unknown means ask, not retry.
The ledger is append-only double-entry, and balances are derived from it
Every money movement writes two entries: a debit from one account and a credit to another, and the entries for any transaction sum to zero. Buyer pays $10: debit buyer $10, credit seller $10 (in practice a third pair for our fee). Nothing is ever updated or deleted — a refund is a new pair of entries in the opposite direction, never a mutation of history. That immutability is what makes the system auditable: the ledger is a fact log, and any balance is a fold over it.
The merchant wallet balance is therefore a cache, not a source of truth — maintained incrementally for fast reads, re-derivable from the ledger at any time, and periodically re-derived as an invariant check. This is also the answer to "how would you detect a bug that double-pays?": the sum-to-zero invariant breaks, or the derived balance diverges from the cached one, and the nightly job catches it even when no human noticed.
No 2PC — persist-then-emit, state machines, and reconciliation as the safety net
Payment, wallet, and ledger updates span services, and the reflex answer — a distributed transaction — is the one I explicitly decline. 2PC couples the availability of every participant, blocks on coordinator failure, and PSPs don't speak it anyway. Instead each hop persists its own state, then emits an event; consumers retry until success. It's a saga in the degenerate, forward-only form: money movements don't roll back, they compensate with new entries.
The glue is explicit state machines: payment_order.status walks pending → executing → succeeded/failed, and each transition is guarded so a duplicate or out-of-order event can't move it backwards. And because some inconsistency will still slip through — a webhook lost for a day, a consumer bug — reconciliation is the design's admission of humility: every night, diff what the PSP says settled against what our ledger says happened, auto-heal the classifiable mismatches, and queue the rest for finance ops.
The PSP's synchronous response is provisional; the webhook is the truth
Card networks are asynchronous underneath, so the PSP's HTTP response to our charge call can be incomplete — 3DS challenges, pending review, network partitions. The design treats the sync response as an optimistic hint and the signed webhook as the authoritative outcome. The UI shows "processing" until the webhook lands; merchants who need certainty poll GET /payments/:id.
Webhook handling is its own minefield and interviewers know it: webhooks arrive duplicated, out of order, or not at all. Handlers must be idempotent (apply-once via the state machine), must verify the signature (anyone can POST to a public URL), and must never process a succeeded after a terminal failed without flagging it. For the not-at-all case, a sweeper polls the PSP status API for orders stuck in executing beyond a threshold.
Follow-ups you should expect
A user double-clicks the pay button. What exactly prevents two charges?
The PSP call times out. Do you retry?
executing, and a sweeper queries the PSP's status API using our nonce to learn the real outcome, then applies the transition. Retrying the charge is only safe because of the nonce, but querying first is cheaper and kinder to the buyer's statement.How would you find out the system double-paid someone?
Why not DynamoDB or Cassandra? They're more available.
How do refunds and chargebacks fit?
Where candidates lose the room
- Using floats for money — binary floating point cannot represent 0.10, and the interviewer is listening for integer minor units within the first few minutes.
- Retrying a timed-out charge without an idempotency key — this is the double-charge bug, and it's the scenario most interviewers run explicitly.
- Proposing 2PC across payment, wallet, and ledger services — it signals pattern-matching "consistency" to "distributed transaction" instead of persist-then-emit plus reconciliation.
- Treating the PSP's synchronous response as final and having no webhook or reconciliation story — the design silently loses payments on every partition.
- Spending ten minutes sharding for scale — at 10 TPS, every minute on throughput is a minute not spent on the correctness problems being scored.