Scope the requirements
Two consumers with wildly different needs: an advertiser dashboard wanting fresh numbers within seconds, and a billing system wanting an audited, immutable, exactly-once-per-day count. Serve both from the same event stream.
Functional — what it must do
- Ingest click events at 100k+/s peak.
- Real-time counts per (adId, dimension) with < 1-minute lag.
- Daily reconciled totals for billing, produced from durable event log.
- Query API for advertiser dashboards: aggregations over ad, campaign, time bucket.
- Explicitly cut: attribution modelling (last-click, multi-touch), fraud scoring — offer them back.
Non-functional — the numbers that shape the design
- Ingest QPS: 100k/s sustained, 500k/s peak.
- Real-time lag under 30 s from click to dashboard.
- Exactly-once for billing — no double-counted click ever appears in daily totals.
- Durability: never lose a click event once acknowledged.
- Query latency: dashboard reads under 200 ms p95.
I'll build one durable log and two views on top: a fast, approximate stream for the dashboard and a slow, exact batch for billing. They don't share code paths, and that's the point.
Back-of-envelope estimates
Ingest and storage are big; the read path is comparatively tiny. The design is entirely a write-path problem.
| Quantity | Estimate | How you got there |
|---|---|---|
| Events/s | ~100k avg, 500k peak | peak during major campaign launches or events like the Super Bowl. |
| Events/day | ~10B | 100k/s × 86,400. Retained forever in cold storage for audit. |
| Storage/day | ~1 TB | ~100 B per event (adId, userId, ts, geo, device, page). Compressed columnar in the warehouse: ~200 GB. |
| Distinct ads | ~10M | the granularity at which we aggregate. Wider than most fact tables want to be. |
| Time buckets | 1 min / 1 hour / 1 day | three rollup levels; queries pick the coarsest one that answers. |
| Dashboard QPS | ~1k/s | advertisers refreshing dashboards. Hits pre-aggregated tables, not raw. |
10 billion events a day is the storage number; 100k a second is the ingest number. Neither should touch Postgres.
Sketch the API
Two APIs: a fire-and-forget ingest endpoint that lives on a CDN edge, and a query endpoint for dashboards.
- Ingest is fire-and-forget over UDP-friendly HTTPS at the CDN edge; the client doesn't wait for downstream durability. A local edge write to a durable queue is enough acknowledgement.
- Every click carries a client-generated
clickIdfor deduplication — retries from the browser don't produce two events downstream.
Data model
Three storage tiers with different jobs: the raw event log (source of truth), real-time rollups (fast dashboard), reconciled batch rollups (billing).
Kafka is the log. S3 is the archive. Cassandra (or Druid/Pinot for OLAP shape) holds the real-time rollups. Snowflake or BigQuery holds the daily reconciled rollups. Never merge these — they answer different questions and picking one destroys the other's guarantees.
High-level design
Two independent pipelines fed by the same log. The stream side optimises for latency and eventual consistency; the batch side optimises for exactness and auditability. The dashboard reads the stream side for freshness; billing reads the batch side and only the batch side.
- Browser fires beacon → edge PoP → ingest service. Ingest dedupes on clickId (Redis TTL 24h) and publishes to Kafka. Latency budget: ~10 ms to ack.
- Kafka fans out to two consumers: a Flink stream job that maintains per-minute aggregations in real time, and an S3 sink that lands raw events as Parquet files by hour partition.
- Real-time rollups: Flink tumbling window of 1 minute per (adId, dimension), writes to Cassandra. Dashboard queries hit this and return in ~10 ms.
- Batch pipeline: nightly job reads yesterday's Parquet from S3, dedupes at the file level (in case of pipeline retries), computes daily totals, writes to warehouse. Billing invoices read only these numbers.
- The two views can disagree by a few percent within the day (real-time hasn't yet processed late-arriving events); by end of day, batch is the reconciled source of truth, and any dashboard drift beyond a tolerance triggers an ops alert.
Deep dives — where the interview is won
The Lambda pattern in plain speech: fast + slow on the same log
The classical name is Lambda architecture: a real-time (speed) layer and a batch layer over the same event source. Modern practice sometimes replaces it with Kappa (streaming only), but for billing you almost always want the batch anchor because reprocessing streaming state after a bug is painful. The interview move: name the trade-off, don't dogmatically pick one. "I have a hard billing SLA and a soft dashboard SLA — those want different guarantees."
The joins-in-Flink problem is where Kappa struggles. If dashboards need to enrich clicks with campaign metadata (which changes), the streaming join has to handle out-of-order updates. Batch does it with a straight SQL join. Modern systems compromise: stream for approximate freshness, batch daily to reconcile — and expose a small, auditable difference between the two.
Exactly-once billing over at-least-once ingest
Kafka guarantees at-least-once delivery, and a retry-happy ingest layer makes duplicates likely. Two dedupe points fix this: at ingest (Redis set of clickIds seen in the last N hours) and at batch (dedupe on clickId in the SQL that produces daily totals — SELECT COUNT(DISTINCT clickId)). The real-time view doesn't strictly need dedupe if the batch reconciles it — an occasional double-count in a per-minute dashboard is acceptable.
The subtle case: a click's retry comes in after its dedupe TTL expired. The batch will still catch it because it dedupes over the full day's dataset, not a sliding window. So the design invariant is: ingest-side dedupe is best-effort for real-time, batch-side dedupe is authoritative for billing.
Late events and window closing
Clicks can arrive minutes late — a phone with a spotty connection, a browser buffering beacons. The stream job has to decide when a window is 'closed' and its number is committed. Flink watermarks are the mechanism: the system tracks the max timestamp seen and considers a window closed when the watermark passes window_end + slack. Late events beyond the slack are dropped or diverted to a side output.
For advertiser dashboards, we set a 5-minute slack — that catches ~99.5% of real events and produces stable numbers within 5-10 minutes of the window closing. Late events after that go into the batch anyway and show up in the next day's reconciliation. This is the moment where the two-view design pays off: the stream lies a little, the batch tells the truth, and the reconciliation report bounds the lie.
Follow-ups you should expect
How do you filter fraudulent clicks?
How does the schema evolve without breaking everything?
What if a Kafka partition falls behind?
Do you need Kafka? Can this be a queue?
How do you serve arbitrary dashboard slice-and-dice?
Where candidates lose the room
- Trying to serve billing off the same real-time table as dashboards — you'll be arguing about pennies with advertisers over stream lag forever.
- Skipping the raw event log — the day someone changes the aggregation logic, you can't re-run history without it.
- Only one dedupe point — either ingest-only (retries after TTL slip through) or batch-only (real-time double-counts confuse advertisers).
- Ignoring late events — an unbounded stream that never closes windows never emits stable numbers.
- Putting the ingest endpoint behind your normal API gateway — a beacon at 500k QPS wants to land on a cheap CDN edge, not your rate-limited public API.