Scope the requirements
Two hard problems: pick the right ad from millions of candidates in <100 ms, and count every impression + click accurately for billing. Split them — serving is a stateless prediction path; accounting is a durable event pipeline.
Functional — what it must do
- Show a targeted ad in response to an ad slot request within 100 ms.
- Support targeting rules (geo, age, interests, custom audiences).
- Budget pacing — spend advertiser budgets across the day, not front-loaded.
- Click + impression tracking with dedup, feeding billing.
- Explicitly cut: creative uploads UI, fraud arbitration, attribution — offer them back.
Non-functional — the numbers that shape the design
- Ad-serve latency: p95 under 100 ms, p99 under 200 ms.
- 1M ad requests/s peak; 10B impressions/day; 100M clicks/day.
- Budget accuracy: overspend under 0.1% per campaign per day.
- Click and impression events land in billing within seconds.
- 99.99% on ad-serve; degrade to a default ad rather than empty slot.
The 100 ms deadline dominates the design. Auction complexity has to fit under it. Everything expensive (targeting features, ranker) is precomputed.
Back-of-envelope estimates
Ad-serve QPS is huge; auction complexity is bounded per request. Click volume is small but must be exact for billing.
| Quantity | Estimate | How you got there |
|---|---|---|
| Ad requests/s | ~1M peak | publisher pages + apps × ad slots per page. Each slot = one auction. |
| Candidates per auction | ~1,000 | narrowed from millions of campaigns by targeting index. |
| Ranker inference | ~10 ms for top 100 | GBM or shallow DNN, batched on GPU or vectorised on CPU. |
| Impressions/day | ~10B | not every request wins — fill rate ~30%. 1M QPS × 86,400 × 0.3. |
| Click events/day | ~100M | 1% CTR average, higher on some placements. |
| Budget updates | ~10k/s decrements | per-campaign counters; Redis INCR then batch to Postgres. |
One million auctions a second, 30 ms budget per auction. That's a hardware + algorithms constraint, not a normal web app.
Sketch the API
One hot request path (bid), one write path (events). Advertiser side runs on a separate cluster — different SLA, different scale.
- Bid path is a hard-deadline stateless call — no database round trips allowed. Everything preloaded into memory or a nearby Redis.
- Tracking beacons are fire-and-forget over HTTP; the client doesn't wait for downstream durability.
Data model
Three stores by function: campaign config (small, RAM-first), user features (per-user vector), event log (append-only, huge).
Campaigns in Postgres for durability; snapshot into ad-server RAM for zero-database-hop serving. Budget counters in Redis because Postgres cannot handle 10k updates/s per campaign. Events log follows the standard [[ad-click-aggregator]] pattern — stream + batch.
High-level design
One tier serves bids fast; another tier ingests events; a third tier runs the advertiser UI + billing. Zero coupling on the hot path.
- Bid request arrives → ad server looks up user's feature vector in Redis (~1 ms).
- Targeting index (in-memory inverted index) narrows to ~1,000 candidate campaigns matching the request context.
- Ranker scores top 100 candidates, applies pacing multiplier from budget Redis, picks the winner.
- Server returns creative + tracking URL. Total budget spent so far this second on this campaign is a Redis INCR — bounded to campaign's per-second cap.
- User's browser fires impression beacon → event ingest → Kafka. Downstream: batch billing computes daily spend; stream job updates budget counters for pacing.
Deep dives — where the interview is won
The 100 ms budget: what fits and what doesn't
100 ms total; ~30 ms is network round trip (edge to server to edge); leaves ~70 ms for compute. Feature fetch ~5 ms (Redis). Targeting filter ~5 ms (in-memory inverted index on tags). Ranker inference ~10 ms (batched GBM on top 100 candidates). Auction logic + pacing check ~5 ms. Response serialisation ~5 ms. Total ~30 ms, leaving headroom for slow tail.
The corollary: no Postgres call on the bid path, ever. Campaigns are snapshotted into ad-server memory and hot-reloaded on change. If a campaign was updated 10 seconds ago and hasn't hot-reloaded yet, we serve the old version — better than a slow query. Consistency is intentionally weak on the serving side.
Budget pacing without hot-row contention
One campaign spends across the day. Naively increment a Postgres row per impression — 10k/s hot-row contention destroys the DB. The pattern: Redis INCR per (campaign_id, minute). Ad server checks the running counter against the campaign's per-minute pacing cap; if over, skip this campaign in the auction. Batch writer flushes minute counters to Postgres every ~1 min.
Pacing itself is a rate limiter: 'spend $X across the day' becomes 'spend up to $X/1440 per minute'. Fluctuating traffic means some minutes underspend and others get relaxed caps — a smoothing algorithm run continuously per campaign. This is where the ranker gets a multiplier: campaigns behind pace bid higher, campaigns ahead of pace bid less.
Click tracking with exactly-once billing
Click volume is small (~100M/day) but every click is billable. The path: user clicks ad → GET /click?id=... → server logs click event (id, campaign_id, user_id, ts, clickId) to Kafka → 302 redirects to advertiser URL. The event carries a clickId; downstream dedupes on it.
Billing runs batch daily against Kafka's durable archive: SELECT COUNT(DISTINCT clickId) FROM clicks WHERE date = X GROUP BY campaign_id. The DISTINCT is the exactly-once guarantee — even if the click endpoint retried, the DB dedups. Stream-side aggregation for the ad server's pacing counters may double-count during outages; batch always corrects it.
Follow-ups you should expect
How do you rank ads?
Ad fraud (bots clicking)?
How do you personalise without a user id (privacy)?
Publishers want to guarantee premium ads for premium slots — how?
Global campaigns across regions — how do you avoid overspending?
Where candidates lose the room
- Postgres call on the bid path — 100 ms budget blown by one slow query.
- Per-impression Postgres update for budget — hot-row contention on any active campaign.
- Stream-only billing — a Kafka lag turns into a support ticket about missing revenue.
- Trying to update campaigns synchronously to all ad servers — a global fanout in the middle of the day; use hot-reload snapshots with 1-min lag instead.
- No dedup on clicks — client retries during network flakiness create phantom revenue.