Scope the requirements
This looks like Netflix until you count tracks. 100M+ items with a long-tail play distribution means CDN caching becomes a shard-selection problem, not a hit-rate problem.
Functional — what it must do
- Search + play any track from a 100M-item catalogue.
- Playlists (user-authored + system-generated daily mixes).
- Gapless playback, offline downloads.
- Track-play events feed recommendations + royalty accounting.
- Explicitly cut: podcasts, video, social — offer them back.
Non-functional — the numbers that shape the design
- Play start latency under 300 ms to first audio.
- 500M MAU, ~200M DAU, ~50M concurrent listens at peak.
- Long-tail catalogue: the top 1% of tracks drives 50% of plays; the bottom 50% drives 5%.
- Track file size ~3-5 MB (Ogg Vorbis); daily upload ~40k new tracks.
- 99.99% on play; browse can degrade.
The tail is the design driver. I need a CDN that can cache the head cheaply and fall through to origin fast enough that the tail still plays under 300 ms.
Back-of-envelope estimates
Storage is huge but bandwidth is modest — audio is 10× smaller than video per hour. The interesting math is per-user play data.
| Quantity | Estimate | How you got there |
|---|---|---|
| Concurrent listens | ~50M peak | 200M DAU × 25% overlap during peak commute windows. |
| Bandwidth | ~15 Tbps | 50M × ~300 kbps average bitrate (Vorbis q5). Small next to video streaming. |
| Catalogue storage | ~5 PB | 100M tracks × 5 encodings × ~10 MB avg (raw + ladder). Growing 40k/day. |
| Play events/day | ~5B | 200M DAU × ~25 plays/day. Big stream, but small compared to feed events. |
| CDN hit rate on head | ~99% | top 1% (1M tracks) fits in ~5 GB per edge box. Tail misses fall through to regional. |
| Daily-mix rebuild | ~1B rows/day | one per user; batch job runs nightly reading last N days of history. |
50M concurrent × 300 kbps is small next to Netflix. The scale story here is the catalogue depth, not bandwidth.
Sketch the API
Browse + play + event ingest. Playback returns a signed URL; the manifest points to a CDN edge nearest the listener.
- Gapless playback needs a hint in the manifest: the response includes
gaplessNextwith a preload URL so the client can start buffering the next track before the current one ends. - Play events are batched (not per-second) — the client accumulates and fires every 30 s or on track change.
Data model
Cold catalogue + hot play history + derived rollups. Postgres for the small stuff, Cassandra for the wide streams.
The play stream is the only large-write dataset — Cassandra keyed by user_id. Track metadata is small enough for Postgres; caching that whole table in Redis costs cents. The daily mix is a derived Redis-only view of yesterday's play history.
High-level design
Three planes: catalogue (small, read-heavy), media (large, one-way, CDN-first), and events (append-only stream feeding both royalty accounting and recommendations).
- Client plays track → GET /playback → catalogue check → sign a manifest URL pointing at the nearest CDN → return.
- Client fetches the audio direct from CDN. On miss, edge pulls from regional cache; last resort is origin (~1% of tail traffic).
- Every ~30 s the client batches play events → events ingest → Cassandra append + Kafka publish.
- Recs pipeline consumes the event stream, aggregates listening patterns, writes daily mixes to Redis before morning peak.
- Royalty pipeline runs monthly against the durable Cassandra + Kafka archive to compute artist payments — batch not stream, because rights disputes are common and re-runs must be deterministic.
Deep dives — where the interview is won
Long-tail caching: keep the head at the edge, let the tail fall through fast
Video streaming (Netflix) has a small catalogue that fits comfortably at every ISP edge. Music has 100M+ tracks; you cannot pre-place all of them everywhere. The pattern: cache the top ~1% at every edge (they serve half the traffic), the next 10% at regional, and let the deep tail fetch from origin. Because audio files are small (~5 MB) and the tail is spread across many artists, origin latency is tolerable — a ~50 ms extra round trip on a niche track is fine.
The routing decision (which cache tier) is made from CDN request logs: if a track is requested more than N times per region per hour, promote it to a hotter tier. Demotion happens naturally through LRU eviction. This is the classic 'popularity-driven caching' pattern, and the interviewer scores on you naming the tail as the design axis.
Gapless playback: the manifest hint that changes everything
Gapless means no audible pause between tracks (albums, mixes). The naive design fetches track N, ends, then requests track N+1 — the round trip is audible. The fix: on the /playback response for track N, include gaplessNext = playback URL for track N+1. The client starts prefetching N+1's first audio segment as N's playback progresses. When N ends, N+1's buffer is already primed.
The subtlety: gapless must survive a skip. If the user skips N halfway through, the prefetch of N+1 is wasted bandwidth. Cap prefetch at the first 10 seconds of N+1 — enough to start playback without noticeable gap, small enough to abandon cheaply.
Daily mix: overnight recall, online serving
The daily mix (personalised playlist rebuilt each morning) is a perfect fit for offline-compute + online-serve. Nightly batch reads yesterday's play events per user, runs collaborative filtering + content signals, writes a ~50-track playlist to Redis keyed by user_id.
The read path is one Redis GET. If the mix expires or the user is new, fall through to a session-based online recommendation using recent history + region trending. The trick is that morning coffee/commute traffic hits Redis first, and the fallback path only serves ~1% of users on any given day.
Follow-ups you should expect
How is this different from Netflix?
How do you handle offline downloads?
Royalty accounting for 5B plays/day?
Search over 100M tracks with typo tolerance?
What breaks if origin S3 goes down?
Where candidates lose the room
- Designing for a Netflix-shaped catalogue — pre-placing 100M tracks everywhere is impossible; you need popularity tiers.
- Fetching next track only when current ends — gap is audible on any transport.
- Streaming royalty accounting — rights are retroactive and disputed; batch with a monthly recompute is the norm.
- Play events through your OLTP database — 5B/day is a stream shape, not a table shape.
- One-shot recommendation on request — the daily mix pattern buys you cheap serving at the cost of one day of staleness.