Scope the requirements
The system built to store the whole web crawl. Optimised for large sequential reads/writes, not random access — that's the whole design philosophy.
Functional — what it must do
- Store files >GB in size across commodity hardware.
- Read + write with sequential-access primary, random-access secondary.
- Append is a first-class operation (many concurrent writers).
- Automatic replication + recovery on node failure.
- Explicitly cut: POSIX semantics, small-file optimisation, transactional writes — offer them back.
Non-functional — the numbers that shape the design
- Throughput over latency: sustained aggregate ~100 GB/s reads across the cluster.
- 10s of PB across thousands of chunk servers.
- Hardware failure is normal — nodes die daily; system self-heals within minutes.
- Chunk size 64 MB — large enough to amortise metadata, small enough to distribute.
- 99.9% availability on read; writes may block during master failover.
The design assumes commodity hardware fails constantly. Everything is replicated 3× by default; the master reconciles the world every few seconds.
Back-of-envelope estimates
Chunks (64 MB) are the unit of everything: storage, replication, metadata. Metadata size determines master RAM.
| Quantity | Estimate | How you got there |
|---|---|---|
| Total storage | ~50 PB | spread across ~1,000 chunk servers × ~50 TB each. |
| Chunks total | ~800M | 50 PB ÷ 64 MB. Each has 3 replicas → 2.4B chunk replicas cluster-wide. |
| Metadata size | ~64 GB | 800M chunks × ~80 B (id, version, replica locations). Fits in master RAM. |
| Read throughput | ~100 GB/s aggregate | clients read direct from chunk servers; master serves metadata only. |
| Master QPS | ~10k/s | metadata lookups. Cached client-side to reduce load. |
| Chunk server health checks | every 10 s | master pings; missed 3 → chunk server marked dead, re-replication triggered. |
One master holds all metadata in RAM. Chunk servers own the data. Clients talk to master once, then straight to chunk servers.
Sketch the API
Client library provides open/read/write/append/close. Under the hood: talk to master for chunk location, then chunk server for bytes.
- Reads bypass the master after the first metadata lookup — clients cache chunk locations and go direct to chunk servers.
- Append is the primary write pattern — most log-like workloads append. Overwrite is supported but discouraged.
Data model
Master holds file → chunk mapping + chunk → replica mapping in RAM. Chunk servers hold the actual bytes as Linux files.
Master is a specialised in-memory database. Chunk servers use the local filesystem — no fancy DB, just files. This is a deliberate design — GFS predates modern key-value stores and the file abstraction was chosen because it maps directly to what the hardware wants.
High-level design
One master (with shadow backup), thousands of chunk servers, client library. Master is the metadata brain; chunk servers own the bytes; clients cache aggressively.
- Client wants to read /path/file offset 100 MB → calls master.getChunkLocations → master returns { chunkId: 42, replicas: [cs17, cs42, cs103] } → client caches this.
- Client picks nearest replica (say cs17), calls cs17.read(chunkId=42, offset=..., len=...) → receives bytes directly. Zero master involvement.
- Write: client → master to allocate a chunk if needed → master picks 3 replicas and designates one as primary → returns primary + secondaries to client → client pipelines bytes to primary → primary orders + forwards to secondaries → confirms back.
- Every chunk server sends heartbeats every 10 s with a list of chunks it holds. Master reconciles: unexpected chunks get deleted; missing replicas trigger re-replication.
- Master failure: WAL + checkpoint replayed on shadow → shadow becomes new master → clients retry and get the new master's address from chubby-style coordination service.
Deep dives — where the interview is won
Single master + client caching: why the master isn't a bottleneck
It sounds fragile: one master for a 50 PB cluster. The trick is that master involvement per byte is nearly zero. Clients call master once per file (or once per chunk, for very large files) to get locations, then talk direct to chunk servers for all the actual bytes. Master QPS scales with client count, not data volume.
The 64 MB chunk size is what makes this work. If chunks were 1 MB, a 1 GB file needs 1000 master calls; with 64 MB chunks, 16 calls. And clients cache chunk locations for the file's lifetime unless the master invalidates. In practice, a busy cluster serves 100 GB/s of data with the master handling <10k RPC/s.
Replication pipeline: primary orders, secondaries follow
On write, the client sends bytes to the primary replica of the target chunk. Primary orders the write within its chunk (assigns a mutation number), then chains the bytes to the secondary replicas — pipelined for throughput, not fanned out star-shape (which would double the client's uplink). Once all secondaries ack, primary confirms to client.
The chaining pattern spreads bandwidth. If client → primary is 1 Gbps, and each replica forwards to the next, total bandwidth cost is 1× at each replica, not 3× at the client. Networks are laid out to make this cheap: replicas in the same rack talk fast to each other, and racks are chosen to survive one rack failure without losing all replicas.
Failure handling: expect death, fix it in the background
Chunk servers die daily in a big cluster. Master notices via heartbeat timeout, marks the server dead, and schedules re-replication of every chunk it held (~50 TB of chunks). Re-replication is throttled to avoid overwhelming the network — a chunk server rebuild happens over hours, not seconds. Since chunks have 2 more replicas alive during rebuild, availability is preserved.
The master itself is the single point of failure. Two mitigations: (1) WAL is replicated to a shadow master in real time, so failover is fast (~30 s including client reconnect), (2) the shadow serves read-only metadata during the primary's slow operations. If both masters die, cluster is unavailable for writes; reads with cached metadata may continue for a while.
Follow-ups you should expect
Why 64 MB chunks and not smaller?
How does this compare to HDFS?
What about S3?
How do you support small files without wasting chunks?
Erasure coding vs 3× replication?
Where candidates lose the room
- Trying to put a database on the master — WAL + in-memory tree is faster than any general-purpose DB for this shape.
- Making the master involved in byte transfer — you re-invented NFS and lost the throughput story.
- Small chunks — metadata explodes and master RAM becomes the bottleneck.
- Synchronous re-replication on failure — you overwhelm the network trying to rebuild in seconds; throttle to hours.
- Expecting POSIX semantics — the API is deliberately less than POSIX to permit concurrent appends without locks.