Scope the requirements
The heart of this question is one scenario: two cursors in the same paragraph, both typing, network latency between them. Everything else — WebSockets, storage, presence — is scaffolding around making that scenario converge. I'd name that scenario in the first minute so the interviewer knows I see where the difficulty lives.
Functional — what it must do
- Multiple users edit the same document concurrently; everyone's edits appear for everyone in real time.
- All collaborators converge to the same document state, whatever the interleaving of edits.
- Presence: live cursors, selections, and who's-viewing indicators.
- Documents are durable across sessions; sharing with viewer/editor permissions.
- Explicitly cut for v1: full version history, offline editing, comments and suggestions — I'll sketch how each slots in if asked.
Non-functional — the numbers that shape the design
- Edit propagation under ~100 ms same-region — collaborators should see keystrokes as typing, not as chunks.
- Strong eventual consistency, deliberately not linearizability: every replica applies every op and all agree at quiescence.
- Zero lost keystrokes once acknowledged — durability before ack on the op log.
- Scale: millions of concurrent documents; each doc has few collaborators (2-100), docs up to ~10 MB of text.
- ~10M concurrent WebSocket connections at peak across the fleet.
The consistency requirement is the unusual one and I want to name it precisely: not linearizability — that would mean locking the document — but guaranteed convergence. Everyone ends at the same state without anyone ever being blocked from typing. That single requirement forces almost every design decision that follows.
Back-of-envelope estimates
The numbers reveal a comforting shape: per-document load is tiny — a handful of humans type slowly — so the challenge is fleet-wide connection count and op-log volume, not per-doc throughput. That's why one sequencer per doc works at all.
| Quantity | Estimate | How you got there |
|---|---|---|
| Concurrent connections | ~10M | 100M DAU, ~10% concurrent at peak. At ~100k WebSockets per host, that's ~100 connection hosts. |
| Ops per second, global | ~5M/s | Each active editor emits ~1 op/s (keystrokes batched into small ops). |
| Op-log ingest | ~500 MB/s | 5M ops/s × ~100 B/op. Fleet-wide firehose; per-doc it's a trickle. |
| Per-doc op rate | ~2-100/s | Bounded by collaborator count and human typing speed — the reason a single sequencer per doc never becomes the bottleneck. |
| Document text | ~50 TB | 1B docs × ~50 KB average. Small. The op logs dominate storage many times over — snapshots plus truncation are mandatory, not optional. |
| Ops between snapshots | ~1,000 | Loading a doc = 1 snapshot + ≤1,000 ops to replay; older ops become prunable. |
Per document, load is human-bounded — nobody types faster than about ten ops a second. So I never need to scale a single document; I need to place millions of small, chatty documents across a fleet and keep each one's ordering authority on exactly one node.
Sketch the API
A thin REST surface for document lifecycle, and a WebSocket protocol where the actual editing lives. The load-bearing design choice is in the message schema: every op carries the version it was based on.
baseVersionon every op is the crux of the protocol: the client says "I made this edit against version 42", and the server knows exactly which concurrent ops (43, 44, …) to transform it against before applying. Without it, the server can't order edits correctly and convergence is unprovable.- The server assigns a monotonically increasing version per document on each accepted op — this total order is what operational transformation needs to be correct, and it's why exactly one server must own a given doc at a time.
Data model
The document is not a row that gets updated — it's an append-only log of operations plus periodic snapshots. That reframing is the data-model insight the interviewer is listening for.
Postgres for metadata and ACLs because permissions want transactions; Cassandra for the op log because it's an append-only, doc-partitioned firehose at 500 MB/s that never needs cross-doc queries — exactly the workload Cassandra is shaped for; S3 for snapshots. If someone pushed me to simplify, I'd concede DynamoDB could hold both metadata and ops in one store — but I would not move the op log into Postgres, because 5M appends a second is not a relational problem.
High-level design
Editors hold WebSockets to a collaboration tier, and the routing rule is the architecture: every editor of a given document reaches the same server, which acts as that document's operational-transform engine and sequencer. Storage hangs off that server; compaction runs behind it.
- Join: client fetches the latest snapshot plus the op tail over REST (ACL checked against Postgres), then opens a WebSocket; the gateway consults the placement map and pins the connection to the doc server that owns this docId.
- Alice types → her client sends
{ insert, pos: 5, "x", baseVersion: 42 }. - The doc server transforms her op against any concurrent ops it accepted after version 42, assigns version 45, appends to the op log, and only then acks — durability before acknowledgement, zero lost keystrokes.
- The server broadcasts the transformed op to Bob and every other connected editor, who apply it locally; cursor positions ride the same channel into Redis with a short TTL.
- Behind the scenes, the compaction worker folds every ~1,000 ops into a new S3 snapshot and truncates the log — keeping doc loads fast and storage finite.
Deep dives — where the interview is won
Convergence: why last-write-wins fails, and what OT does
Start from the failure: the doc is cat, Alice inserts s at position 3 while Bob inserts black at position 0. If the server applies both literally, Bob's insert shifts the text and Alice's s lands mid-word — the replicas diverge or corrupt. Last-write-wins on the whole document is worse: someone's edit vanishes entirely. Operational transformation fixes this by adjusting each incoming op against the concurrent ops that were accepted before it: the server sees Alice's insert was based on version 42, Bob's op 43 inserted 6 chars before position 3, so Alice's position transforms from 3 to 9. Both orderings now converge to black cats.
The precondition — and the sentence interviewers wait for — is that OT needs a total order of operations per document, which is why exactly one server sequences each doc. Transformation without an agreed order is where a decade of buggy OT papers went to die. The client mirrors this: it applies its own edits optimistically, tags outgoing ops with baseVersion, and transforms incoming remote ops against its unacknowledged local ones, so typing never blocks on the network.
OT vs CRDT: pick OT with a central server, and name what CRDTs buy
CRDTs make concurrent ops commute by construction: every character gets a globally unique, ordered position identifier (fractional indexing, RGA, as in Yjs or Automerge), so replicas can apply ops in any order and still converge — no sequencer, no server authority. That's genuinely better for peer-to-peer editing and long-offline merges. The costs are real, though: position identifiers and tombstones grow document memory well beyond the raw text, garbage-collecting them safely is subtle, and naive implementations interleave concurrently-typed runs of text mid-word.
My interview answer: since the requirements already include a server (permissions, durability, presence), I take server-ordered OT — smaller state, simpler correctness argument given the total order, and it's what Google Docs runs. I'd flag the escape hatch explicitly: if offline-first or P2P became a requirement, I'd reach for a mature CRDT library like Yjs rather than hand-rolling either algorithm — both have famously sharp edges.
Doc affinity: consistent hashing on docId, and what happens when a server dies
All editors of a doc must reach the sequencer, so I hash docId onto a consistent-hash ring of doc servers, with membership and lease coordination in etcd or ZooKeeper. The gateway resolves docId to owner at connect time. Consistent hashing means a server joining or dying reshuffles only its own arc of documents, not the world. The failure sequence matters: the dead server's leases expire, its docs' arcs pass to neighbours, clients reconnect through the gateway to the new owner, which rebuilds state from snapshot plus op-log tail. Clients then resend unacknowledged ops with their baseVersions — the same machinery as normal editing, because an op is never lost once acked and never acked before it's in Cassandra.
The alternative worth naming: skip affinity, let any server accept ops, and serialise through Redis pub/sub or a per-doc Kafka partition. Simpler routing, but every op pays extra hops, and you've moved the sequencing problem rather than solved it. There's also a hot-doc wrinkle: a document with 5,000 viewers (a company all-hands doc) shouldn't hold 5,000 connections on the sequencer — read-only followers subscribe via pub/sub fan-out on other nodes; only editors talk to the sequencer.
The op log grows forever unless snapshots truncate it
Storage math says text is ~50 TB but the op log ingests ~500 MB/s — about 40 TB a day. Unbounded, it also makes document loads unbearable: opening a year-old busy doc would replay millions of ops. The fix is classic log compaction: every ~1,000 ops per doc, a worker folds the log into a materialised snapshot in S3, updates the metadata pointer, and truncates ops older than the snapshot. A doc load is then one snapshot fetch plus at most a thousand small ops — tens of milliseconds.
Version history comes almost free from the same machinery: instead of deleting old snapshots, retain a sampled series of them (hourly, then daily, thinning with age) as the restore points users see. What you deliberately don't keep is every keystroke forever. And presence never enters any of this — cursors live in Redis with a TTL and die with the session; persisting them is a small mistake that reliably signals confusion about what's data and what's ephemera.
Follow-ups you should expect
Why not lock the document, or sections of it, instead of all this transformation?
How do cursors stay correct when other people's edits move the text?
How do offline edits reconcile on reconnect?
What happens when someone's edit access is revoked mid-session?
A doc goes viral inside a company — 5,000 people open it. What breaks?
Where candidates lose the room
- Proposing HTTP polling with whole-document saves — it turns concurrent editing into last-write-wins and fails the core requirement in one move.
- Saying "use OT" or "use CRDTs" as a magic word without walking through one concrete two-edit transform — interviewers probe exactly here, and the hand-wave collapses.
- Letting two servers accept ops for the same doc — OT's correctness rests on a per-doc total order; without affinity and fencing the whole argument is void.
- No snapshot/compaction story — the op log is the dominant storage cost and unbounded replay makes old docs unopenable.
- Persisting presence and cursors to the database — ephemeral data in durable storage signals you're not distinguishing state from signal.