Scope the requirements
First question back to the interviewer: what is the crawl for — a search index or LLM training data? Both want extracted text at scale. Then I scope hard, because the un-scoped version of this problem is infinite.
Functional — what it must do
- Start from seed URLs, fetch pages, and follow extracted links to discover the web.
- Store raw HTML and extracted text for downstream consumers (indexer or training pipeline).
- Respect robots.txt — disallow rules and crawl-delay — and stay polite per domain.
- Deduplicate both URLs and page content, and survive crawler traps.
- Explicitly cut: JavaScript rendering, images and video, authenticated pages, adversarial cloaking — each is its own project, and I'll say what it would take if asked.
Non-functional — the numbers that shape the design
- Scale: 10B pages in ~5 days — a full-web-scale batch crawl.
- Politeness ceiling: ~1 request/s per domain — throughput comes from breadth across millions of domains, never depth on one.
- Fault tolerant: a crashed worker loses zero progress — days of crawl state must survive any single failure and be resumable.
- Efficiency: don't fetch or store the same content twice; bandwidth is the dominant cost.
- Freshness and re-crawl scheduling scoped out of v1 — I'll sketch it as a follow-up.
The interesting constraint is that the two scale numbers point in opposite directions — 25 thousand pages a second overall, but at most one per second to any single domain. The whole architecture is about resolving that tension politely, and about making a multi-day job restartable.
Back-of-envelope estimates
This estimate decides the fleet size, which is rare — and it shows the system is network-bound, not CPU-bound, which decides where the money goes.
| Quantity | Estimate | How you got there |
|---|---|---|
| Crawl rate needed | ~25k pages/s | 10B pages ÷ 5 days (~430k s) ≈ 23k/s sustained — call it 25–40k with retries and overhead. |
| Data transferred | ~20 PB | 10B pages × ~2 MB average transfer (page + assets fetched). Bandwidth is the bill. |
| Per-machine throughput | ~4k pages/s | A 200 Gbps node at ~30% practical utilisation moves ~2 MB × 4k/s. One machine alone would take a month. |
| Fetcher fleet | ~8 machines | 25k/s ÷ 4k/s per machine. Tiny fleet, huge pipes — the design is network-bound. |
| Domains in flight | millions | At ≤1 req/s/domain, 25k pages/s requires ~25k+ domains actively being fetched at any instant — breadth is mandatory. |
| Storage | ~1 PB + 10 TB | 10B × ~100 KB stored HTML ≈ 1 PB in blob storage; 10B × ~1 KB metadata ≈ 10 TB in the URL table. |
Eight big machines can move the bytes — the hard part is that 25 thousand pages a second must be spread across tens of thousands of domains at once to stay polite. So the frontier and the per-domain scheduler are the real design, not the fetchers.
Sketch the API
There's no end-user API — the crawler's interface is seeds in, text in blob storage out. What I expose is an operator surface, and the interesting decision is that queues carry pointers, never page bodies.
- Queue messages carry URLs and S3 pointers, never HTML bodies — a 2 MB page through SQS or Kafka per hop would multiply the 20 PB across every stage. Fetchers write raw HTML to blob storage once and pass a reference downstream.
- Seed submission normalises URLs (lowercase host, strip fragments, resolve relative paths, sort query params) before the dedup check — dedup on un-normalised URLs is dedup in name only.
Data model
One table holds the crawl's entire brain-state — which is exactly what makes the system resumable. Everything else is derived or cached.
depth is the crawler-trap fuse; content_hash feeds dedup.Metadata goes in DynamoDB — 10 TB of point lookups by URL and heavy write concurrency from thousands of workers is exactly a partitioned KV workload, and there is nothing relational here. Raw HTML and extracted text go straight to S3; a petabyte does not belong in any database. If the interviewer wants richer crawl analytics — link graphs, per-domain reporting — I'd stream metadata changes into a warehouse rather than querying the hot table.
High-level design
A staged pipeline glued together by queues: frontier → fetch → parse → back to frontier. Splitting fetch from parse is deliberate — they scale on different resources and fail for different reasons, and the queue between them is the checkpoint that makes the whole crawl resumable.
- A URL is dequeued from the frontier; the fetcher checks cached robots.txt rules and takes the per-domain rate-limit token in Redis — if the domain is on cooldown, the message is deferred back to the queue with a delay, never held by a blocked worker.
- The fetcher resolves DNS through a local DNS cache, downloads the page, writes raw HTML to S3, and drops a message with the S3 pointer onto the parse queue.
- A parser pulls the pointer, extracts text to S3, hashes the content for dedup, and normalises every outbound link.
- Each new link is checked against the metadata DB (URL seen?) and the content Bloom filter, gets
depth = parent + 1, and — if new and under the depth cap — is enqueued back to the frontier. - The queue message is deleted only on success; a worker that dies mid-page lets the visibility timeout expire and the URL is re-claimed by another worker. After ~5 failed attempts the message lands in the DLQ for inspection.
Deep dives — where the interview is won
Politeness is architecture, not etiquette — and it's also self-preservation
Two mechanisms per domain. First, robots.txt: fetched once per domain, parsed, cached in the domains table with a daily TTL; every URL is checked against disallow rules before fetch, and a Crawl-delay directive overrides my default rate. Second, a per-domain rate limiter in Redis — a sliding window or a SET NX lock with TTL capping at ~1 req/s — with a little jitter so thousands of workers don't synchronise their retries against the same domain.
The scheduling subtlety is what to do when a domain is on cooldown: a worker must never sleep holding the message, or a handful of slow domains stalls the fleet. Instead it pushes the message back with a delay (SQS ChangeMessageVisibility) and moves on. This is Mercator's front-queue/back-queue idea in modern dress: priority decides what to crawl, per-domain queues decide when. And politeness is not altruism — an impolite crawler gets IP-banned across half the web, and a banned crawler fetches nothing.
Fault tolerance comes free from the queue — if state lives in the right place
Splitting fetch and parse into separate stages with a queue between them means each stage checkpoints its work. A fetcher that crashes after writing HTML to S3 but before enqueueing loses one page's work, not a batch; the frontier message reappears after the visibility timeout and another worker re-claims it. Messages are deleted only on success — "a dead crawler's work is re-claimed" is the property that makes a 5-day job survivable.
Retries ride the same mechanism: the queue's receive count drives exponential backoff, and after ~5 attempts the URL parks in a dead-letter queue for humans, so one permanently broken page can't burn bandwidth forever. The anti-pattern is in-memory retry timers inside workers — a crash loses every pending retry silently. All durable crawl state lives in the metadata DB and the queues; workers hold nothing worth mourning, which also means I can autoscale them freely — parsers on queue depth, fetchers on bandwidth.
Dedup twice — URLs and content — and cap depth or the web never ends
URL dedup first: normalise (case, fragments, query-param order, trailing slashes), then check the metadata DB before enqueueing. That stops re-crawling known pages but misses the sneakier duplicate: the same content under different URLs — mirrors, www vs bare domain, session IDs in paths. So parsers also hash page content and check a Bloom filter: ~an order of magnitude less memory than an exact hash set across 10B entries, answering in microseconds. False positives mean we occasionally skip a genuinely new page that hash-collided in the filter — for a crawl, an acceptable loss; if it weren't, I'd confirm positives against an exact hash index in the metadata DB.
Then crawler traps: calendars that link to the next day forever, auto-generated URL spaces that are infinite by construction. No dedup catches URLs that are all technically new. The fuse is a max depth of ~15–20 from the seed, stored per URL and incremented on extraction — real pages are rarely that deep, and traps are cut off mechanically. Belt-and-braces: per-domain page budgets catch traps that stay shallow but sprawl wide.
The bottleneck nobody names: DNS
At 25k pages/s across millions of distinct domains, every fetch starts with a DNS lookup, and a synchronous resolver round-trip can be 10–100 ms — measured crawls have spent up to 70% of fetch time on DNS. The fix is boring and essential: an aggressive local DNS cache on every fetcher (most crawl bursts hit the same domains repeatedly, so hit rates are high), asynchronous resolution so lookups don't block fetch threads, and multiple upstream providers round-robined — both for capacity and because a single provider will rate-limit a client doing millions of lookups an hour.
This is also where I'd volunteer the scaling asymmetry of the whole system: fetchers scale on bandwidth, parsers on CPU, and they autoscale independently on different signals — fetchers on frontier depth and network utilisation, parsers on parse-queue depth. Coupling them into one monolithic fetch-and-parse service forces you to scale both on the worst of either, which is precisely why the pipeline is staged.
Follow-ups you should expect
How would you prioritise which URLs to crawl first?
How do you re-crawl for freshness?
What about pages that need JavaScript to render?
How do you avoid getting IP-banned at this scale?
Where does the extracted text go — how do consumers use it?
Where candidates lose the room
- Treating robots.txt and politeness as trivia to mention in passing — it's a core requirement, it shapes the frontier design, and skipping it reads as "I would DDoS the internet".
- One monolithic fetch-and-parse service — the stages scale on different resources (bandwidth vs CPU) and the queue between them is what makes the crawl resumable.
- Pushing HTML bodies through the queues — 20 PB through SQS per stage; queues carry S3 pointers, always.
- URL dedup only — mirrors and session-ID URLs mean the same content gets stored thousands of times; content hashing is the second, mandatory dedup layer.
- No depth limit — the first calendar widget the crawler meets becomes an infinite loop; and in-memory retry timers that a crash silently erases instead of queue-based backoff.