Design a job scheduler

Scope the requirements

The question sounds like cron, and the trap is designing cron: one box, one loop, in-memory timers. The interviewer is asking how you get precision, at-least-once execution, and no single point of failure at the same time.

Functional — what it must do

Submit jobs to run immediately, once at a future time, or on a recurring cron schedule.
Jobs execute registered tasks with parameters — a job is a task plus a schedule.
Query job and execution status: pending, running, succeeded, failed.
Automatic retries with backoff on failure, bounded attempts, then dead-letter.
Explicitly cut: inter-job dependencies and DAGs — that's Airflow, a different question — plus cancel/reschedule, which I'd add back if time allows.

Non-functional — the numbers that shape the design

Throughput: 10,000 job executions per second.
Precision: a job fires within ~2 seconds of its scheduled time.
At-least-once execution — a job is never silently skipped, even if a worker or scheduler node crashes mid-flight.
No single point of failure in the scheduling tier — this is the anti-cron requirement.
Burst tolerance: minute-boundary spikes of 10–50× average, because everyone schedules on the hour.

Say this out loud

The two numbers that shape everything are 10k executions a second and 2-second precision. Neither is hard alone — together they rule out both a plain cron box and polling the database every second, which is why this needs two layers.

Back-of-envelope estimates

Ninety seconds of arithmetic, and it flips the intuition: job definitions are tiny, execution history is the storage problem, and the queue absorbs the spikes.

Quantity	Estimate	How you got there
Executions	10k/s	≈ 860M executions/day — the write rate that sizes the queue and worker fleet.
Execution history	~1 TB/day	860M × ~1 KB per record; retain 30 days hot ≈ 30 TB, then archive to object storage.
Job definitions	~10M rows	recurring jobs are few relative to executions — gigabytes, not terabytes.
Minute-boundary spike	10–50×	everyone schedules at :00; the queue absorbs it, and I'd add jitter for jobs that tolerate it.
Worker fleet	~2,000	10k/s × ~200 ms average task admission ≈ 2k concurrent workers; autoscaled on queue depth.
Poll window	5 min	each poll fetches ~3M due executions (10k/s × 300 s) — a range scan on a time-bucketed index, not a table scan.

Say this out loud

Ten thousand a second is 860 million a day, so execution history is a terabyte a day — that's the table that needs partitioning by time, while the jobs table stays small enough to ignore.

Sketch the API

A small API; the design decision hiding in it is that a job and an execution are different resources, because one cron job produces millions of runs.

POST/api/v1/jobsbody: { taskId, schedule: { type: CRON|ONCE, expression|timestamp }, params } → { jobId }. Validates the cron expression up front.

GET/api/v1/jobs/{jobId}→ definition plus recent executions with per-attempt status — the debugging surface.

GET/api/v1/jobs?userId=&status=List and monitor; paginated, served from the executions store.

DELETE/api/v1/jobs/{jobId}Pause/cancel: marks the job inactive and tombstones any not-yet-fired execution rows.

Job vs execution is the load-bearing distinction: the job is the recurring definition; each planned run is its own execution row with its own status and attempt counter. Conflating them makes retries, history, and 'did the 3am run happen?' unanswerable — it's the modelling mistake this question is built to catch.
Task code is registered, not uploaded: jobs reference a taskId that workers already know how to run. Arbitrary code execution turns this into a sandboxing question I'd explicitly scope out.

Data model

Two tables: small jobs, huge executions. The one clever move is partitioning executions by planned-time bucket so "what's due next?" is a cheap range read, never a scan.

jobs

job_id PK · task_id · schedule (cron|once) · params · owner · status

~10M rows; the definition of record. Keyed by job_id.

executions

time_bucket PK · execution_id · job_id · planned_at · status · attempt · result

Partitioned by minute/hour bucket of planned_at — the poller reads only the next window's partitions.

dead_letters

execution_id · job_id · attempts · last_error · ts

Exhausted retries land here with an alert; humans decide, never the scheduler.

The database call

I'd put executions in DynamoDB or Cassandra partitioned by time bucket — a terabyte a day of append-mostly writes with one query pattern ("due in the next window") is exactly what they're built for. At a tenth the scale I'd happily use Postgres with an index on next_run_time and say so; I'd switch when the executions table outgrows one primary's write throughput, not before. Recurring jobs are materialised lazily: completing a run computes and inserts only the next execution row — never pre-expand an infinite cron into rows.

High-level design

The signature pattern is two-layer scheduling: a sharded watcher polls the database in coarse five-minute windows, and a delay queue provides the final seconds of precision. Workers pull with leases, so a crash means redelivery, not a lost job.

two layers: the watcher is coarse (minutes) and the queue is precise (seconds). notice workers write the next recurrence back to the executions table — the loop closes through the database, not through memory.

A client POSTs a job → the job service validates the cron expression, stores the definition, computes the first planned_at, and inserts one execution row in the right time bucket.
Every few minutes, each watcher shard range-reads its partitions for executions due in the next window and enqueues them with a delay set to the exact fire time.
The delay queue releases each message at its fire time — this is where 2-second precision comes from; the database was only ever accurate to the window.
A worker pulls the message under a visibility-timeout lease, flips the execution pending → running with a conditional update, runs the task, and writes the terminal status.
For recurring jobs the worker computes the next planned_at and inserts the next execution row. If the worker dies mid-task, the lease expires, the message reappears, and another worker retries — at-least-once by construction.

Deep dives — where the interview is won

Phase 01

Two layers, because neither the database nor the queue can do it alone

A database poll alone gives minutes of precision — poll every second and you're hammering the store with 86,400 range queries a day per shard for mostly-empty results; that's the 'thundering scan' anti-pattern. A queue alone can't hold a job scheduled for next March — delay queues cap out (SQS at 15 minutes) and holding millions of far-future timers in memory dies on restart. The composition fixes both: the database is the durable, unbounded schedule; the watcher moves only the imminent slice — due in the next five minutes — into the queue; the queue's per-message delay lands the final seconds.

The window length is a real dial: longer windows mean fewer, bigger DB reads but more messages sitting in the queue where a schedule change can no longer reach them; shorter windows mean the opposite. Five minutes is the conventional sweet spot, and cancelling a job inside the window needs a tombstone check at the worker — cheap, and worth mentioning unprompted.

poller reads DB for jobs due within window; hands to delay queue for precise firing.

Trade-offOnce an execution is in the queue it's committed — edits and cancellations within the window need a worker-side recheck against the database, which is one extra read per execution buying you a mutable schedule.

Phase 02

At-least-once comes from leases; correctness comes from idempotency

The guarantee that survives crashes is lease-based: a worker's pull starts a visibility timeout, and heartbeats extend it for long tasks. Worker dies → no heartbeat → the message reappears → another worker runs it. Combined with retries, this means the same execution will occasionally run twice — at-least-once is a euphemism for 'duplicates happen'. So every task must be idempotent, keyed on execution_id: a conditional status flip (pending → running via optimistic concurrency) stops two live workers claiming the same run, and side effects dedupe on the execution ID downstream.

The subtle case interviewers probe: worker completes the task but crashes before writing succeeded. The lease expires, the job runs again — and that's the contract. If the task sends an email, the email service dedupes on the idempotency key; if it can't, you've chosen at-most-once for that task and you say so explicitly rather than pretending.

worker takes a lease on the job row. renews mid-work. missed renewal → job re-picked.

Trade-offExactly-once is not on the menu — you pick at-least-once plus idempotent tasks, and accept a rare double-run in exchange for never silently skipping one. The alternative direction loses jobs, which is worse for almost every workload.

Phase 03

Retries must distinguish 'task failed' from 'worker died'

Two failure classes with different signals and different responses. A task that returns an error is an application failure: re-enqueue with exponential backoff and an incremented attempt counter, and after N attempts route to the dead-letter table with an alert — a poison job must never retry forever or block the fleet. A worker that stops heartbeating is an infrastructure failure: the lease expiry re-delivers automatically, and the attempt counter still increments so a job that crashes its workers also converges to the DLQ instead of cycling.

Backoff matters at fleet scale: a downstream dependency blips, 50,000 executions fail in the same second, and naive immediate retry re-creates the spike exactly when the dependency is trying to recover. Exponential backoff with jitter spreads the retry wave — the same reasoning as the minute-boundary spike, applied to failures.

failure has two causes. explicit failure → retry table. missed lease → auto re-schedule at expiry.

Trade-offBackoff trades recovery speed for stability — a transient blip now delays affected jobs by minutes. For genuinely latency-critical jobs I'd allow a per-job retry policy rather than one global curve.

Phase 04

No single scheduler: shard the watcher, don't elect one cron king

A single watcher is cron with extra steps — it's the SPOF the whole question exists to remove. Instead the watcher is a small fleet where each member owns a slice of the execution partitions, coordinated by partition leases (a lease row per partition in the database, or ZooKeeper if it's already in the stack). A watcher that dies stops renewing its leases and a peer picks up its slice within seconds. Because enqueueing is idempotent on execution_id — the queue or the worker's conditional flip dedupes — a brief overlap where two watchers read the same partition is harmless.

That idempotency is what makes the coordination cheap: you don't need consensus on exactly-one-watcher-per-partition, only a lease that's mostly exclusive, because duplicates are already handled downstream. Weak coordination plus idempotent operations beats strong coordination almost every time, and saying that generalisation out loud is worth points.

poller sharded by hash of job_id. no leader, no single point of failure.

Trade-offSharded watchers add lease-management machinery you don't need at small scale — one watcher with a hot standby is honest for thousands of jobs; the fleet earns its keep at millions.

Follow-ups you should expect

Why not use Kafka as the queue layer?

Kafka has no per-message delay — messages are consumed in log order, so a job due in four minutes blocks the partition or forces consumer-side timer loops, and you've rebuilt the problem. The primitive here is a delay queue: SQS with DelaySeconds, or a Redis sorted set scored by fire time drained every second. Kafka is the wrong tool and naming why is the point of the question.

Do you need 2-second precision? What would 1-minute buy you?

A minute of tolerance deletes the queue layer entirely — the watcher polls every 30 seconds and dispatches directly to workers, and the system halves in complexity. I'd ask the interviewer which they want, because the answer changes the architecture; asking is itself the signal. Most business jobs are fine at a minute; the 2-second target is what forces the interesting design.

How do you handle everyone scheduling at midnight?

Three layers: the queue absorbs the burst so nothing is dropped, workers autoscale on queue depth so the backlog drains in minutes, and for jobs that tolerate it I add scheduling jitter — run at 00:00 plus up to 60 random seconds. The spike is 10–50× average, so smoothing even half the jobs is a large win.

How would you add priorities or per-tenant fairness?

Separate queues per priority tier with workers draining high first, plus a per-tenant token bucket at enqueue time so one tenant's million-job burst can't starve everyone — without the rate limit, priority queues alone let a noisy tenant monopolise the high tier. Weighted fair dequeue across tenant queues is the heavier follow-on if tenants are large.

A job's execution takes longer than its recurrence interval — what happens?

Policy decision, and I'd surface it per job: skip the overlapping run, queue it to run late, or run concurrently. Default to skip-with-alert — concurrent runs of a non-reentrant task corrupt state, and silently queueing builds unbounded backlog. The mechanism is easy; choosing the default is the design work.

Where candidates lose the room

Designing a single cron server with in-memory setTimeout timers — one restart loses every pending job, and it's the exact design the question is testing you out of.
Polling the whole jobs table every second for due work — the thundering scan that turns your database into the bottleneck at precisely the moment everything is due.
No idempotency story, so retries and lease expiries send duplicate emails and double-charge payments — at-least-once without idempotent tasks is a bug generator.
Using database row locks as the work queue — workers holding transactions while tasks run serialises the fleet on the database's lock table.
Conflating the job definition with its executions — one row per cron job means no per-run status, no retry counts, and no answer to "did last night's run succeed?"