Design a data pipeline (Uber-style)

Scope the requirements

Data infra is unglamorous but foundational. The interview scores on knowing the pieces (ingestion, storage, compute, orchestration) and the trade-offs between streaming freshness and batch correctness.

Functional — what it must do

Ingest events from ~100 microservices at 10M events/s peak.
Land raw events in an immutable data lake (S3/HDFS) partitioned by day.
Batch jobs produce curated tables (dimensions + facts) for BI.
Streaming jobs produce real-time aggregates for dashboards + ML features.
Explicitly cut: BI tooling itself, ML training platform — offer them back.

Non-functional — the numbers that shape the design

End-to-end freshness: raw lands within 5 min; curated within 30 min; batch tables refreshed hourly.
10M events/s peak ingest; 100 PB of durable storage.
Exactly-once semantics on curated tables (dedup on natural keys).
Backfill: reprocess a year of history in <24 h when schemas or logic change.
99.9% on ingest; downstream can lag under pressure.

Say this out loud

The trick is one durable log (Kafka) feeding two independent paths — a stream side for freshness and a batch side for correctness. Consumers pick their SLA.

Back-of-envelope estimates

The scary number is throughput. Storage grows fast but is cheap; compute for backfills is where the money goes.

Quantity	Estimate	How you got there
Events/s	~10M peak	100 services × ~100k events/s average during business hours.
Storage/day (raw)	~40 TB	10M events/s × 86400 × ~50 B avg × compression 5×.
Storage/year total	~15 PB raw + ~5 PB curated	Iceberg / Parquet columnar, S3-tiered by age.
Kafka partitions	~5,000	5 topics × 1,000 partitions each; sized for peak throughput × replay headroom.
Batch compute	~500 Spark executors peak	hourly refreshes + nightly rollups + backfills.
Streaming compute	~200 Flink task slots	smaller — only the queries that need <5 min freshness.

Say this out loud

10M events/s at ingest is Kafka scale. Storage is a five-year cost curve, and Iceberg partitioning is what keeps queries from scanning it all.

Sketch the API

Two ingest surfaces (SDK + CDC), one consumer API (SQL over Iceberg + streaming reads over Kafka). Nothing exotic.

POST/sdk/eventsClient SDK sends batched events. Handles retries + client-side dedup key.

TAILpostgres://source/table WALCDC connector reads DB transaction logs, emits row-change events into Kafka.

SQLtrino://warehouse/db.tableQuery engine over Iceberg tables — analytics, BI dashboards, ad-hoc.

CONSUMERkafka://curated_eventsStreaming consumer for real-time dashboards + ML feature store.

POST/api/v1/jobs/backfillTrigger a backfill of a curated table over a date range.

Every event carries a natural key + producer_ts: consumers dedup on the key inside a bounded window (batch does DISTINCT; stream does state-store dedup).
Schemas are enforced via a registry — producers register schemas before emitting; consumers pin to a version to survive additive changes.

Data model

Three storage tiers with different jobs: Kafka (short-term log), object storage in Iceberg (long-term source of truth), curated tables (facts + dimensions).

kafka topics

topic × partition · event bytes

7-day retention. Consumer offsets tracked per consumer group. Replay for reprocessing.

raw_events (Iceberg)

date PK · event_type · payload (json)

landed by Kafka Connect. Partitioned by date + type. Immutable per file.

curated_facts

(entity_id, event_type, ts) PK

produced by batch jobs (Spark). Dedup on natural key. Columnar Parquet.

dimensions

user_id → attrs (SCD type 2)

slowly-changing dims from CDC. History preserved via valid_from / valid_to.

feature_store

(entity_id, feature_name) → value

hot store (Redis) for online serving + offline store for training. Same feature computed both places.

The database call

Kafka for the log because it's the industry standard for this shape. Iceberg over S3 because it gives ACID + schema evolution + time-travel over columnar files. Feature store (Feast, Tecton) sits on top for ML — the same feature must be computable in both batch and stream to avoid train-serve skew.

High-level design

Two ingest paths flow into one Kafka log. From Kafka, three consumers fan out: raw landing, streaming aggregation, batch ETL. Orchestrator drives the batch side.

one log, three consumers. spark reads both live topic + iceberg for hybrid batch. flink feeds features + real-time dashboards.

Producers (services + CDC) emit to Kafka with schema-registered payload + natural key + producer ts.
Kafka Connect (S3 sink) lands raw events as Parquet files into Iceberg partitioned by date + type. This is the source of truth forever.
Streaming jobs (Flink) read from Kafka in real time: windowed aggregations, feature computations, dashboard rollups. Latency: seconds.
Batch jobs (Spark, orchestrated by Airflow) run hourly against Iceberg raw tables: dedup on natural key, join with dimensions, produce curated facts. Latency: minutes.
Backfills read historical Iceberg partitions and reprocess with fresh logic; the batch code and streaming code both write to the same downstream tables with the batch value winning on conflict.

Deep dives — where the interview is won

Phase 01

One log, two SLAs: streaming for freshness, batch for correctness

Streaming aggregations are fast but weak on correctness — late events, out-of-order data, windowing decisions all introduce approximation. Batch is slow but exact — it reads the full day's data, dedups against natural keys, joins with the latest dimension snapshot. Neither is enough alone.

The pattern (Lambda in old vocabulary, unified batch+stream in modern speak): let each downstream pick its SLA. Real-time dashboards subscribe to Flink. Billing subscribes to Spark batch. When they disagree by more than a tolerance, reconciliation reports flag the drift and the batch number is the truth.

same log, two consumers with different guarantees. reconciliation surfaces drift.

Trade-offTwo code paths that must agree. Kappa-only architectures (Flink for both) reduce duplication but struggle with retroactive corrections. Most FAANGs still keep both.

Phase 02

Iceberg / lakehouse: ACID over Parquet without a database

Raw Parquet in S3 is fast to scan but has no ACID — concurrent writers corrupt tables, and there's no MERGE-INTO semantics. Iceberg (or Delta, Hudi) adds a metadata layer on top: manifest files track which data files are current, snapshots enable time-travel, and atomic swaps of the metadata pointer commit new versions. Readers see consistent snapshots.

The killer feature is schema evolution: add a column, rename a field, without rewriting existing files. Old files stay untouched; new writes use the new schema; readers unify via the metadata. This is what makes a data pipeline evolvable at petabyte scale.

metadata layer is the lightweight index. data files are immutable parquet. snapshots swap atomically.

Trade-offSmall files problem: streaming writes create many tiny parquet files. Periodic compaction jobs merge them or reads get slow. Compaction is a first-class operational concern, not an afterthought.

Phase 03

Backfill without breaking downstream

Bug in a batch job means yesterday's numbers are wrong. Reprocessing requires reading a range of raw partitions and rewriting curated outputs. The trap: downstream consumers read curated tables continuously — mid-backfill, they see partial results. The pattern: write backfilled data to a shadow table, validate, then atomic swap via Iceberg snapshot switch.

For streaming state, backfill means resetting Flink offsets to the earliest and letting it re-consume. This is why Kafka retention must exceed the longest tolerable backfill window (7-14 days typical). For truly historical corrections (year-old data), Flink can't help — batch is the only option because the Kafka log is aged out and Iceberg holds the durable truth.

backfill writes to shadow. downstream reads never see partial state. swap flips the pointer.

Trade-offDoubles compute + storage during backfill window. Worth it — a live rewrite of a prod table observed by hundreds of consumers is a disaster.

Follow-ups you should expect

Why not just use one system (say, Kafka + KSQL for everything)?

KSQL / Flink can serve small use cases end-to-end, but breaks down at petabyte scale for two reasons: retention costs (Kafka is expensive per TB) and analytical queries (columnar Parquet is 10-100× faster for scans). The tiered architecture is what makes it economical at scale.

How do you handle schema evolution when hundreds of producers emit different versions?

Schema registry (Avro/Protobuf). Producers register schemas before publishing; each event carries the schema version. Consumers pin to a version; new fields are additive; deprecations get a quarter of overlap before removal. The registry rejects backward-incompatible changes at publish time.

Exactly-once — how far can you actually guarantee it?

Ingest is at-least-once (retries). Kafka has transactional writes so Flink can achieve exactly-once processing within Kafka. Batch is exactly-once by construction (idempotent job on immutable input). End-to-end exactly-once from producer through consumer requires idempotent sinks — usually achieved by natural-key dedup at the final table.

How do you serve real-time features for ML without breaking training?

Feast/Tecton pattern: same feature definition compiled to two implementations — batch (Spark on Iceberg) for training data, stream (Flink on Kafka) for online serving. Both write to a feature store. Nightly reconciliation checks that batch and stream compute the same feature over the same window to prevent train-serve skew.

What breaks first as ingest grows from 10M to 100M events/s?

Kafka partition count — you can't add without a rebalance. Plan capacity ahead. Second: consumer parallelism (Flink task slots) — scales linearly with partitions. Third: the small-files problem in Iceberg gets worse as file rate rises; compaction becomes more aggressive.

Where candidates lose the room

Skipping the schema registry — new producers break existing consumers silently.
Streaming only, no batch — no way to run correct historical reprocessing when a bug ships.
Writing directly to curated tables without a shadow — downstream consumers see torn state during a run.
Kafka retention shorter than the longest tolerable backfill — you can't rewind past the horizon.
Landing raw as CSV/JSON — everything downstream pays a parse tax. Parquet + Iceberg is the norm.