Describe how you would design a scalable data pipeline for real-time data processing.

Example Answer

I would start with the required event volume, acceptable delay, ordering, and recovery behavior. For order events, a durable log such as Kafka plus a stream processor and an idempotent destination may be enough. I would partition by order ID when per-order ordering is needed, persist checkpoints, and define how late events are handled. I would not promise end-to-end exactly-once just because the processor supports checkpointing. A replay can repeat a sink write unless the sink transaction or deduplication design covers it. I would test failure after the sink commits but before progress is recorded.

Design example

For Spark micro-batches written through foreachBatch, the default write guarantee is at least once. The batch ID can help deduplicate a retried batch, but the destination must implement the required atomic check/write or equivalent idempotency. Keep checkpoint identity and replay boundaries stable.

Track consumer lag, processing delay, checkpoint failures, and destination errors. Watermarks express a late-data policy; they do not make every delayed event disappear safely. Add another processing or storage service only when it solves a concrete requirement.

Practice follow-up

How would you replay a partition without duplicating already committed destination records?

Sources

Spark Structured Streaming sink and foreachBatch guarantees.