Design a Kafka-based system for real-time stream processing that includes data aggregation, windowing, and joins. Explain your design choices.

Instruction: Outline a system architecture utilizing Kafka for real-time data processing involving aggregation, windowing functions, and joining streams. Detail the components used, including any external systems, and justify your design decisions.

Context: This question challenges the candidate to demonstrate their ability to architect advanced Kafka-based solutions for complex real-time data processing requirements. Candidates should cover the use of Kafka Streams or KSQL for tasks such as data aggregation, the implementation of windowing to process data within specific time frames, and the execution of joins across different Kafka topics or streams. Emphasis should be on scalability, fault tolerance, and processing efficiency.

Updated

Official answer

A representative design uses Kafka Streams to compute five-minute purchase totals per account and correlate purchases with payment events. Keep input/output schemas, keys and event timestamps explicit; choose serializers and timestamp extraction deliberately.

Consume purchases as a KStream, normalize the account key and call groupByKey or groupBy before aggregation. Aggregations such as aggregate and reduce belong to grouped streams/tables, not directly to an arbitrary KStream. For five-minute tumbling windows with a one-minute grace period, use TimeWindows.ofSizeAndGrace and aggregate each account's amounts into the windowed result table. Use exact monetary units or an appropriate decimal representation, with an overflow policy.

Windows use the configured record timestamps. The grace period concerns late records relative to stream-time progress, not simply waiting one wall-clock minute after each arrival. Define whether consumers need continuously updated totals or only final results, and bound any suppression buffer.

For a purchase/payment KStream-KStream join, key both streams by the join identifier, choose an explicit time-difference bound and grace period, and ensure compatible partitioning. One event can match multiple events in the other stream; a join is not automatically one-to-one.

For customer enrichment, a normal KStream-KTable join looks up the table state available to the processor. An event-time “as of purchase” lookup requires suitable versioned-table/history semantics and late-arrival policy; do not assume a current-state lookup prevents future-data leakage.

State stores, changelog topics and repartition topics support stateful computation and recovery. Provision their retention, replication and storage, and test restore time. Configure exactly-once processing when atomic Kafka outputs/state/progress are required; external service calls still need their own correctness contract.

Test window-boundary and late events, duplicate business identifiers, missing/multiple join partners, key skew, restart and state restoration before drawing throughput conclusions.

Streams DSL, TimeWindows API, processing configuration

Related Questions