What is Kafka Streams and what is a basic use case?
Instruction: Explain the concept of Kafka Streams and describe a basic scenario where it can be applied.
Updated
Official answer
Kafka Streams is a Java client library for building applications that process Kafka records. The application runs in ordinary service processes; it is not a separate broker-side processing service. Its topology can filter, transform, join and aggregate streams, with local state and Kafka-backed recovery where configured.
A basic use case is an order dashboard. Consume validated order events, group them by store, aggregate a count or monetary total for a defined time window, and write the resulting updates to an output topic. A downstream service can materialize those updates for the dashboard.
Define the meaning of an event and of the window before adding records: an order-created event is different from a payment-completed event, and duplicate business events can double-count revenue. Choose event-time handling, a grace policy for late arrivals, currency treatment and correction/refund behavior. Use stable event identities or another deduplication strategy where required.
Streams distributes tasks over input partitions and can recover state after failures. Processing guarantees, topic replication, store retention and external side effects still need explicit design. Kafka transaction guarantees do not automatically make a dashboard database write exactly once.
Measure end-to-end age and correctness under out-of-order events, duplicates, restart and rebalance. Continuous processing reduces dependence on scheduled batches, but records are not guaranteed to be processed instantly: load, buffering, transactions and recovery all introduce latency.