How would you use PySpark to process streaming data?

Use Structured Streaming to express a streaming computation with the DataFrame API. Define the event schema, latency target, expected volume, event-time semantics and destination before selecting operators.

A practical flow is: supported streaming source → explicit parsing and validation → transformations or bounded state → a compatible output sink. File sources and Kafka have different replay and retention properties; extra systems such as Kinesis require an appropriate connector. Do not assume every connector from legacy DStream examples is a built-in Structured Streaming source. Structured Streaming overview

For a five-minute error-rate dashboard, parse event timestamps, group by event-time window and service, and calculate errors divided by eligible requests. Define malformed records, duplicates and late arrivals separately. A watermark helps bound state for supported operations; it is not a timer guaranteeing every window closes five minutes after wall-clock time. Watermark API

Select an output mode supported by both the operation and sink: append emits new finalized rows where applicable, update emits changed rows, and complete emits the full result table. A trigger is a scheduling policy, not a promise that processing completes within its interval. Start the writer with a unique durable checkpoint location and monitor query progress and failures. Streaming writer

Recovery guarantees depend on replayable input, retained checkpoints, deterministic processing and the sink's commit behavior. Kafka output can contain duplicates; external side effects need idempotency or a suitable transaction protocol. Test restart, duplicate delivery, bad records, late data and sink failures, then measure processing lag, state growth and dropped records. Do not equate a checkpoint with universal end-to-end exactly-once delivery.