Design a real-time anomaly detection system in a data stream

Official answer

Define the anomaly and response before choosing infrastructure. For a sensor stream, an unusually high value, a sustained shift and a missing heartbeat need different detectors. Set event-level recall, false-alert burden, detection delay and service-latency targets.

Use a durable input with event ID, entity ID, event timestamp, ingestion timestamp and schema version. Validate records, deduplicate retries and partition by the state ownership needed by the detector. Define allowed lateness, corrections and when a window is final.

A simple starting design maintains a bounded past-only baseline per sensor and scores a new reading before adding it to that baseline. Use calendar/operating context where appropriate, define warm-up and zero-variation behavior, and investigate flagged values rather than automatically removing them. A trained model is an alternative only after time-ordered evaluation against this baseline; do not assume a framework ships a particular algorithm without checking its version and integration.

Persist state/checkpoints and make alert writes idempotent, for example with an entity/window/detector-version key. Decide how a corrected window revises or withdraws an alert. Bound state size, handle hot entities and overload, and specify fallback behavior when features or a model are unavailable.

Test duplicate replay, out-of-order events, gaps, restart and changed model versions. Evaluate entire incidents rather than rewarding many flags for one incident. Monitor both data freshness and model outcomes, and route alerts to an owned review workflow. A dashboard and a list of technologies are not evidence that the detector is accurate or operationally reliable.