Design a scalable notification system for data-driven alerts

Official answer

Separate detection, logical alert creation, recipient/channel delivery and individual attempts. Define eligibility, severity, owner, deduplication window, suppression and resolution rules before choosing a queue or provider.

Evaluate versioned rules against a well-defined data window. Create an alert with stable identity, affected entity, evidence and rule version. Persist the alert and an outbox entry atomically where possible, then dispatch asynchronously with bounded retries and idempotency keys.

Track one delivery per eligible recipient/channel and separate each attempt's status. A provider acceptance response is not proof that a person received or read the message. Handle callbacks idempotently, rate limits, expired alerts and invalid destinations. A timeout can mean the provider accepted the message but the response was lost, so retries may duplicate delivery when the provider lacks adequate idempotency.

Group repeated symptoms into incidents, use cooldowns/escalation rules and protect critical alerts from noisy lower-priority traffic. Dead-letter queues need monitoring, ownership and an explicit retry/repair process; moving a message there does not ensure delivery.

Measure end-to-end delay, eligible deliveries completed, unresolved failures, duplicate notifications and human acknowledgement where available. Include eligible deliveries with zero attempts in the denominator. Test restart between persistence and send, lost acknowledgements, bursts and downstream outages; autoscaling alone does not guarantee that critical alerts arrive.