Design a system for dynamic resource allocation in ML deployments.
Instruction: Outline an approach for dynamically allocating computational resources to ML models based on demand and performance requirements. Use the supplied capacity assumptions to size the service and explain how warmup and scale-down behavior affect the controller.
Fictional sizing model: one ready replica sustains 10 requests/second at the tested latency target. Use 70% of that rate as the planning limit. Peak demand is 70 requests/second. New replicas take 90 seconds to become ready. Size for loss of one replica, assuming remaining replicas share traffic evenly and there is no other bottleneck.
Updated
Official answer available
Read the opening below, then unlock the full answer and practical guidance.
I would design the system around workload awareness: request volume, latency targets, model complexity, hardware availability, and the cost of degraded service...