How would you assess this assistant’s operating cost and peak capacity?

Instruction: Calculate monthly operating cost, cost per completed task, and peak demand against both limits. Recommend an operating plan and explain what evidence is needed for the latency target.

Context:

A fictional assistant completes 50,000 tasks per month. Each model attempt uses exactly 4,000 input and 800 output tokens. Exactly 10% of tasks require one additional identical attempt; all tasks then complete. Fictional rates are $2 per million input tokens and $8 per million output tokens. Retrieval costs $0.003 per attempt, and fixed operating cost is $600 per month. Ignore all other costs for this exercise.

Peak demand is 120 new tasks per minute, with the same 10% retry rate in that minute. The contracted limits are 150 attempts per minute and 600,000 combined input/output tokens per minute. The desired completed-response latency at the 95th percentile is 10 seconds. Limits and rates are exercise assumptions, not vendor claims.

Updated

Official answer available

Read the opening below, then unlock the full answer and practical guidance.

The stated operating cost is $1,557 a month, or about 3.11 cents per completed task. However, I would not approve the requested peak on the current contract. With retries, peak demand uses 132 attempts per minute, within the 150-attempt allowance, but 633,600 tokens per minute, which exceeds the token allowance by 5.6%...

Related Questions