Why do pass rates alone give a weak picture of AI reliability?

Example Answer

The way I'd think about it is this: Pass rate is a coarse summary. It tells you how often the system cleared some bar, but it says almost nothing about what kinds of failures remain, how severe they are, or whether the misses cluster in exactly the places users care about most.

Two systems can both show an 85 percent pass rate and be radically different in practice. One may fail on harmless formatting issues. The other may fail on escalation, safety, or core business logic. A single blended rate hides that distinction.

I would rather see pass rates broken down by failure class, workflow, customer segment, and risk level. Reliability is about predictability under real conditions, not just how many green checks you can show in aggregate. A good scorecard keeps the headline simple without throwing away the failure structure.

Common Poor Answer

A weak answer is treating pass rate as the reliability metric. Pass rate can be useful, but by itself it hides severity, failure shape, and segment-level risk.